#192 figure out why tmpwatch cron jobs aren't working on taskotron masters
Closed: Fixed Opened by tflink.

While we did put cronjobs in place on the various taskotron masters (dev, stg, prod) to trim the number of artifacts and buildmaster logs kept, we keep getting "almost out of disk" warnings because for some reason, those cronjobs aren't doing what they're supposed to.

I've been running the tmpwatch commands by hand when the warnings come up but I'd rather figure out a way to let this get handled with cron.


As a side note, taskotron01.qa is currently low on disk and triggering nagios warnings. Since that's a production system, the free space isn't critical yet and infra freeze lifts tomorrow, we'd deal with it after freeze to avoid any freeze break requests

I haven't been able to find out why it's not working after an hour or so debugging. However, I used tmpfiles.d to clean the directories on taskotron-dev, if that works, I'll put that into ansible and deploy to our taskotron masters if there are no objections.

For some reason, even tmpfiles.d doesn't work on /srv/taskotron and /srv/buildmaster. It does work though on /var/lib/taskotron-trigger/cache. Since I don't know why tmpwatch nor tmpfiles.d work on those dirs, I set up cronjob that cleans the dirs with find, let's see if that works.

I set up cronjob that cleans the dirs with find, let's see if that works.

This seems to work, it's in ansible for dev and stg. I'll put that for prod tomorrow, if there no issues. Also, I think that 4 months old logs take up 80% of disk, so we might be getting nagios warnings either way. We might want to decrease to 3 months until we rebuild and have more disk space.

OK, seems to be finally working, I am going to close this. Please reopen if any issue appears.

Metadata Update from @tflink:
- Issue tagged with: infrastructure

Metadata