2314 - Fedora has tons of old repos which are not going to be deleted fast enough. tryDelete is limited by delete_batch_size. If we move it to separate thread we can drop this limitation.

Might it also be possible to run kojira in a 'delete' mode? ie, just one shot to delete everything old/stale/marked expired ?

That would be helpfull for us before we fill up our storage again. ;(

We have:

delete_batch_size=500

but in the last 21 hours kojira has been running:

journalctl -l -u kojira -S 'Wed 2020-06-24 19:18:57 UTC'| grep rmtree | wc -l
88

So, something is weird with the existing batching too.

I've filed #2340 - it uses threaded version without limit. So, it could help. It is one thread compared to 4 processes in original variant, but it should run all the time. Can you try it?

So, still not seeing deletes with this, but after adding debugging perhaps I just misunderstood how this is supposed to work.

I expected: kojira would mark any repos older than a week (based on their mtime) for delete, then go delete them.

What seems to be happening: kojira is checking repos and setting the expire_ts to current time, checking it and seeing that it's not been a week and skipping them.

example:

[root@koji02 ~][PROD-IAD2]# stat /mnt/koji/repos/f33-build-side-22329/1604194
File: /mnt/koji/repos/f33-build-side-22329/1604194
Size: 4096 Blocks: 8 IO Block: 32768 directory
Device: 28h/40d Inode: 1434771851 Links: 9
Access: (0755/drwxr-xr-x) Uid: ( 48/ apache) Gid: ( 48/ apache)
Context: system_u:object_r:unlabeled_t:s0
Access: 2020-06-22 22:19:46.499265000 +0000
Modify: 2020-06-12 12:58:19.807116000 +0000
Change: 2020-06-12 12:58:19.807116000 +0000
Birth: -

repo is 14 days mtime

Jun 26 19:19:26 koji02.iad2.fedoraproject.org kojira[1246300]: 2020-06-26 19:19:26,397 [DEBUG] {1246300} koji.repo.delete:211 Repo 1604194 (/mnt/koji/repos/f33-build-side-22329/1604194) age: 283 sec
Jun 26 19:19:26 koji02.iad2.fedoraproject.org kojira[1246300]: 2020-06-26 19:19:26,397 [DEBUG] {1246300} koji.repo.delete:212 Repo 1604194 (/mnt/koji/repos/f33-build-side-22329/1604194) mtime: 1591966699 sec
Jun 26 19:19:26 koji02.iad2.fedoraproject.org kojira[1246300]: 2020-06-26 19:19:26,397 [DEBUG] {1246300} koji.repo.delete:213 Repo 1604194 (/mnt/koji/repos/f33-build-side-22329/1604194) self.event_ts: 1591966698 sec
Jun 26 19:19:26 koji02.iad2.fedoraproject.org kojira[1246300]: 2020-06-26 19:19:26,397 [DEBUG] {1246300} koji.repo.delete:214 Repo 1604194 (/mnt/koji/repos/f33-build-side-22329/1604194) self.expire_ts: 1593198882 sec
Jun 26 19:19:26 koji02.iad2.fedoraproject.org kojira[1246300]: 2020-06-26 19:19:26,397 [DEBUG] {1246300} koji.repo.delete:215 Repo 1604194 (/mnt/koji/repos/f33-build-side-22329/1604194) max: 1593198882 sec

so expire_ts got set to current time, max of times is that and so it 'sees' it as only 283seconds.

Is this the expected behavior? If so, I think it's not good... it means you have to run kojira for deleted_repo_lifetime before it ever starts deleting things.

I thought we changed this behavior when adding the mtime check... but seems not so?

FYI, I changed the delete keep time to something low and it's now deleting away.

There is TODO line for that :-)
I've added some "heuristic" It looks for newer repo and set expire_ts to time that newer repo was created. Anyway, not exactl sure, if it covert all situations (regenerated repo for older event)

@mikem Do you think, it makes sense this way (last commit)? I wonder what should be better expire_ts for such repo.

FWIW, I set our deleted_repo_lifetime to 1 day and with this threaded delete it was able to catch up and has deleted 40TB (yes, thats TB) of old repos.

It might be worth checking internal brew to make sure it hasn't been keeping a ton of old repos also....

brew is ok as it has long-running kojira process and lifetime of 36 hours, so it is keeping up.

Commit ee93da41 fixes this issue

Commit 74f18b68 fixes this issue

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/2336

Please continue any further discussion there.