Currently we have simple failed_buildroot_lifetime option for these. As times has changed we now can have failed buildroots which have tens of GBs and they can effectively cause DoS until they are cleaned few days later. We should have some better policy for this.
failed_buildroot_lifetime
Simple variant would be: minimal expiration time, maximal expiration time, if there is a lack of space, delete oldest buildroot which has not reached max time, never touch anything under min time)
Any other proposals?
It might be nice to have it also be able to be set as a % of available disk space? I know that would be harder to implement, but if you could say 'always keep 50% disk available' it would be nicer than a lifetime. Think of the case where you have 20 builders... but 2 of them are one arch and 18 are another. Those 2 builders will be building a lot more than the other 18, and thus be running lower on disk space more quickly.
I suppose in some ideal world it might be nice to have it setable per builder, as builders are not always the same with the same disk, etc.
Metadata Update from @kevin: - Custom field Size adjusted to None
I think we can continue to honor failed_buildroot_lifetime on a best effort basis (i.e. absent other reasons to remove it).
Seems like the simplest thing is to make space by deleting the oldest buildroots. I don't know if we need to have a flag to turn that off, but we could certainly add one.
Adding a min_buildroot_lifetime as you suggest is tempting in such a scenario, though it does leave room for the DoS scenario to continue.
min_buildroot_lifetime
It might be nice to have it also be able to be set as a % of available disk space?
That's not really applicable here. We already have the minspace setting that causes kojid to stop taking builds. That's our line, and when we cross it, we know to clear old buildroots.
minspace
Another thing to consider is prioritizing clearing based on buildroot size (to keep giant builds from hogging too much of this space)
Metadata Update from @mikem: - Issue priority set to: Low (was: Normal)
I've created simple variant: Delete oldest failed buildroot, if there is no minspace available: PR #2082
Metadata Update from @tkopecek: - Issue tagged with: testing-ready
Metadata Update from @mfilip: - Issue tagged with: testing-done
Metadata Update from @mfilip: - Issue untagged with: testing-done
Commit b972b572 fixes this issue
Commit 98d25bf3 fixes this issue
This issue has been migrated to Fedora Forge: https://forge.fedoraproject.org/koji/koji/issues/1913
Please continue any further discussion there.