#1913 Improve failed buildroot retention policy
Closed: Fixed by tkopecek. Opened by tkopecek.

Currently we have simple failed_buildroot_lifetime option for these. As times has changed we now can have failed buildroots which have tens of GBs and they can effectively cause DoS until they are cleaned few days later. We should have some better policy for this.

Simple variant would be: minimal expiration time, maximal expiration time, if there is a lack of space, delete oldest buildroot which has not reached max time, never touch anything under min time)

Any other proposals?


It might be nice to have it also be able to be set as a % of available disk space? I know that would be harder to implement, but if you could say 'always keep 50% disk available' it would be nicer than a lifetime. Think of the case where you have 20 builders... but 2 of them are one arch and 18 are another. Those 2 builders will be building a lot more than the other 18, and thus be running lower on disk space more quickly.

I suppose in some ideal world it might be nice to have it setable per builder, as builders are not always the same with the same disk, etc.

Metadata Update from @kevin:
- Custom field Size adjusted to None

I think we can continue to honor failed_buildroot_lifetime on a best effort basis (i.e. absent other reasons to remove it).

Seems like the simplest thing is to make space by deleting the oldest buildroots. I don't know if we need to have a flag to turn that off, but we could certainly add one.

Adding a min_buildroot_lifetime as you suggest is tempting in such a scenario, though it does leave room for the DoS scenario to continue.

It might be nice to have it also be able to be set as a % of available disk space?

That's not really applicable here. We already have the minspace setting that causes kojid to stop taking builds. That's our line, and when we cross it, we know to clear old buildroots.

Another thing to consider is prioritizing clearing based on buildroot size (to keep giant builds from hogging too much of this space)

Metadata Update from @mikem:
- Issue priority set to: Low (was: Normal)

I've created simple variant: Delete oldest failed buildroot, if there is no minspace available: PR #2082

Metadata Update from @tkopecek:
- Issue tagged with: testing-ready

Metadata Update from @mfilip:
- Issue tagged with: testing-done

Metadata Update from @mfilip:
- Issue untagged with: testing-done

Metadata Update from @mfilip:
- Issue tagged with: testing-done

Commit b972b572 fixes this issue

Commit 98d25bf3 fixes this issue

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/1913

Please continue any further discussion there.

Metadata
Related Pull Requests