koji

#2501 a sudden glut of newRepo tasks can stall the createrepo builders
Opened by mikem. Modified

Open
Close Issue

An unfortunate scheduler situation can happen if the total capacity of builders in the createrepo channel is small compared to the number of newRepo tasks.

If this situation, the builder(s) in the createrepo channel fill up to their maxjobs capacity on newRepo tasks before those tasks manage to create any of their createrepo subtasks. The hosts are stuck, running a bunch of tasks that are waiting for their subtasks, but there is no capacity left to take those subtasks.

This can in principle happen with other types of top level tasks that wait on subtasks they create, but I think it is much more likely with newRepo because:

  • kojira can regularly generate a large number of such tasks
  • there can be a bit of a gap between the time a newRepo task starts and the time that the createrepo subtasks are created

Note that subtasks are given a higher priority than their parents for exactly this reason. However, that doesn't help if hosts fill up on the parents before the children are created.

Metadata Update from @mikem:
- Custom field Size adjusted to None
- Issue tagged with: tech-debt

Possible workarounds

If this is happening on your system, you can try the following:

  • add more builders in the createrepo channel if you can

  • cancel a few of the open newRepo tasks that are clogging up the builders. This will free up some slots and allow the subtasks to move forward

  • to avoid this situation happening, you should keep kojira's max_repo_tasks setting in line with the size of your createrepo channel (i.e. the total of maxjobs across that set of builders)
Edited by mikem

This behavior is hardly new. I believe this has been a possible failure mode since day one. I think for the most part, the solution to this is "have enough createrepo builders".

When we eventually refactor the scheduler, we should definitely consider this case, but perhaps there is a smaller fix we can do.

Note, that is not the problem only of newRepo vs createrepo but of all parents/subtasks.

Btw, raising maxjobs on given builder (or sum(maxjobs) in createrepo channel) should be bigger than kojira's max_repo_tasks value as a rule of thumb. In such case there would be at least one createrepo running, so queue will not be blocked. Of course, it is not 100% solution, but maybe worth to note in example kojira.conf + docs.

This is popping up again now that kojira no longer uses max_repo_tasks.

This is popping up again now that kojira no longer uses max_repo_tasks

  1. I'm surprised that on-demand regens would be enough to overload. What are your numbers like? Are you making heavy use of repo.auto?
  2. Koji still has a limit on repo tasks. It has just moved to the hub config. See the migration notes for 1.35. The default value is 10.

My bad, turns out those were manual regens, not kojira-made ones. Thanks for pointing out that bit in the migration docs!

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/2501

Please continue any further discussion there.
Metadata
None

discussion tech-debt

None
None
Low
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.