#4250 Koji batch submission compilation
Opened by houlifei. Modified

Hello, I am using the Koji tool and encountered a scenario where I submit software packages in bulk for compilation. Kojihub receives the task and distributes it to various compilation machines, but there is a bottleneck in Kojihub. Is there any way to optimize it, or are there other configurations that can solve it.


Can you describe in more detail what you mean by bottleneck?

Metadata Update from @mikem:
- Custom field Size adjusted to None

Hello, the bottleneck I'm referring to is that, for example, I submitted 2000 software packages, but Kojihub can only open a limited number of dozens or hundreds of tasks at the same time. The rest of the tasks will be in a free state and then wait for Kojihub to distribute them to the compiler. Can we expand the number of open tasks at the same time.

What version of koji are you running? How many build hosts do you have active? How many arches are you building for?

There are two major limits for getting tasks onto hosts. The first is the maxjobs value. This can be set per host in the kojid config file. This is a hard limit on how many tasks the host will handle at one time.

The second is host capacity. This is an abstract measure of how much work the host can handle. It is compared to the weight value for tasks, which is a correspondingly abstract measure of how much resource the the task requires. The capacity value can be set per host using the koji edit-host command. The default capacity is 2.0, which is rather low. If you want your builders to handle more tasks in parallel, then you probably want to increase this.

https://docs.pagure.org/koji/server_howto/#a-note-on-capacity

Hello, Koji version is 1.32.0. I have set maxjob=32 to the maximum for each host, and the host capacity=32 has also been set to 32. My current problem is: batch submission of software package construction, all tasks are waiting in queue on the Kojihub side, and the tasks have not been assigned to each host yet. Even if each host sets the capacity to the maximum of 32, it may not solve this problem.

1.32.0 is over a year old. I strongly recommend you update the current version. Task scheduling works rather differently since 1.34.

all tasks are waiting in queue

all of them? That sounds like either 1) your build hosts are not connecting properly, or 2) there are no build hosts in the channel for your tasks

Also note, that one build does not consume "1" of capacity. Typically buildArch tasks starts with consuming 1.5 "capacity points". Task will get higher weight based on how long previous buildArch tasks for given package have run previously. So, e.g. this kernel task in fedora's koji eat 3.46 of capacity giving you 32 / 3.46 ~= 9 similar tasks being able to run in the same time on given builder.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/4250

Please continue any further discussion there.

Metadata