Of course I see we have:

channel =
    method osbuildImage :: use osbuild
    method newRepo distRepo buildSRPMFromSCM :: use createrepo

but why wasn't it hitting this before?

why did you have that rule for buildSRPMFromSCM in your policy?

afaict, this has nothing to do with it being a chain build

but... we don't have any ppc64le builders in the createrepo channel

It appears that you do
https://koji.fedoraproject.org/koji/channelinfo?channelID=2

The one that took this task was added to the channel by you earlier today according to list-history.

This task is from before you deployed 1.35 and it is also in the createrepo channel
https://koji.fedoraproject.org/koji/taskinfo?taskID=123025378

Zooming out a bit.
This came up in chat because the dev was complaining that their chain-build was stalled.
This task did wait to start for quite a bit.

Created Wed, 09 Oct 2024 23:11:12 UTC
Started Thu, 10 Oct 2024 00:15:08 UTC

It was not assigned to a host until just before this

[mikem@localhost koji]$ koji scheduler-logs --task 124643933
Task       Host                 Time                      Message                       
----------------------------------------------------------------------------------------
124643933  buildvm-ppc64le-26.iad2.fedoraproject.org Wed Oct  9 20:15:04 2024  Assigning task            

This means the scheduler wasn't getting to it. I'm not 100% sure why, but I'd guess that this is because the createrepo channel was getting hammered, and newRepo tasks have a higher priority than this buildSRPMFromSCM task. I see a lot of newRepo tasks in that time, predominantly from side tags.

I'm surprised to see so many with the on-demand changes. Was there perhaps an 1.34 kojira instance still running?

Until I added some tonight we had 0 ppcle64 builders in the createrepo channel. Only x86_64 and aarch64. ;(
I added one when I saw this was stalled waiting for one.

Ok, this situation didn't happen in 1.34 because before 1.35 buildSRPMFromSCM was always noarch. This changed in #4013. Now the tasks will get a specific arch when needed, following the same rules as noarch buildArch tasks (i.e. honoring ~~exclusivearch, excludearch, and~~ noarch_arches).

EDIT: well for this task we don't know exclusivearch and excludearch yet, so just honoring noarch_arches setting for the tag.

ok. So, looking back in our history, we set buildSRPMFromSCM to use createrepo channel to avoid them happening on s390x builders (since those are in a remote DC and slepping around the source, etc was expensive).

Will ponder on what we want to do here. It might be nice if there was a way to do different things with buildSRPMFromSCM and noarch tasks, but I guess it makes sense that they would be treated the same...

we set buildSRPMFromSCM to use createrepo channel to avoid them happening on s390x builders

Since the code is now honoring noarch_arches and you have set noarch_arches for all your base build tags to "aarch64 x86_64 ppc64le", these tasks will not hit s390 regardless of channel (as long as that tag setting is in place for the build tag in question).

So, I think you can just revert that channel rule.

Or drop it even. Was there a reason for an explicit rule for newRepo and distRepo? Koji always sends these to createrepo channel anyway.

Well, yes, thats an option, but we also later removed all the ppc64le builders from createrepo because they were a lot slower than x86_64/aarch64... it might be thats no longer true, or doesn't matter so much now that there are a lot less newRepos.

Well, yes, thats an option, but we also later removed all the ppc64le builders from createrepo because they were a lot slower than x86_64/aarch64.

I'm not sure I understand the problem. In my suggestion above, newRepo would still be in the createrepo channel (this is the default behavior, the newRepo call passes an explicit channel to make_task).

The change to use noarch_arches for buildSRPMFromSCM caused us problems because:

• Until last week we had s390x in noarch_arches because people wanted to know their noarch packages might/could build on s390x. I removed that because we are constrained for s390x resources (largely due to CI sending new builds on PR rebases without canceling the old ones and when that is llvm...)
• We did not have any ppc64le builders in createrepo (because they were slow)

So, with this, I guess we need noarch_arches to have builders from every arch listed in createrepo channel.
Which means we cannot re-add s390x to noarch_arches (because they are in another datacenter and have no access to /mnt/koji normally), and we must re-add ppc64le builders no matter if they are slow or not.

Does that make sense?

So I could just re-add all the ppc64le builders to createrepo, drop the no longer needed rules and go on, but it would be nice if there was a way to still build noarch stuff on s390x, but not send createrepo jobs there?

So you want:

1. no s390x builders for buildSRPMFromSCM tasks (to reduce network costs), but other arches ok
2. no ppc64le or s390x builders in createrepo channel (because they're slow or lack access), but other arches ok there
3. s390x builders included for noarch buildArch tasks

Easiest option is to let go of the first requirement and let s390x handle the srpm tasks (after all they are still potentially getting rebuildSRPM tasks and noarch buildArch tasks). If you do that, you can stop routing buildSRPMFromSCM to the createrepo channel (leaving it only for repo tasks as intended). I don't know how much of a problem that network hit is.

it would be nice if there was a way to still build noarch stuff on s390x, but not send createrepo jobs there?

The above does this (the cost is that you also get buildSRPMFromSCM tasks on s390x)

The third requirement implies that s390x must be in noarch_arches, which means you get some s390x builders handling buildSRPMFromSCM tasks.

A somewhat heavy option would be to patch your builders to force these tasks to noarch, effectively reverting part of #4013. E.g. in BuildTask.getSRPMFromSCM:

                                             label='srpm',
-                                            arch=taskarch,
+                                            # workaround for https://pagure.io/koji/issue/4223
+                                            # arch=taskarch,
+                                            arch = 'noarch',
                                             parent=self.id)

Yeah, definitely don't want to carry a patch. ;)

So, I guess we can try dropping requirement 1 for now and see how it goes.

That would be:
drop: method newRepo distRepo buildSRPMFromSCM :: use createrepo from hub policy,

remove the ppc64le builders from createrepo that I added to it

readd s390x to noarch_arches.

Looks like distRepo is also hardcoded to pass 'createrepo' channel too.

I'll be thinking about better ways to address this within koji itself, but that will likely amount to a new feature

Fair enough, thanks for listening to my being confused. ;)

I'm going to do the above plan for now, will see how it goes.

Filed #4235 for follow up.

If the workaround is ok for now, we can close this one.

So far no complaints on the workaround... so yeah, can close. Thanks!

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/4223

Please continue any further discussion there.