The ELNBuildSync does the following:

1. Request a side-tag
2. Tag some number of packages into that side-tag
3. Listens for newRepo "start" and "done" messages
4. Submits builds.

This is because we need to have those tagged packages in the buildroot before we can submit builds. The lack of an implicit newRepo (at least after the tagging) is surprising.

We replaced the direct newRepo trigger in this plugin with a call to request_repo, which accomplishes the same thing. See commit e3707bed7

This process is probably getting hung up for a different reason than is suggested above.

1. Request a side-tag
2. Tag some number of packages into that side-tag
3. Listens for newRepo "start" and "done" messages
4. Submits builds

You wrote that the creation didn't trigger the regen (it does if configured as before), but you're actually wanting a secondary regen post-tag here. Tagging does not automatically trigger a regen; this is the point of on-demand.

With on-demand repos, I don't think there is much reason for this process to wait on the newRepo. Let koji do it. After tagging, skip the wait and submit the builds either with --wait-repo (build will request/wait for a repo matching current tag contents) or --wait-build=NVR ... (build will wait for given builds to be tagged, then request/wait for a fresh repo).

So, for a full explanation of what we do here (code here)

1. We request a side-tag
2. We request the initial repo info from Koji
3. We tag some number of packages into that side-tag
4. We listen for newRepo "start" for ten minutes. If that times out, we assume we missed it due to a race and start listening for newRepo "done". We
5. We wait for newRepo "done" for twenty minutes. If that times out, we request the repo once more from Koji and check to see if it changed since the initial one in step 2.
6. If it's unchanged, we abort.

I find it difficult to believe that the repo was fully-created in the tiny window between 1) and 2) above, so even if we missed both fedora messages, after thirty minutes the repo info should at least have changed, no?

The example tag you list above was deleted by you after 13 seconds. Do you have a better example?

Sorry, that specific one was just me checking whether the message included --request.

Maybe try eln-build-side-98204 which is one that EBS tried to use and timed out.

With on-demand repos, I don't think there is much reason for this process to wait on the newRepo. Let koji do it. After tagging, skip the wait and submit the builds either with --wait-repo (build will request/wait for a repo matching current tag contents) or --wait-build=NVR ... (build will wait for given builds to be tagged, then request/wait for a fresh repo).

How would I do this from the API? Is there something I can pass to the opts field in session.build(src, target, ops, priority, channel) that would cause that to happen or is this only available in the CLI, which would implicitly handle the newRepo and wait in the client before actually submitting the build?

How would I do this from the API?

Both the --wait-repo and --wait-build cli options to the build command are basically passed through to the build task inside the opts parameter. So, e.g.

opts['wait_repo'] = True

or

opts['wait_builds'] = ['NVR1', 'NVR2']

(note the slight name variation for the last. the --wait-build option has action="append", dest="wait_builds")

Alright, I'm going to take that approach (and simplify things considerably on my end). I'm still unsure why the existing approach didn't work though; after 30 minutes I would have expected that repo ID to have changed, at least.

I'm still unsure why the existing approach didn't work though

I'm puzzled too. I don't even see a repo request for tag 98204. They do get cleaned up, but not this fast. Could the hub have dropped its trigger_new_repo setting?

I know I see a ton of side tag repos and successful builds in the system.

Ok, it appears that Fedora does not have trigger_new_repo = True in sidetag.conf. Maybe they never did? I thought we'd added the option for them, but...

I can enable that if it's desired/needed.
In the common case it shouldn't be needed though right? the sidetag is made, then someone does a build and it makes the repo then?
But I suppose it would avoid confusion for people doing a wait-repo for the sidetag?

In the common case it shouldn't be needed though right?

Yeah, demand will still trigger the regen later. I believe the main motivation for this was optimizing regen speed. See #3841 and #3808. However, there are other much more significant optimizations made there, so I guess that's why this option never got enabled.

I can enable that if it's desired/needed.
In the common case it shouldn't be needed though right? the sidetag is made, then someone does a build and it makes the repo then?
But I suppose it would avoid confusion for people doing a wait-repo for the sidetag?

We've been training people to do a wait-repo since the beginning. And fedpkg tells them to do it. So the principle of least surprise suggests to me that we should retain the automatic regen. If not, then at minimum we need to remove that message from fedpkg.

I'm enabling it for now, but we should get rpkg to stop suggesting people use wait-repo in that case. ;)

Is this working better now, or still a problem?

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/4225

Please continue any further discussion there.