Fred and I spoke with @mikem and @tkopecek in early October about our ideas regarding a Koji-based mass-prebuilder backend. We turned around some of the presented ideas as possible solutions to implement a Koji-based backend for mass-prebuilder. I add this comment here just to highlight that we spent time talking to Koji developers to determine what might be possible and what would and wouldn't work. I think what Fred is proposing here is achievable, but requires policy, and some code changes. The benefit of this would be that we could really start staging more complex transitions earlier in the release cycle and evaluate their impact on an ongoing basis. Staging a GCC update on an annual basis would also help the Ada developers who have to usually follow up with mini-mass-rebuilds of all the Ada packages due to the SONAME bumps; all of which could be done in the Step 3 RC candidate side-tag.

The biggest problem with this is all the NVRs that get locked as part of that. That's the main reason we don't allow non-mergeable side tags. Between the Dist-Git rules and Koji's archival and NVR locking policies, this becomes really difficult to do.

I agree with Neal - what you want here is fraught with issues.

Have you talked with COPR developers if it would be possible to add features / knobs to make COPR work better for what you need? I know that the Python team uses COPR to prepare for upcoming Python rebases for years now, and it seems to work there.

Other than that ... maybe instead of non-mergable side tags, use a side-koji - i.e. a different koji instance?

PS: For the Ada stuff that frequently breaks due to soname bumps in gcc, you could already avoid that with using a side tag for gcc + ada builds.

The biggest problem with this is all the NVRs that get locked as part of that. That's the main reason we don't allow non-mergeable side tags. Between the Dist-Git rules and Koji's archival and NVR locking policies, this becomes really difficult to do.

Could you expend why there is a need to lock NVR on non-mergeable side tags ? Conceptually, that shouldn't be that different from scratch-builds, with the noticeable exception that a repository is made temporarily available with the created artefacts.

I agree with Neal - what you want here is fraught with issues.

Have you talked with COPR developers if it would be possible to add features / knobs to make COPR work better for what you need? I know that the Python team uses COPR to prepare for upcoming Python rebases for years now, and it seems to work there.

Other than that ... maybe instead of non-mergable side tags, use a side-koji - i.e. a different koji instance?

---

PS: For the Ada stuff that frequently breaks due to soname bumps in gcc, you could already avoid that with using a side tag for gcc + ada builds.

We already have a backend using COPR as a build infrastructure.

They can try to be as close as possible, they may still not be close enough for some use cases, due to the simple fact that the underlying hardware is different, that the kernel used on the build machines is different etc...

One of the goals is to enable CI to perform rebuilds of dependant packages in order to gate updates. Sure they could rebuild the dependant packages in COPR, but the argument above holds true. Making use of a different build infrastructure increases uncertainties.

Apologies, but I don't understand why is this a FESCo ticket.

Apologies, but I don't understand why is this a FESCo ticket.

To my knowledge, these capabilities can be added by modifying the policies of the Fedora Koji hub.
Now if you tell me that FESCo doesn't have to agree upon these, fine with me.

I would find that very useful personally.

In the past, we encountered Koji-only related problems several times when deploying a new version of DNF. Testing proposed changes by building other packages against this new version in Koji became quite challenging. You either had to push those testing changes to dist-git, leading to a messy situation there after several iterations, or ask a Koji administrator to do the regular build in the side-tag without the dist-git push. However, this approach was time-consuming for both parties.

IMHO a FESCo ticket is not a place to brainstorm ideas and start a design discussion. A FESCo ticket is to vote about stuff that has usually already been discussed elsewhere.

If you want to do that, I suggest the devel mailing list (or perhaps discussion.fedoraproject.org, albeit I am unsure if you will reach the desired audience there).

If you already have designed this and are not interested in the initial design discussion (hard to tell from this ticket) I suggest you submit a change proposal explaining in detail how this is supposed to work.

I didn't think that what you propose is possible in Koji, so I assumed you'd opened this ticket to request somebody implement it. If what you propose actually is possibly by tweaking policies only, without NVR clashes and without the ability for folks to tag a scratch build from the non-mergeable side tag to a mergeable side tag, my apologies. The sentence "This request seeks an enhanced Koji infrastructure to support and facilitate the above-described workflow" made me think there is no such infrastructure yet.

@churchyard @decathorpe @ngompa Please consider this ticket as a FESCo Vote for pre-approval of the concept of non-mergeable side tags. I asked Fred to file this ticket with FESCo because I want to start with a smaller set of stakeholders, and keep expanding the discussion. All of the issues you have raised here are good, and we will take them all to the Fedora Devel list as a next step, but only if nobody here has a sustained objection. There could be reasons we didn't think about, so we first started with the smallest set of stakeholders, particularly Mike and Tomas, to discuss if it was possible given the current Koji code, and they agreed it should be possible with just policy changes. Next in my mind was to have FESCo review and Vote on the concept, and we've now had @jkolarik give positive feedback that this would be useful for them too. I don't see any sustained objections here, so I expect our next step to be to circle out to Fedora Devel and discuss the issues.

At some point we will have a fully refined set of semantics... what do we do then? Do we file a system-wide change request to transition the Koji configuration at some point after a Fedora major release?

I would still like to have a concrete answer for these questions:

Have you talked with COPR developers if it would be possible to add features / knobs to make COPR work better for what you need? I know that the Python team uses COPR to prepare for upcoming Python rebases for years now, and it seems to work there.

I don't hate the idea off hand, but a few things I would want to look at before approving things:

• We would want to look at garbage collection policies here. I wouldn't want people to keep big stacks of things in non mergable sidetags for a long time and we would want to delete the builds perhaps more quickly than normal builds when the non mergable sidetag is deleted.

• I'm not sure we would want to allow this globally for any fedora account holder. I mean, anyone can do scratch builds now, but if they could also build entire stacks of things they might start doing so and putting more stress on our storage/db/network, etc. Perhaps restrict to packagers? Or just have a seperate group to request access?

• I think it would be good to note why/how copr and koji builds are different for your needs and can't be converged. (basically same question as @decathorpe )

So, I think the idea is worth discussing more/persuing but I don't want to approve anything without all the details hashed out. :)

@kevin You ask some really good questions, so what we'll do is start making a draft of a system-wide change request that details more answers to the technical questions. Garbage collection policies is something we talked to the Koji developers about, and yes we'd want to outline that, along with access restrictions.

@kevin @decathorpe As long as we build in Koji there will be a developer need to do final validation in Koji. Any improvements to COPR mean we can move builder load to COPR for validation that happens frequently. We have a COPR backend for mass-prebuild it's the first thing we did and we've done large distro rebuilds with it and it's great, now we want to support Koji. We can go talk to the COPR developers about additional features, but the need to use Koji remains so long as Koji is the final build location. Everyone wants to validate on the final builders that will be used for the real builds.

In summary:
- We will write up a draft system-wide change for the new non-mergeable branches to provide background for a developer focused conversation and to answer FESCo questions.
- We will reach out to the COPR developers to discuss differences between Koji and COPR and include the details in our system-wide change.

I think at this point the answer is that FESCo is not yet ready to vote on the idea of non-mergeable side tags, but that we're moving in the right direction by gathering more input. Does that sound correct?

That sounds right to me... other fesco members might have different thoughts, but if they do I'm sure they will share them. :)

[copr rebuilds] can try to be as close as possible, they may still not be close enough for some use cases, due to the simple fact that the underlying hardware is different, that the kernel used on the build machines is different etc...

In general, packages must not depends on the kernel version or other hardware in the build system. When that happens, than that is a bug to be fixed. In fact, efforts are being done towards achieving build reproducibility in Fedora, which means we will expect package contents to be bit-for-bit identical on different builders, not just functionally the same.

Also, the proposed solution doesn't seem to provide the stated stability, because different workers in koji may have different kernel versions and/or will have different hardware characteristics and/or those may change in subsequent rebuilds. So even building in "official koji" doesn't mean that subsequent builds will see the same environment.

Overall, it sounds like copr is very much the solution we're looking for here. It should handle the need to do rebuilds, has support for automatic building on dist-git changes, chain builds, quite a few architectures, etc. You mentioned that you're already using it…

I would like to understand why copr is not considered to be enough.

Overall, it sounds like copr is very much the solution we're looking for here. It should handle the need to do rebuilds, has support for automatic building on dist-git changes, chain builds, quite a few architectures, etc. You mentioned that you're already using it…

As far as I know, there's no chain building capability in COPR. But the build process between COPR and Koji is basically identical since they use the same tools and similar invocations.

The only difference between the two is that Koji doesn't use bootstrap chroots to ensure the build environment is created by the target distribution. So in that regard, Koji is actually worse than COPR.

Overall, it sounds like copr is very much the solution we're looking for here. It should handle the need to do rebuilds, has support for automatic building on dist-git changes, chain builds, quite a few architectures, etc. You mentioned that you're already using it…

As far as I know, there's no chain building capability in COPR. But the build process between COPR and Koji is basically identical since they use the same tools and similar invocations.

The only difference between the two is that Koji doesn't use bootstrap chroots to ensure the build environment is created by the target distribution. So in that regard, Koji is actually worse than COPR.

So long as Koji is the Fedora build solution there are going to be developers who ask us to provide tooling to do final validation with Koji.

We will continue to move forward the points I raised here and come back with more information:
https://pagure.io/fesco/issue/3087#comment-880971

If you think there is anything else we need to do please raise it here and I can add it to the list. Thank you for your feedback Neal and Zbigniew.

So long as Koji is the Fedora build solution there are going to be developers who ask us to provide tooling to do final validation with Koji.

In an ideal world, Koji and COPR would be a single system (similar to openSUSE's OBS), but we don't have that in Fedora-land. 🤷‍♂️

The only difference between the two is that Koji doesn't use bootstrap chroots to ensure the build environment is created by the target distribution.

It does use bootstrap chroots.

The main differences between Koji and Copr we observed during Python rebuilds:

1) Copr builds SRPMs in one arbitrary chroot, while Koji builds them in the target environment (on one arbitrary architecture). This leads to problems with packages that use %if-%fedora Versions/Patches/Sources (e.g. python-apsw) or ExclusiveArch macros not available on all Fedora versions (e.g. until recently, packages using %{qt6_qtwebengine_arches}). It also complicates testing changes in such macros, see https://lists.fedoraproject.org/archives/list/buildsys@lists.fedoraproject.org/thread/25H6N4ZGDX2IM3I352JAF6VVI76UHMXH/

2) When you build package foo in a Koji side tag, the "official" build from the Fedora repositories is no longer available for your subsequent builds. However, when you build package foo in Copr, the foo package from Fedora repos is still considered by the dnf resolver (and may be used if the Copr-built one is not installable or is not satisfying a versioned dependency). See https://lists.fedoraproject.org/archives/list/copr-devel@lists.fedorahosted.org/thread/YO2K2D2JJ3AW2B643CNXYNJGOUOTE7GF/

3) Network and filesystem differences, leading to weird test failures only happening in Copr, e.g. https://src.fedoraproject.org/rpms/pyxattr/c/918e8e4b6389ca12303c357c0aa538853de57303?branch=rawhide

the Python team uses COPR to prepare for upcoming Python rebases for years now, and it seems to work there.

Yeah, we do use Copr for testing library updates and Python interpreter bumps, but we do sometimes struggle with differences between Copr and Koji, mainly the way it adds the Copr repo's packages to the rawhide packages as opposed to overriding them. If packageb is being updated to 2.0.0 in the Copr, but packagea depends on packageb<2.0.0, the Copr build will happily pull in packageb-1.1.0 from rawhide and build successfully when it would fail in a real Koji tag.

That being said, it does seem that the better solution would be to enhance Copr to make it easier to pull from the buildroot repositories and make it possible to change the way repodata works to make it behave more like real builds do.

For better or for worse, Koji is the production build system and Copr is for creating test repos.

EDIT: It looks like @churchyard and I wrote responses at the same time :)

Thanks all for your contributions.
I believe we can close this issue, we'll come back if we need more on that topic.