That requires separate source repos per architecture, right?

If that is not already the case, yes.

I would also settle for an alternative way to get "correct" SRPM metadata / repository data, so long as it does not involve locally rebuilding all Fedora SRPMS ...

Isn't arch-specific SRPM design flaw? I already don't like ArchExclusive noarch packages. SRPM which builds differently on different archs is bit of a warning. What I could be thinking of is building SRPM on all archs and compare them (in same way what we do with the noarch packages) and if they differ I would fail such build. But I wouldn't store them (just one). Anyway, it still means jump of needed build resources for maybe a few problematic packages.

What are the real-world usecases for arch-dependent SRPM? Are they reasonable or should distribution hunt them and remove/fix them?

Isn't arch-specific SRPM design flaw?

It is a neccessary design flaw. E.g. if you need to BR a library for additional features support, but it only is possible on some architectures, you guard it. E.g.:

# Support for the Valgrind debugger/profiler
%ifarch %{valgrind_arches}
BuildRequires: valgrind
%endif

SRPM which builds differently on different archs is very common.

I wonder if this could be solved by using rich deps, e.g. (pseudocode):

for <arch> in %{valgrind_arches}:
    BuildRequires: (valgrind if rpm(<arch>))

Obviously, this would need to be macronized

@churchyard It is different problem (and it is working for years). SRPM will be same for all archs here. BuildRequires will be interpreted in buildArch step. We're talking about git->SRPM step. SRPM itself has no BuildRequires. So, not sure what I'm missing.

Untrue. SRPM files do have RPM headers, which includes the BuildRequires. Those end up in repository metadata via createrepo.

Right now, it is impossible to query SRPM requires (BuildRequires) from repository metadata, because those SRPM files claim to be "noarch" but actually carry architecture-dependent RPM headers.

Interesting - will look into it more deeply.

For my use cases, something like an rpmdiff output between the SRPM files from different architectures would be enough, since I only care about the package metadata and not their actual contents.

But koji is the only place in the Fedora pipeline that has the complete information, since SRPM files from the actual builders are discarded.

In fact - they don't even exist. We build only one srpm on random arch from buildroot architectures. So that it is the only SRPM created in koji and used for everything.

koji might not produce it, but mock builds (look at the mock result directory) will produce a rebuilt SRPM file. koji just ignores that file.

Sure, the srpm headers can vary across arches, but the source they contain does not.
If you go to reproduce an rpm build, then as you point out mock will remake the srpm for that arch. That reproduces the arch-specific srpm, and the build itself should proceed as before.

Saving what is essentially the same srpm 5-8x over for every rhel build sounds incredibly wasteful to me.

For my use cases, something like an rpmdiff output between the SRPM files from different architectures would be enough, since I only care about the package metadata and not their actual contents.

What exactly is your use case? I'm not getting a clear picture from this issue so far. What is the specific problem caused by this behavior (which predates Koji itself)?

The use case generally is: querying build dependencies for all architectures, not just a random one.

Why is this issue coming up out now after 15 years of the same behavior in Koji (and going back to the rpm build systems that Koji replaced)?

To clarify: It has always been an issue.

I agree, this is not a new issue, it's probably just that there's now more services relying on "correct" dependency graph information. For example, the repochecker service provides the backend for the FTI/FTBFS/broken dependencies data shown in the Packager Dashboard.

The only data source it can rely on is yum repository metadata, and that is still frequently wrong since architecture-dependent BuildRequires are garbled and randomized by which koji builder built things.

I can see that there is some room for improvement here, but I don't like the idea of keeping multiple copies of the srpm.

We might want to do something like we do for noarch rpms, where we actually compare the differently produced files and make sure there are no unexpected changes. In the srpm case, the payload contents should be identical and only the headers should vary. We could assert the former and log the differences in the latter.

Yeah, that would be a good start. For everything else, it's probably hard to work around due to limitations in how RPM works ...

Ideally, there would be specialized repository data for different arches, but only one copy of the SRPM is kept, but that's probably hard to achieve with how the responsibilities are split between koji / pungi / bodhi and the build / compose tasks?

Saving what is essentially the same srpm 5-8x over for every rhel build sounds incredibly wasteful to me.

+1. This would be an very costly solution to a very niche problem.

for in %{valgrind_arches}:
BuildRequires: (valgrind if rpm())

I think this would very cool. It'd also allow very easy introspection of dependencies for other architectures from a single srpm file locally.

Here is the latest example of a packager (me) confused by this issue. I want to check the impact of upgrading a package. To do that appropriately, I need a full list of all reverse dependencies of a package. This issue means that I don't get a full list from repoquery: I get a partial list, with no way of knowing that anything is missing, much less what is missing.

I like the rich dependency idea, too.

Can we at least not choose the i386 build to promote (unless a package is only built for i386, of course)? Given the number of packages not built for i386 these days, that would eliminate a large class of these issues.

@jjames you can now edit the buildtag, so noarch and srpm are built on specific arches in given tag. It is not ideal but it solves the issue for now:
https://pagure.io/koji/issue/3809 e.g.

koji edit-tag fedora-41-build -x noarch_arches=x86_64,aarch64

This is already set in Fedora.

Doesn't help for packages that are not noarch packages.

The richdep solution is IMO the only viable proposal. But somebody would need to scope out if it's actually feasible, and design some guidelines for it. A complication that arose in the last few years is %generate_buildrequires. The result of that might be arch-specific. It's still compatible with the richdep solution, but would require some guidelines and maybe changes in the implementation. Anyway, I don't think koji needs to do anything here, at this point. If we decide to require BuildRequires to be identical on all arches (by using richdeps to express arch-specific requirements), then as a final step we could change koji to check that all the src.rpm packages actually have indentical BuildRequires.

The richdep solution is IMO the only viable proposal. But somebody would need to scope out if it's actually feasible, and design some guidelines for it. A complication that arose in the last few years is %generate_buildrequires. The result of that might be arch-specific. It's still compatible with the richdep solution, but would require some guidelines and maybe changes in the implementation. Anyway, I don't think koji needs to do anything here, at this point. If we decide to require BuildRequires to be identical on all arches (by using richdeps to express arch-specific requirements), then as a final step we could change koji to check that all the src.rpm packages actually have indentical BuildRequires.

That would work.

At least the dependency generator for Rust always produces arch-independent BuildRequires, so we should be safe on that front.

This is already set in Fedora.

Doesn't help for packages that are not noarch packages.

It should affect also all srpm tasks as we've treated them as "noarch" in this case. See getSRPMFromSCM part in https://pagure.io/koji/c/65758fe . If this doesn't work it is probably a bug.

Maybe this only affects where the initial SRPM is built, but not which one of the buildArch SRPM outputs is copied into the final build output?

I wonder if this could be solved by using rich deps, e.g. (pseudocode):

for in %{valgrind_arches}:
BuildRequires: (valgrind if rpm())

same thought here

if in rpm -q sympy-1.12.1-1.fc41.src.rpm --requires

we got : (python3dist(tomli) if python3-devel < 3.11)

we should have something like : (antlr4 if arch is %{java_arches} )

That'd require changes to rpm. (And probably other things.)
Requires: (antl4 if (rpm(aarch64) or rpm(ppc64le) or rpm(s390x) or rpm(x86-64)))
works today.

(Note that %{java_arches} evaluates to … x86_64, but the rpm Provides is with -. That'd need to be handled explicitly.)

That'd require changes to rpm. (And probably other things.)
Requires: (antl4 if (rpm(aarch64) or rpm(ppc64le) or rpm(s390x) or rpm(x86-64)))
works today.

(Note that %{java_arches} evaluates to … x86_64, but the rpm Provides is with -. That'd need to be handled explicitly.)

LGTM, I think we should explore this way.

Solutions like replacing the, if arch R: or BR: foo, with R: or BR: (foo if (rpm(aarch64) or rpm(ppc64le) or rpm(s390x) or rpm(x86-64))) are welcome since it does not involve changes to rpm

%pyproject_buildrequires now produces different outputs on different architectures because it simply asks Python if the requirement is relevant in our environment (and that includes architecture) and prints it (or not) based on that. Changing it to print the requirements with the conditionals is not trivial but it is probably possible.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/2726

Please continue any further discussion there.