Interesting - rpminfo is the only table containing payload hashes and add_external_rpm is the only method checking the existence of the content. Anyway, last message says that koji was able to download cb0 from somewhere. What could be confusing is that @ceph-el9-copr part says that it is the rpm in the db, not necessarilly the new one's origin. Could it be that there are more external repos containing this nvrea (e.g. in the inheritance)?

There are, you can see it in the first SQL query I showed, but those are with a different hash (f8b...). I attribute the difference to how the packages are signed (ceph-copr vs. centos). Anyway, according to the repo priorities, ceph-el9-copr should win out.

~ > koji list-external-repos --tag ceph-pacific9-build --inherit
Tag                  Pri Mode       External repo name
-------------------- --- ---------- -------------------------
ceph-pacific9-build  5   bare       ceph-el9-copr
centos9-defaults     30  bare       cs9-baseos
centos9-defaults     40  bare       cs9-appstream
centos9-defaults     50  bare       cs9-crb
epel9                59  bare       epel-next9
epel9                60  bare       epel9

Ah, priority with bare mode doesn't make much sense. "bare" says, merge everything we found together. It simply runs mergerepo_c --pkgorigins --all -r repo1 -r repo2, ... with repos ordered by priority, but maybe it internally ignores order in such case. Can you check the repodata for those two buildroots? If paths for that file differs?

It seems to be using the one from ceph-el9-copr, as expected:

/koji/repos/ceph-pacific9-build/latest/x86_64/repodata > zgrep ncurses-base *-primary.xml.gz | grep location
  <location xml:base="http://linuxsoft.cern.ch/mirror/download.copr.fedorainfracloud.org/results/ceph/el9/centos-stream-9-x86_64" href="02530084-ncurses/ncurses-base-6.2-8.20210508.el9.noarch.rpm"/>

Ah, found that. That's the limitation of copr. These noarch packages are built separately for aarch64 and x86_64, so cb0 is for aarch, while 9ac for x86_64. You're probably building for both? It looks that some part of the build uses one while the second other. There should be only one noarch "binary". /thinking how to best workaround this - it would be simple if you build only x86_64 with hardcoding arch to url, but that's probably not your setup/

Compare
https://download.copr.fedorainfracloud.org/results/ceph/el9/centos-stream-9-x86_64/02530084-ncurses/
https://download.copr.fedorainfracloud.org/results/ceph/el9/centos-stream-9-aarch64/02530084-ncurses/

@ktdreyer Have you hit it with ceph?

/note, mergerepo honours order even for bare mode - tested meanwhile/ so this could be an ugly workaround (e.g. prepend hard-coded arch external repo first, so noarch packages will be always pulled from there) - but there must be some more elegant solution.

Wow, that's kind of messed up. Yes, we're building for both arches. I guess another workaround would be to add two external repos, one for each arch, and add both to the tag?

No, it would still use two different noarch rpms. So, it will be still failing for one of them.

Indeed, you're right, but I don't understand why. Why doesn't ceph-el9-copr-x86_64 show 9ac...?

koji=> select payloadhash, external_repo.name from rpminfo left outer join external_repo on external_repo_id = external_repo.id where rpminfo.name = 'ncurses-base' and version='6.2' and release='8.20210508.el9' and arch='noarch';
           payloadhash            |         name          
----------------------------------+-----------------------
 f8b190b2f6b39bcae862a39f8f784c55 | cs9-baseos
 f8b190b2f6b39bcae862a39f8f784c55 | cs9-testing-baseos
 f8b190b2f6b39bcae862a39f8f784c55 | cs9-latest-baseos
 cb0a55af38e6a518fe02a53a38616595 | ceph-pacific-9-crapr
 cb0a55af38e6a518fe02a53a38616595 | ceph-el9-copr-x86_64
 cb0a55af38e6a518fe02a53a38616595 | ceph-el9-copr-aarch64
(6 rows)

It will be "random". First encountered rpm will be the only one in the db. So, whichever buildArch task will get to that point first will win all.

My previous test failed because I renamed the dual-arch ceph-el9-copr to ceph-el9-copr-x86_64 to not add yet another repo to the koji database. This was a bad idea, because the hashes are not cleaned up.

I tried again on our test instance with two brand-new repos and now it looks better:

kojitest2=> select payloadhash, external_repo_id, external_repo.name from rpminfo left outer join external_repo on external_repo_id = external_repo.id where rpminfo.name = 'ncurses-base' and version='6.2' and release='8.20210508.el9' and arch='noarch';
           payloadhash            | external_repo_id |         name          
----------------------------------+------------------+-----------------------
 f8b190b2f6b39bcae862a39f8f784c55 |               61 | cs9-baseos
 cb0a55af38e6a518fe02a53a38616595 |               76 | ceph-el9-copr-aarch64
 9ac637d5d59ee837432e748b2913b49d |               77 | ceph-el9-copr-x86_64
(3 rows)

Indeed, a build on the test instance is running now and looks like it's working. So, two arch-specific repos is a valid workaround, if a bit ugly.

Now if only I could easily clean up the mess I've made in the DB... :smile:

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/3254

Please continue any further discussion there.