fedora-infrastructure

#11257 Koji setup in Fedora infra might be prone to inconsistencies in certain scenarios
Closed: Will Not/Can Not fix by obudai. Opened by obudai.

Closed: Will Not/Can Not fix
Reopen Issue

The osbuild integration in koji is occasionally hitting a weird error when uploading builds to koji. I think that it's related to how koji is set up in Fedora infra. Thus, I'm opening this issue in order to discuss this with the Fedora Infra team and koji maintainers.

Koji DNS setup

koji.fedoraproject.org currently points at two replicas of kojihub/kojiweb, see

$ host koji.fedoraproject.org
koji.fedoraproject.org has address 38.145.60.21
koji.fedoraproject.org has address 38.145.60.20

This can also be confirmed by looking at the ansible inventory containing two hosts:
- koji01.iad2.fedoraproject.org
- koji02.iad2.fedoraproject.org

However, these domains are not hosted in the public DNS zone:

$ host koji01.iad2.fedoraproject.org
Host koji01.iad2.fedoraproject.org not found: 3(NXDOMAIN)

Koji storage setup

koji uses /mnt/koji for storing its data. If I'm reading the playbooks correctly, /mnt/koji is an NFS mount on both koji replicas. This will be an important fact a bit later.

Osbuild uploading issue

The osbuild project is integrated Fedora Koji as a Koji content generator.

When osbuild builds an image, it uploads it to koji via its fastUpload method. Then, it calls the CGImport method in order to import the uploaded files as a new build.

When osbuild calls CGImport, it sometimes gets the following issue:

"Fault(1000): File size 751828992 for Fedora-IoT-38.20230419.1-20230419.1.aarch64.raw.xz (expected 751944620) doesn't match. Corrupted upload?"

However, when I inspect the uploaded files at kojipkgs.fedoraproject.org, the file size is as expected. An example of such a task is this one.

The theory

I think that the issue is that the uploading part is talking to a different replica than the following import call, basically:

osbuild calls upload on koji.fedoraproject.org, it's resolved as 38.145.60.20
osbuild calls import on koji.fedoraproject.org, but this time it's resolved as 38.145.60.21

Since the storage (/mnt/koji) is backed by NFS and the calls happen quite quickly after each other (10s of milliseconds), I think that the root cause is that NFS isn't fast enough to sync the data, and thus the second replica doesn't see the file as fully uploaded.

Note that I cannot think of a way to confirm this, so this is just a theory. Nevertheless, I simply cannot think of a different cause for this behaviour. If you have a better explanation, I would be more than happy. :)

What's different between osbuild and OSBS/MBS?

I asked myself this question: OSBS and MBS don't seem to be affected by this race condition, but why osbuild is?

It turns out that OSBS and MBS implement both upload and import in basically one Python method, see OSBS and MBS.

However, osbuild is different. In the case of multiarch builds, it uses a different set of machines for each architecture. In order to save bandwidth, they upload directly to koji. After all of them are done, another machine (might be one of the uploaders, but often, it might not) calls CGImport.

Thus, OSBS and MBS probably always talk to the same replica because everything is done on one machine in a short period of time. It's probably fast enough that the DNS cache still contains the same IP address for koji.fedoraproject.org.

However, osbuild can use different replicas for upload and import since they might run on different machines, thus not sharing the DNS cache.

Possible solutions

I came to the conclusion that retrying the CGImport call when osbuild gets the "Corrupted upload" is a safe option, so I implemented it. Note that this isn't deployed yet, so I cannot confirm that the fix and the theory is correct.

However, I would like to discuss whether this is alright in the long-term for the following reasons:

1) koji library doesn't retry on any xmlrpc faults, thus osbuild is diverging from the canonical implementation of the koji API client. This is slightly concerning.
2) because of DNS-based load-balancing of koji.fedoraproject.org, I think it might be prone to race conditions. This might bite someone else in the future, tbh...

For the record, I think that Red Hat's internal koji instance has also two replicas, but they don't share the same DNS record. Thus, clients need to choose which one they want to talk to, and thus this setup is certainly less prone to races. I believe that the split is done in a way that one replica is used for builders/content generators, the other one is used for external calls. I might be wrong, though. Note that we haven't seen this issue in Red Hat's koji ever, so it seems that the issue is indeed Fedora Koji specific.

I think that leaving koji.fedoraproject.org as it currently is set up is fine, because it apparently doesn't cause issues for many people. However, I think that introducing domains that are tied to just one replica might be a good solution for integrations that are known to have issues with occasional inconsistencies between replicas. Of course, this comes with the price that by using a specific replica, the setup would lose a bit of its HA properties.

@tkopecek, I want to ask you, as one of the maintainers of koji upstream, what do you think of this situation? Is using multiple replicas for koji.fedoraproject.org something you would discourage from, or is this fine? Do you see any way to improve the potential raciness of the deployment? Finally, are you fine with osbuild just retrying CGImport? It should solve the situation, hopefully, but it feels a bit dirty.

also cc @kevin from the fedora infra team.

Closing remarks: This is a theory and there might be a different explanation. I'm happy to hear your opinions, because I would really like to see the osbuild issue fixed properly. :)

Ok some complications to add to this. The network isn't exactly the same. Build systems look at the internal ip addresses which are 

$ host koji.fedoraproject.org
koji.fedoraproject.org has address 10.3.163.76
koji.fedoraproject.org has address 10.3.163.77
[smooge@batcave01 ~]$ host 10.3.163.76
76.163.3.10.in-addr.arpa domain name pointer proxy101.iad2.fedoraproject.org.

This leads to an additional layer of indirection.

[osbuild] <-> [koji]

is really

[osbuild] <-> [proxy101/proxy100- apache+haproxy] <-> [koji01/koji02]

so there could be multiple 'races' going on here in the builds. The client is going to talk to different front ends (proxyA or proxyB) and the proxies are going to talk to different backends (koji01.iad2 or koji02.iad2)

The NFS layer could still be a problem between koji01 and koji02 but there could be other problems involved.

Metadata Update from @kevin:
- Issue priority set to: Waiting on External (was: Needs Review)

I'm a bit unclear from what 'view' osbuilder is coming? Is this one of our builders reaching out to osbuild and downloading to the builder then uploading to the hub right?

So, from internal builders it's proxy101 and proxy110 to koji01 and koji02. (to expand on what smooge already said):
So, there's all the combos of those:

proxy101 -> koji01
proxy101 -> koji02
proxy110 -> koji01
proxy110 -> koji02

And yes, this could be a nfs sync issue. We have hit some other corner cases possibly related to this.

I don't think we can easily go back to one hub... we just have too much traffic, so hopefully we can solve this in koji or tweaking infrastructure.

Yep, I believe it is NFS and mostly what Ondrej found I agree with.
- Internally we've one hub instance dedicated to "koji communication" as Ondrej describes. This is for a few reasons (no need for load balancer, easier monitoring how much traffic is generated by consumers and how much by producers) Of course, there are some limitations. In fedora case it would mean first checking the access logs to determine how traffic is split between these sources and if it makes sense here. Number of calls one hub can handle is pretty high (we've started hitting syn queue overflows around 400-500 requests per second if I remember correctly)
- I would say that retry is ok solution (maybe simple sleep(1) should be sufficient). Koji client does automatically retry some "safe" calls. It wouldn't help here - CGImport is not "safe" (or idempotent) call. So, it is ok to me if CG which knows better what happened will retry it.

So looking at the koji proxy logs, I see that we regularly see peaks of traffic of 1600/second happening (these seem to occur several times per hour.) The average traffic per second is 250/second.

The largest user of the proxy is koji itself, with koji02 talking back to koji.fedoraproject.org with a "POST /kojihub" transaction. This is about 17.5% of total traffic. Next comes koji01 talking back to koji for 2% of the traffic. After that bodhi-backend for 1.3%. The builders all cycle around 1% or below.

I don't remember when we started doing it this way. I think it was to alleviate koji timeout issues happening between builders and servers during those peak loads.. but not sure.

So, I think we can close this then? osbuild will retry and if there's other corner cases we will try and deal with them?

The heavy traffic from 02 -> 01 is due to koji02 running all the koji-gc cron jobs. So, it does a bunch of queries to find old builds, etc. So, I think thats completely expected.

I'm a bit unclear from what 'view' osbuilder is coming? Is this one of our builders reaching out to osbuild and downloading to the builder then uploading to the hub right?

We have a quite nice diagram of all flows in the README here: https://github.com/osbuild/koji-osbuild/ The osbuild worker pool is a separate thing from the koji builders, and it does all the heavy lifting with koji builds - reserving a build ID, uploading all artifacts and importing them via CGImport. No big artifacts flow through a koji builder - workers talk directly to the hub.

And yes, this could be a nfs sync issue. We have hit some other corner cases possibly related to this.

Thanks for confirming.

I will go on and close this issue. I think that the results are following:

  • The Koji setup in Fedora infra is prone to NFS-related race conditions in corner cases. It's good that we all acknowledge them, and we can use this ticket as a reference if someone gets hit by them in the future. However, it doesn't make much sense to fix them, since this would probably require quite a lot of effort for little gain: I think that the consensus in this discussion is that these issues are quite rare and "workaroundable".
  • Speaking of workaroundable, we will implement retries for this issue in osbuild. I think it will solve this issue, but we will know more once we deploy it. :)

Thank you, guys, for this discussion. It felt very productive to me, and I'm glad that we reached these conclusions. :)

Metadata Update from @obudai:
- Issue close_status updated to: Will Not/Can Not fix
- Issue status updated to: Closed (was: Open)

I had marked this ticket as something I wanted to come back to and read, but
I've only been able to do that right now.

Please forgive my stupid question but: does osbuild run in the internal (Fedora)
network? If so, couldn't it use a specific koji instance (koji02.iad2....)
instead of calling koji.fp.o? Wouldn't it ensure that it's always talking to the
same koji instance and thus reduce the race-condition?

If osbuild does not have access to the internal network, this idea is of course
moot :]

Currently, no osbuild machines are inside the internal network, so this wouldn't work. It might change in the future, so I will surely keep this in mind. :)

Not on the Fedora VPN either?
Metadata
None
None
None
None
Highest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.