fedora-infrastructure

#12673 Pulling from registry.fedoraproject.org fails on Copr builders
Closed: Fixed by praiskup. Opened by frostyx.

Closed: Fixed
Reopen Issue

Describe what you would like us to do:

We've seen many Copr build failures caused by this:

[root@copr-hv-x86-64-03-dev-00544144-20250721-080827 ~]# podman run -it registry.fedoraproject.org/fedora:rawhide bash
Trying to pull registry.fedoraproject.org/fedora:rawhide...
WARN[0030] Failed, retrying in 1s ... (1/3). Error: initializing source docker://registry.fedoraproject.org/fedora:rawhide: pinging container registry registry.fedoraproject.org: Get "https://registry.fedoraproject.org/v2/": dial tcp 38.145.32.20:443: i/o timeout
WARN[0061] Failed, retrying in 1s ... (2/3). Error: initializing source docker://registry.fedoraproject.org/fedora:rawhide: pinging container registry registry.fedoraproject.org: Get "https://registry.fedoraproject.org/v2/": dial tcp 38.145.32.20:443: i/o timeout
WARN[0092] Failed, retrying in 1s ... (3/3). Error: initializing source docker://registry.fedoraproject.org/fedora:rawhide: pinging container registry registry.fedoraproject.org: Get "https://registry.fedoraproject.org/v2/": dial tcp 38.145.32.21:443: i/o timeout
Error: internal error: unable to copy from source docker://registry.fedoraproject.org/fedora:rawhide: initializing source docker://registry.fedoraproject.org/fedora:rawhide: pinging container registry registry.fedoraproject.org: Get "https://registry.fedoraproject.org/v2/": dial tcp 38.145.32.20:443: i/o timeout

Is it possible that our builders got accidentally denylisted in some effort to block some AI scrappers or something?

The same command works fine on my workstation.

When do you need this to be done by? (YYYY/MM/DD)

Please soon, this fails many Copr builds.

Can you give a traceroute from the systems which are failing to get an idea where the block is

Thank you @smooge for jumping in.

Traceroute looks like this:

[root@copr-hv-x86-64-03-dev-00544144-20250721-080827 ~]# traceroute registry.fedoraproject.org
traceroute to registry.fedoraproject.org (38.145.32.21), 30 hops max, 60 byte packets
 1  _gateway (192.168.122.1)  0.550 ms  0.507 ms  0.462 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

By the way I am seeing similar errors also from DNF:

Updating and loading repositories:
 Fedora Infrastructure tag 42 - x86_64                                                                                                                                                                ???% |   0.0   B/s |   0.0   B |  02m00s
>>> Curl error (28): Timeout was reached for https://kojipkgs.fedoraproject.org/repos-dist/f42-infra/latest/x86_64/repodata/repomd.xml [Connection timed out after 30000 milliseconds] - https://kojipkgs.fedoraproject.org/repos-dist/f42-inf
>>> Curl error (28): Timeout was reached for https://kojipkgs.fedoraproject.org/repos-dist/f42-infra/latest/x86_64/repodata/repomd.xml [Connection timed out after 30000 milliseconds] - https://kojipkgs.fedoraproject.org/repos-dist/f42-inf
>>> Curl error (28): Timeout was reached for https://kojipkgs.fedoraproject.org/repos-dist/f42-infra/latest/x86_64/repodata/repomd.xml [Connection timed out after 30001 milliseconds] - https://kojipkgs.fedoraproject.org/repos-dist/f42-inf
>>> Curl error (28): Timeout was reached for https://kojipkgs.fedoraproject.org/repos-dist/f42-infra/latest/x86_64/repodata/repomd.xml [Connection timed out after 30000 milliseconds] - https://kojipkgs.fedoraproject.org/repos-dist/f42-inf
>>> Librepo error: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

The URL is valid, I can curl it on my workstation. On the Copr builder, this happens

[root@copr-hv-x86-64-03-dev-00544144-20250721-080827 ~]# curl https://kojipkgs.fedoraproject.org/repos-dist/f42-infra-stg/latest/x86_64/repodata/repomd.xml
 curl: (28) Failed to connect to kojipkgs.fedoraproject.org port 443 after 270909 ms: Could not connect to server

With verbose:

[root@copr-hv-x86-64-03-dev-00544144-20250721-080827 ~]# curl -v https://kojipkgs.fedoraproject.org/repos-dist/f42-infra-stg/latest/x86_64/repodata/repomd.xml
* Host kojipkgs.fedoraproject.org:443 was resolved.
* IPv6: (none)
* IPv4: 38.145.32.21, 38.145.32.20
*   Trying 38.145.32.21:443...
* connect to 38.145.32.21 port 443 from 192.168.122.145 port 55994 failed: Connection timed out
*   Trying 38.145.32.20:443...
* connect to 38.145.32.20 port 443 from 192.168.122.145 port 50662 failed: Connection timed out
* Failed to connect to kojipkgs.fedoraproject.org port 443 after 267825 ms: Could not connect to server
* closing connection #0
curl: (28) Failed to connect to kojipkgs.fedoraproject.org port 443 after 267825 ms: Could not connect to server

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: Needs investigation, copr, high-gain, ops

So this is a virtual machine on top of some other system? Can that system get to proxy01.fedoraproject.org or proxy02.fedoraproject.org ? If it can, then the VM may need a reboot or some other change. If the host system can't get then we need to see where it can't get access (can it get to www.redhat.com or www.smoogespace.com)?

I was going to say routing might be an issue but depending on where this VM is located, the ICMP packets off the host machine may be blocked (and it would require something like using traceroute with TCP packets to port 443 to see if it can get more data). However, your system seems to be able to get DNS for the various hosts correctly so if that is working.. something seems to be possible.

This exact instance is a libvirt VM running on vmhost-x86-copr03.rdu-cc.fedoraproject.org. Same problem on other Copr HVs:

  • vmhost-x86-copr02.rdu-cc.fedoraproject.org
  • vmhost-x86-copr04.rdu-cc.fedoraproject.org

I suppose other HVs are affected as well, but cannot run that exact commands there because they don't even finish their provisioning playbook now.

[copr_hypervisor]
vmhost-x86-copr01.rdu-cc.fedoraproject.org
vmhost-x86-copr02.rdu-cc.fedoraproject.org
vmhost-x86-copr03.rdu-cc.fedoraproject.org
vmhost-x86-copr04.rdu-cc.fedoraproject.org
vmhost-p08-copr01.rdu-cc.fedoraproject.org
vmhost-p08-copr02.rdu-cc.fedoraproject.org
vmhost-p09-copr01.rdu-cc.fedoraproject.org

I suspect some iptables vs nftables shenanigans :-/

Copr builders that run in AWS and other clouds works fine.

I can ping the vmhost-x86-copr01.rdu-cc.fedoraproject.org systems from bastion over ipv4 but not ipv6. 

[smooge@bastion01 ~][PROD-RDU3]$ ping -4 vmhost-x86-copr04.rdu-cc.fedoraproject.org
PING  (8.43.85.60) 56(84) bytes of data.
64 bytes from vmhost-a64-copr01.rdu-cc.fedoraproject.org (8.43.85.60): icmp_seq=1 ttl=49 time=18.0 ms
64 bytes from vmhost-a64-copr01.rdu-cc.fedoraproject.org (8.43.85.60): icmp_seq=2 ttl=49 time=17.9 ms
64 bytes from vmhost-a64-copr01.rdu-cc.fedoraproject.org (8.43.85.60): icmp_seq=3 ttl=49 time=18.0 ms
^C
---  ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 17.942/17.953/17.966/0.009 ms
[smooge@bastion01 ~][PROD-RDU3]$ traceroute -4 vmhost-x86-copr04.rdu-cc.fedoraproject.org
traceroute to vmhost-x86-copr04.rdu-cc.fedoraproject.org (8.43.85.60), 30 hops max, 60 byte packets
 1  reserved (10.16.163.252)  4.507 ms  4.479 ms reserved (10.16.163.253)  0.707 ms
 2  10.16.190.49 (10.16.190.49)  0.103 ms  0.072 ms  0.076 ms
 3  209.132.181.205 (209.132.181.205)  0.632 ms 209.132.181.204 (209.132.181.204)  4.360 ms  4.325 ms
 4  38.32.212.41 (38.32.212.41)  0.283 ms  0.265 ms  0.241 ms
 5  * * *
 6  te0-0-0-32.rcr71.rdu02.atlas.cogentco.com (154.24.79.141)  1.677 ms  1.805 ms  1.866 ms
 7  be2190.rcr21.clt01.atlas.cogentco.com (154.54.83.77)  4.901 ms  4.756 ms  4.736 ms
 8  * port-channel8193.ccr91.dca04.atlas.cogentco.com (154.54.170.81)  11.528 ms port-channel8194.ccr92.dca04.atlas.cogentco.com (154.54.170.225)  11.507 ms
 9  be8481.ccr42.iad02.atlas.cogentco.com (154.54.25.18)  12.988 ms  12.832 ms be3025.ccr42.iad02.atlas.cogentco.com (154.54.28.30)  12.957 ms
10  lag-10.bear1.wdc111.sp.lumen.tech (4.68.73.197)  10.973 ms  10.933 ms  10.915 ms
11  * * *
12  4.16.240.122 (4.16.240.122)  17.543 ms  17.525 ms  17.363 ms
13  8.43.84.1 (8.43.84.1)  95.422 ms  34.808 ms  34.794 ms
14  8.43.84.3 (8.43.84.3)  17.852 ms  17.858 ms  17.825 ms
15  8.43.84.4 (8.43.84.4)  95.319 ms * *
16  8.43.85.254 (8.43.85.254)  26.663 ms  37.983 ms  26.576 ms
17  vmhost-a64-copr01.rdu-cc.fedoraproject.org (8.43.85.60)  17.997 ms  18.180 ms *

I think this a problem with ansible having moved these vmhosts to nftables, but they are rhel8, so libvirt cannot insert the libvirt rules correctly, so nat doesn't work for them.

I do see nftables stopped and iptables started, but there's 0 rules in there, so libvirt might be still confused. ;(

@james could perhaps look at it?

So just looking at vmhost-x86-copr02.rdu-cc.fedoraproject.org the firewall seems to be in a very weird state.

Looks like the machine started on the 16th, and had nftables enabled, but then on the 18th it got "switched" to iptables:

```Jul 16 15:03:38 vmhost-x86-copr02.rdu-cc.fedoraproject.org systemd[1]: Starting Netfilter Tables...
Jul 16 15:03:38 vmhost-x86-copr02.rdu-cc.fedoraproject.org systemd[1]: Started Netfilter Tables.
Jul 18 09:03:00 vmhost-x86-copr02.rdu-cc.fedoraproject.org systemd[1]: Stopping Netfilter Tables...
Jul 18 09:03:00 vmhost-x86-copr02.rdu-cc.fedoraproject.org systemd[1]: nftables.service: Succeeded.
Jul 18 09:03:00 vmhost-x86-copr02.rdu-cc.fedoraproject.org systemd[1]: Stopped Netfilter Tables.

...with iptables being brought up then:
```Jul 18 07:58:54 vmhost-x86-copr02.rdu-cc.fedoraproject.org systemd[1]: Starting IPv4 firewall with iptables>
Jul 18 07:58:54 vmhost-x86-copr02.rdu-cc.fedoraproject.org iptables.init[1036231]: iptables: Applying firew>
Jul 18 07:58:54 vmhost-x86-copr02.rdu-cc.fedoraproject.org systemd[1]: Started IPv4 firewall with iptables.

...but as kevin said, there's no firewall rules for either iptables or nftables atm (and thus. no NAT rules). So my first guess would be to pick one and have the NAT rules up.

Also if I try the above curl, or a traceroute -T -p 80 kojipkgs.fedoraproject.org from the hypervisor machine ... it works.
Edited by james

This ticket suggests that the hypervisor shouldn't be running nftables:

https://pagure.io/fedora-infrastructure/issue/12531

...so my guess is that during the move those hosts got installed with a "default" variable of nftables=true, and then it got backed out to iptables on the 18th ... but it hasn't been reset/rebooted to the point where iptables rules are actually there (with nftables the rules are loaded on each boot, but IIRC with iptables there's some iptables restore magic that happens)?

I ran systemctl restart iptables on vmhost-x86-copr02 ... and there are rules now. curl still works on it. But I don't see any NAT rules, so not sure if anything changed for the builders.

In order for the guests to work libvirtd needs to insert some nat rules... rhel8 libvirt has no clue about nftables... but if iptables is active, perhaps restarting libvirtd will get it to re-insert rules?
Or I suppose rebooting?

Yeh, basically ran:

systemctl restart iptables
systemctl restart libvirtd

...looks like it should work now.
Edited by james

Cool. It may be affecting other of the copr_hypervisor hosts too...

Thank you very much @james! I tested, and these three seem to work fine now:

vmhost-x86-copr01.rdu-cc.fedoraproject.org
vmhost-x86-copr02.rdu-cc.fedoraproject.org
vmhost-x86-copr03.rdu-cc.fedoraproject.org

These still fail to provision the builders.

vmhost-x86-copr04.rdu-cc.fedoraproject.org
vmhost-p08-copr01.rdu-cc.fedoraproject.org
vmhost-p08-copr02.rdu-cc.fedoraproject.org
vmhost-p09-copr01.rdu-cc.fedoraproject.org

When I was working on the hypervisors, I switch them all back to iptables. They had nftables because of one package that requires it. So I updated the playbook to make sure the nftables package is gone and iptables are used instead.

Let me check the remaining ones, but it would be the same as @james said before, just restarting iptables and libvirtd should fix it.

I restarted iptables and libvirtd on the remaining ones. Could you try it now?

Thank you very much @zlopez, all the HVs work now. Except for this one:

vmhost-p08-copr01.rdu-cc.fedoraproject.org

but it looks like a different issue. I can't even SSH to it

[jkadlcik@hive ~]$ ssh frostyx@vmhost-p08-copr01.rdu-cc.fedoraproject.org
frostyx@vmhost-p08-copr01.rdu-cc.fedoraproject.org: Permission denied (publickey).

And when Copr tries to spawn the VMs, this appears in logs:

Starting install...
DEBUG:root: -> exit_status=1, time=2.092s
WARNING:root:Can't boot the machine, retry after 5
DEBUG:root:cmd: virt-install --connect qemu+ssh://copr@vmhost-p08-copr01.rdu-cc.fedoraproject.org/system --ram 4096 --osinfo fedora-unknown --vcpus 2 --vnc --features acpi=off --noautoconsole --import -n copr_hv_ppc64le_01_dev_00548467_20250722_220433 --channel 'unix,target_type=virtio,name='"'"'org.qemu.guest_agent.0'"'"'' --rng /dev/random --network network=default,model=virtio --network bridge=br0,model=virtio --disk vol=images/copr_hv_ppc64le_01_dev_00548467_20250722_220433_root,device=disk,bus=virtio --disk vol=images/copr_hv_ppc64le_01_dev_00548467_20250722_220433_config,device=cdrom,bus=scsi --disk vol=images/copr_hv_ppc64le_01_dev_00548467_20250722_220433_swap,device=disk,bus=virtio
ERROR    Requested operation is not valid: network 'default' is not active
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu+ssh://copr@vmhost-p08-copr01.rdu-cc.fedoraproject.org/system start copr_hv_ppc64le_01_dev_00548467_20250722_220433
otherwise, please restart your installation.

@frostyx The virsh default network was not running on the machine, so I started it with virsh net-start default. Could you try it now?

I don't see any reason why your ssh key should be rejected. You are in the correct group and the machines is correctly enrolled as IPA client in RDU3 IPA cluster. Maybe that was related to virsh network issue and it got solved with it.

@zlopez It didn't solve my SSH, but it definitely solved the spawning issue. All of those HVs spawn builders correctly.

I am so sorry though, I missed one more HV because we don't use it in STG. It's this one:

vmhost-p09-copr01.rdu-cc.fedoraproject.org

I tried running these myself:

[frostyx@vmhost-p09-copr01 ~][PROD]$ sudo systemctl restart iptables
[frostyx@vmhost-p09-copr01 ~][PROD]$ sudo systemctl restart libvirtd

but it didn't fix the issue.

What is the problem with vmhost-p09-copr01.rdu-cc.fedoraproject.org?

Could you run the ssh with -vvv for vmhost-p08-copr01.rdu-cc.fedoraproject.org?

What is the problem with vmhost-p09-copr01.rdu-cc.fedoraproject.org?

I will try to find some useful information

Could you run the ssh with -vvv for vmhost-p08-copr01.rdu-cc.fedoraproject.org?

Uploading as a file: ssh-debug

Looking at the vmhost-p08-copr01.rdu-cc.fedoraproject.org and probably see what is the issue, the user homes are owned by random UID instead of user. That means that the ssh frostyx@... doesn't know where to look. Need to check why that is happening as it's not the issue on other hypervisors.

Metadata Update from @zlopez:
- Issue assigned to zlopez

Metadata Update from @zlopez:
- Issue untagged with: Needs investigation
- Issue tagged with: high-trouble

Not sure what was exactly the issue there, but I re-enrolled it to IPA cluster and it now correctly recognizes the users. @frostyx Could you try it now?

the user homes are owned by random UID instead of user.

Good catch @zlopez

What is the problem with vmhost-p09-copr01.rdu-cc.fedoraproject.org?

I think iptables -L shows that there are missing rules related to libvirt. I am comparing the output with a different HV that works (vmhost-x86-copr01.rdu-cc.fedoraproject.org).

I tried restarting iptables and libvirtd which didn't help. I checked that nftables is disabled, I eventually rebooted the HV, didn't help either.

Not sure what should we do now. Maybe try running a playbook if it fixes itself?

Not sure what was exactly the issue there, but I re-enrolled it to IPA cluster and it now correctly recognizes the users. @frostyx Could you try it now?

Thank you very much, ssh frostyx@vmhost-p08-copr01.rdu-cc.fedoraproject.org works now :-)

What is the problem with vmhost-p09-copr01.rdu-cc.fedoraproject.org?

I think iptables -L shows that there are missing rules related to libvirt. I am comparing the output with a different HV that works (vmhost-x86-copr01.rdu-cc.fedoraproject.org).

I tried restarting iptables and libvirtd which didn't help. I checked that nftables is disabled, I eventually rebooted the HV, didn't help either.

Not sure what should we do now. Maybe try running a playbook if it fixes itself?

You are right, the vmhost-x86-copr01 has libvirt iptables rules which are missing on both vmhost-p08-copr01 and vmhost-p09-copr01. Will check out why.

So I checked other VM hosts and most of them doesn't have the special libvirt rules. So I don't think they are the problem as you would see the same issue with vmhost-p08-copr01.rdu-cc.fedoraproject.org

But I still don't understand what is the problem with vmhost-p09-copr01, what exactly doesn't work?

AFAICT, the problem is that VMs on that machines start, but do not get a working public ipv6 stack - making the outside world unable to ssh there (ipv4 is behind nat).

Now I realized that when we switch back from F42 bootc to non-bootc F41, machines boot fine. Please give us time to analyze the problem, it's likely problem in our (copr) scripting.

Ok, I switched the machine to nftables, because it's just too new for iptables:
c1f51e0f32eaa3e53db409fdcc8faa3322179e55
And the system seems to behave just fine.

Metadata Update from @praiskup:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata

copr high-gain high-trouble ops

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.