fedora-infrastructure

#12993 mirrorlist-statistics cronjob is failing in OpenShift
Closed: Fixed with Explanation by kevin. Opened by zlopez.

Closed: Fixed with Explanation
Reopen Issue

Describe what you would like us to do:

Today I noticed that the mirrorlist-statistics cronjob is failing when trying to reach proxy03.fedoraproject.org. From what I understand proxy03.fedoraproject.org is now proxy03.rdu3.fedoraproject.org and the old one is pointing to IP that we no longer own.

Here is the log from the failing cronjob:

+ for proxy in ${MIRRORLIST_PROXIES}
+ '[' '' == yesterday ']'
+ for logfile in ${MIRRORLIST_LOGFILES}
+ ssh -i /etc/mirrormanager-ssh/ssh_mirrorlist_proxies.key -o StrictHostKeyChecking=no -o BatchMode=yes mirrormanager@proxy02.fedoraproject.org '( cat /var/log/mirrormanager/mirrorlist1.service.log | grep -v 127.0.0.1 | gzip -4 )'
Warning: Permanently added 'proxy02.fedoraproject.org' (ED25519) to the list of known hosts.
+ for logfile in ${MIRRORLIST_LOGFILES}
+ ssh -i /etc/mirrormanager-ssh/ssh_mirrorlist_proxies.key -o StrictHostKeyChecking=no -o BatchMode=yes mirrormanager@proxy02.fedoraproject.org '( cat /var/log/mirrormanager/mirrorlist2.service.log | grep -v 127.0.0.1 | gzip -4 )'
+ for proxy in ${MIRRORLIST_PROXIES}
+ '[' '' == yesterday ']'
+ for logfile in ${MIRRORLIST_LOGFILES}
+ ssh -i /etc/mirrormanager-ssh/ssh_mirrorlist_proxies.key -o StrictHostKeyChecking=no -o BatchMode=yes mirrormanager@proxy03.fedoraproject.org '( cat /var/log/mirrormanager/mirrorlist1.service.log | grep -v 127.0.0.1 | gzip -4 )'
ssh: connect to host proxy03.fedoraproject.org port 22: Network is unreachable
+ rm -f /tmp/tmp.wtnZaur0g6/mirrorlist.log.gz
+ rmdir /tmp/tmp.wtnZaur0g6

When do you need this to be done by? (YYYY/MM/DD)

I'm not sure if this is urgent as it affects only statistics.

Metadata Update from @kevin:
- Issue assigned to kevin

This could be a RHIT firewall issue.

proxy03.fedoraproject.org correctly resolves to the external ip, there may be a path of openshift -> out -> externalip -> nat to proxy03 thats broken.

I will look into it.

It was a firewall issue.

I got them to fix that and re-ran the stats job.

It seems to be back on track now.

Let us know if there's still any issue.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

Looks like stats are down again

From what I see, it's trying to reach proxy05, which is down already for some time. We should probably remove it from DNS. The last successful run was January 14th, which corresponds to when the proxy05 started to had issues.

+ ssh -i /etc/mirrormanager-ssh/ssh_mirrorlist_proxies.key -o StrictHostKeyChecking=no -o BatchMode=yes mirrormanager@proxy05.fedoraproject.org '( cat /var/log/mirrormanager/mirrorlist1.service.log | grep -v 127.0.0.1 | gzip -4 )'
ssh: connect to host proxy05.fedoraproject.org port 22: Connection timed out
+ rm -f /tmp/tmp.efvGFbkcMU/mirrorlist.log.gz
+ rmdir /tmp/tmp.efvGFbkcMU

I removed proxy05 from the loop in https://pagure.io/fedora-infra/ansible/pull-request/3052, but the mirrorlist-statistics job is still on proxy11.

+ ssh -i /etc/mirrormanager-ssh/ssh_mirrorlist_proxies.key -o StrictHostKeyChecking=no -o BatchMode=yes mirrormanager@proxy11.fedoraproject.org '( cat /var/log/mirrormanager/mirrorlist1.service.log | grep -v 127.0.0.1 | gzip -4 )'
kex_exchange_identification: Connection closed by remote host
Connection closed by 67.219.144.68 port 22

I have been trying to regain access to proxy05's mgmt, so far without too much luck. ;(

Perhaps the script should continue when a machine times out instead of erroring out entirely?

For proxy11, there shouldn't be any blocks there... can you then check /var/log/secure on it after it fails?

It looks like it's working tho:

Jan 19 01:34:02 proxy11 sshd-session[1886235]: Starting session: command for mirrormanager from 38.145.32.3 port 55422 id 0
Jan 19 01:34:06 proxy11 sshd-session[1886235]: Received disconnect from 38.145.32.3 port 55422:11: disconnected by user

It seems that it was just temporary issue I hit when trying to run it manually. The cronjobs is finishing successfully now.
Metadata

OpenShift medium-gain medium-trouble ops

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.