This ticket had assigned some Differential requests:
D697

With @mkrizek we have found out that while overlords have dnf repos with sources like this:

baseurl=http://infrastructure.fedoraproject.org/pub/fedora/linux/releases/$releasever/Everything/$basearch/os/

all our minions use default dnf repos configuration, which is this:

metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch

and that returns dl.phx2.fedoraproject.org as the first option (which has a different IP from infrastructure.fedoraproject.org, but I assume it could be the same machine).

I think we might be seeing some issues with mirrormanager, which either times out or gets extremely slow, or there's some bug which sometimes returns an external repository which is extremely slow. So as the first countermeasure, I propose making sure our minions use the primary infrastructure repo directly (the same way our overlords are configured), using baseurl=http://infrastructure.fedoraproject.org/.... This could be done the same way we currently copy /etc/hosts from overlord to minion, using testcloud. Let's also copy /etc/yum.repos.d/fedora{,-updates,-updates-testing}.repo. Thoughts?

If this doesn't help, we can also try to configure dnf on disposable minions using these options:

       minrate
              storage size
              This sets the low speed threshold in bytes per second. If the server is sending data at the same or slower speed than this
              value for at least timeout option seconds, DNF aborts the connection. The default is 1000. Valid units are 'k', 'M', 'G'.
       timeout
              time in seconds
              Number  of seconds to wait for a connection before timing out. Used in combination with minrate option option. Defaults to
              30 seconds.

That of course wouldn't fix the problem, but we would get at least some further debugging info (currently we have no idea why it's taking so long and receive no output, this way we would at least receive some output).

It's also worth noting that dnf currently contains this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1272977
In certain situations (it's not immediately clear when) it waits for repo connection with extremely long timeout intervals, e.g. 10 minutes. Since there are at least 2 official repos enabled, this can mean 20+ minutes delay. That's enough for us to kill the process for providing no output. It is possible that the issues we're seeing can be related to this bug.

@mkrizek @tflink Could one of you please take this ticket and adjust our testcloud configuration on taskotron-dev (using ansible, I assume) to copy the dnf repo files from overlord to minion (the same way we currently copy /etc/hosts)? Let's do it as the first step, and if it doesn't help, let's dig deeper.

https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=71f302e6fd9fd789faa8edf4ca9bc841d9d922b8

Thanks. Please also add 'fedora-updates-testing.repo' just in case somebody enables it in their task. And then let's see if the issue disappears or not.

Thanks. Please also add 'fedora-updates-testing.repo' just in case somebody enables it in their task. And then let's see if the issue disappears or not.

Done.

The issue hasn't seemed to appear since the fix deployment.

Great. I have looked at the logs and I also don't see any further occurrences of this. Let's close?

Closing. Thanks @kparal for figuring this out.