fedora-qa / fedora_openqa

#44 Worker VMs occasionally cannot reach the network
Closed: Fixed Opened by adamwill.

Closed: Fixed
Reopen Issue

Sometimes, an openQA test fails because the VM cannot reach the network. Classic symptoms are the test failing at the hub with "Error setting up base repository", and subsequently not being able to upload any log files with error curl: (7) Couldn't connect to server

Here are affected tests I've found from production (and the worker they ran on):

https://openqa.fedoraproject.org/tests/30679 - qa14:7
https://openqa.fedoraproject.org/tests/30240 - qa14:5
https://openqa.fedoraproject.org/tests/27743 - qa14:7
https://openqa.fedoraproject.org/tests/26857 - ?
https://openqa.fedoraproject.org/tests/29159 - qa14:9
https://openqa.fedoraproject.org/tests/29979 - qa14:7
https://openqa.fedoraproject.org/tests/29290 - qa14:8
https://openqa.fedoraproject.org/tests/28394 - qa05:4
https://openqa.fedoraproject.org/tests/30220 - qa14:7
https://openqa.fedoraproject.org/tests/27370 - qa14:8
https://openqa.fedoraproject.org/tests/27767 - qa14:5
https://openqa.fedoraproject.org/tests/26581 - ?
https://openqa.fedoraproject.org/tests/26957 - ?

In staging:

https://openqa.stg.fedoraproject.org/tests/32517 - qa06:3
https://openqa.stg.fedoraproject.org/tests/36508 - qa06:4
https://openqa.stg.fedoraproject.org/tests/36149 - qa07:2
https://openqa.stg.fedoraproject.org/tests/32860 - qa06:1
https://openqa.stg.fedoraproject.org/tests/31170 - qa06:4
https://openqa.stg.fedoraproject.org/tests/30939 - qa07:3

I'm currently trying to debug this, because I want to try and increase the robustness of the openQA tests in general, avoid these kinds of incorrect and sporadic failures so comparisons of results from run to run are more reliable.

So far I've set up a branch of the tests - called net-test - which adds a product which just runs four tests that boot to anaconda and immediately switch to a console and try to upload a file, then do a bunch of network diagnostics if that fails. I've run that five or six times on staging (so 20-24 tests so far) and it has not yet failed once.

I did kinda suspect this may have something to do with tests that run on the same box and at the same time as the openvswitch tests (the ones that use that special network setup so they can talk to each other), but we do have one case on qa05 and two on qa07, which are the non-openvswitch worker hosts, so I guess that can't be it.

So my test does occasionally reproduce this. I've been refining the debug output. Here's a fairly good one: https://openqa.stg.fedoraproject.org/tests/36730

from the 'ip addr' output we can see that we have no IPv4 address. The NetworkManager logs are in the serial console log: https://openqa.stg.fedoraproject.org/tests/36730/file/serial0.txt . Here is the log from a successful run for comparison: https://openqa.stg.fedoraproject.org/tests/36736/file/serial0.txt

Basically it looks like DHCP is failing. NM considers the connection up because it gets an IPv6 address, but really we have no connectivity. Now trying to figure out if this is an issue in NM somehow, or an issue in qemu (which is presumably providing the DHCP server).

https://openqa.stg.fedoraproject.org/tests/36805 shows that re-running dhclient after the initial failure seems to work.

I modified the test to go to dracut's emergency shell (by booting with rd.break) and run dhclient ens4 from there, instead of booting to anaconda. I ran that version ~100 times and it did not fail once. So this does seem to be somehow something to do with NM and/or anaconda.

So with the help of NM dev Thomas Haller we've got further in diagnosing this, and I've filed an RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1372854

it seems basically like dhclient is actually working, but NM doesn't realize it is, and kills it after a 45 second timeout, so we lose the connection...

https://openqa.stg.fedoraproject.org/tests/37112/file/serial0.txt is the best info we have so far, that contains the complete system journal and /tmp/syslog from a run which hit the bug.

This is now resolved, with the NM bug fixed and pushed stable for F25.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/quality/fedora_openqa/issues/44

Please continue any further discussion there.
Metadata
None
None
None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.