fedora-qa / fedora_openqa

#102 Optimize handling of advisory and workaround repos for update/scratch build tests
Closed: Fixed by adamwill. Opened by adamwill.

Closed: Fixed
Reopen Issue

Currently, nearly every update test does the work of setting up the advisory and workaround repos, including downloading all the packages. That's a lot of repeated work and network transfer for every update. Especially for updates with a lot of packages, it wastes quite a lot of time, and is a potential point of failure.

I think it should be possible to optimize this. Instead of doing the work in the tests, we could do most of it in the scheduler. We could have the scheduler download the builds (for each arch to be tested), createrepo them, and maybe write .repo files as well, then make a disk image out of the results and place it in factory/hdd. It would then include that image as HDD_2 (uh...probably...that's one of the finicky bits I need to work out) when scheduling the tests. All the test itself need do is mount the image and copy the repo files into place (if we have the scheduler create them) or write them.

We'd probably need to twiddle with the tests a bit for this, especially the tests that use more than one hard disk themselves (primarily the image building tests - they use an extra disk for scratch space). It would also make manual scheduling via POST more difficult (well, really, impossible, unless you duplicated all that work manually), unless we left the code in the tests as a fallback when the disk image isn't available (and somehow had the tests figure out when to fall back). But overall I think it might be worth the effort, it should save a decent amount of test execution time and increase reliability.

Another drawback I guess is that this would make the scheduler block for some amount of time when scheduling an update. But I don't think that should be too terrible - we don't have such a heavy press of messages that it'd be a disaster if it takes maybe 5-10 minutes to schedule each update test. If that's a problem we'd have to go async, which always makes things more fragile and complex.

Another minor drawback: if we do this for the workarounds repo (not just updates), you can't 'apply' a workaround just by adding it in utils.pm and restarting the test; you'd have to reschedule to get the scheduler to rebuild the image with the new workaround. We might want to only do this for the advisory/task builds, and always let the test do the workarounds repo itself, I guess. I'll mull it over.

an alternative here is to have a single test do this work for each update flavor and upload the disk for subsequent tests to use, but the scheduler approach seems like the biggest win - we only need to do the work once per update run per arch, and we can probably optimize things better in Python in the scheduler than we can in an openQA test (I started out down this path by thinking about ways to optimize the download process itself within the test).

I actually did this last month:

https://pagure.io/fedora-qa/fedora_openqa/c/d4ad4d9426b7ce3a864461078013b24974bc5320?branch=main

I...think I reinvented the idea, and forgot I'd already had it half a year ago. :P Oh, well. It seems to be working pretty well. The 'you have to bump fedora_openqa and reschedule to change the workarounds' thing isn't that big of a deal in practice, rescheduling is easy.

The ultimate implementation uses ISO images (as they turned out to be much easier to build than HDD images), one for the update/task to be tested, one for the workarounds. If there are no workarounds, the workarounds one isn't built or attached. If we're testing a side tag, the updates image isn't built or attached either (and the tests know how to cope with this).

Metadata Update from @adamwill:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/quality/fedora_openqa/issues/102

Please continue any further discussion there.
Metadata

enhancement

None
None
Low
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.