pungi

#1903 Consider dropping some/all reuse paths
Opened by adamwill. Modified

Open
Close Issue

There are complex paths in some old phases - gather, buildinstall, createiso, extra_isos - which attempt to determine whether "nothing has changed" since the previous compose, and reuse the results from it if so.

I contend that these are perhaps no longer worth the effort.

The gather phase reuse path only reuses the gather_packages results. This isn't really saving much time. In Fedora (where we have zillions of packages, and we disable this and all the other reuse paths in the pungi config), this whole process takes just a few minutes:

2026-04-10 05:20:07 [INFO    ] [BEGIN] ---------- PHASE: GATHER ----------
...
2026-04-10 05:22:11 [INFO    ] [SKIP ] Gathering packages (arch: x86_64, variant: Workstation)

so, 2 minutes 4 seconds. On a recent RHEL 10 nightly - where reuse is enabled, but didn't work on several variants - it took less than 2 minutes. The whole gather phase takes substantially longer on Fedora, but the time is not taken in gather_packages; it's taken in link_files, which has no reuse path (and I don't think could have one). That takes nearly half an hour on Fedora. On RHEL it's very fast, though, because there are fewer packages. Still, I suspect if we disabled gather phase reuse on RHEL we'd barely notice any difference; it might be worth testing?

buildinstall phase takes half an hour on Fedora, about 15 minutes on a recent RHEL where it looks like reuse failed for all four arches it was run on. Is the complexity really worth occasionally saving 15 minutes?

createiso, likewise, seems to be really fast whether it reuses results or not. All it really does is run xorriso or mkisofs; on Fedora the phase takes four minutes to do four Server images, on RHEL it seems to take like 80 seconds at worst (RHEL 9 nightly where no reuse worked) - I guess RHEL has faster storage and/or the images are smaller? Anyhow, unless we can find a slower case, is the complexity of the reuse logic really worth it?

extra_isos takes about 8 minutes on RHEL 9. reuse cuts it to less than a minute. OK, we save ~7 minutes. Maybe worth it?

Figured it's worth talking about, anyway.

It's a bit tricky to figure whether the gather phase reuse actually saves any time even when it kicks in. In RHEL 9 nightlies it does kick in, but the gather process still takes a while...

2026-04-05 01:43:53 [INFO    ] [BEGIN] ---------- PHASE: GATHER ----------
...
2026-04-05 01:43:53 [INFO    ] Loading old GATHER phase results: /mnt/odcs/prod/production/RHEL-9.9.0-20260404.1/work/x86_64/gather_result/BaseOS.x86_64.result
2026-04-05 01:43:56 [INFO    ] Copying old gather log /mnt/odcs/prod/production/RHEL-9.9.0-20260404.1/work/x86_64/pungi to /mnt/odcs/prod/odcs-4622237/work/x86_64/pungi
2026-04-05 01:44:00 [INFO    ] [DONE ] Gathering packages (arch: x86_64, variant: BaseOS)
...
2026-04-05 01:47:53 [INFO    ] [DONE ] Gathering packages (arch: s390x, variant: SAP)

so reusing isn't "free", we still took ~7 seconds in that example case to determine that reuse was possible and actually copy the files, and despite extensive reuse, the whole 'gathering packages' process still took 4 minutes. Would it actually have been any slower if we just disabled reuse? On that compose, link_files took 10 minutes, so the non-reusable-portion of the process is still taking the lion's share of the time, too...
Edited by adamwill

The biggest win is in pkgset phase (especially around modules). The other pieces are admittedly not that great.

The original request that caused inclusion of this code was to speed up composes in final part of development lifecycle, where there can be respins with very few changes and thus the reuse had potential to save a bit of effort and load on build systems. However, even in this situation it's not a silver bullet. The respin would very frequently include core system packages, which are included in most images and thus invalidate the reuse.

I'd be in favour of removing reuse from gather, buildinstall and iso related phases. Those rarely help and the complexity is indeed very high.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/pungi/pungi/issues/1903

Please continue any further discussion there.
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.