pungi.global.log
[INFO ] [FAIL] Buildinstall (variant Everything, arch s390x) failed, but going on anyway. [INFO ] Runroot task failed: 28583042. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/s390x/buildinstall-Everything.s390x.log for more details.
[INFO ] [FAIL] Buildinstall (variant Server, arch s390x) failed, but going on anyway. [INFO ] Runroot task failed: 28583043. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/s390x/buildinstall-Server.s390x.log for more details.
[INFO ] [FAIL] Ostree (variant AtomicWorkstation, arch x86_64) failed, but going on anyway. [INFO ] Runroot task failed: 28585189. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/x86_64/AtomicWorkstation/ostree-4/runroot.log for more details.
[LIVE_IMAGES ] [INFO ] [FAIL] Live (variant Server, arch armhfp, subvariant Server) failed, but going on anyway. [LIVE_IMAGES ] [INFO ] LiveImage task failed: 28586027. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/armhfp/liveimage-Fedora-Server-armhfp-Rawhide-20180725.n.1.armhfp.log for more details.
[LIVE_IMAGES ] [INFO ] [FAIL] Live (variant Workstation, arch armhfp, subvariant Workstation) failed, but going on anyway. [LIVE_IMAGES ] [INFO ] LiveImage task failed: 28586091. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/armhfp/liveimage-Fedora-Workstation-armhfp-Rawhide-20180725.n.1.armhfp.log for more details.
[LIVE_MEDIA ] [INFO ] [FAIL] Live media (variant Labs, arch *, subvariant Jam_KDE) failed, but going on anyway. [LIVE_MEDIA ] [INFO ] Live media task failed: 28586067. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/i386-x86_64/livemedia-Labs-Jam_KDE.i386-x86_64.log for more details.
[INFO ] [FAIL] Image build (variant Cloud, arch s390x, subvariant Cloud_Base) failed, but going on anyway. [IMAGE_BUILD ] [INFO ] Hardlinking /mnt/koji/packages/Fedora-Cloud-Base/Rawhide/20180725.n.1/images/Fedora-Cloud-Base-Rawhide-20180725.n.1.ppc64.qcow2 to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/compose/Cloud/ppc64/images/Fedora-Cloud-Base-Rawhide-20180725.n.1.ppc64.qcow2
[INFO ] [FAIL] Live media (variant Labs, arch x86_64, subvariant Scientific_KDE) failed, but going on anyway. [LIVE_MEDIA ] [INFO ] Hardlinking /mnt/koji/packages/Fedora-Scientific_KDE-Live/Rawhide/20180725.n.1/images/Fedora-Scientific_KDE-Live-i386-Rawhide-20180725.n.1.iso to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/compose/Labs/i386/iso/Fedora-Scientific_KDE-Live-i386-Rawhide-20180725.n.1.iso
[IMAGE_BUILD ] [INFO ] [FAIL] Image build (variant Workstation, arch *, subvariant Workstation) failed, but going on anyway. [IMAGE_BUILD ] [INFO ] ImageBuild task failed: 28586104. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/aarch64/imagebuild-Workstation-Workstation-raw-xz.aarch64.log for more details.
[INFO ] [FAIL] Image build (variant Labs, arch i386, subvariant Python_Classroom) failed, but going on anyway. [IMAGE_BUILD ] [INFO ] Hardlinking /mnt/koji/packages/Fedora-Python-Classroom-Vagrant/Rawhide/20180725.n.1/images/Fedora-Python-Classroom-Vagrant-Rawhide-20180725.n.1.x86_64.vagrant-virtualbox.box to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/compose/Labs/x86_64/images/Fedora-Python-Classroom-Vagrant-Rawhide-20180725.n.1.x86_64.vagrant-virtualbox.box
[IMAGE_BUILD ] [INFO ] [FAIL] Image build (variant Server, arch *, subvariant Server) failed, but going on anyway. [IMAGE_BUILD ] [INFO ] ImageBuild task failed: 28586090. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/aarch64/imagebuild-Server-Server-raw-xz.aarch64.log for more details.
[INFO ] [FAIL] Image build (variant Labs, arch i386, subvariant Scientific) failed, but going on anyway. [IMAGE_BUILD ] [INFO ] Hardlinking /mnt/koji/packages/Fedora-Scientific-Vagrant/Rawhide/20180725.n.1/images/Fedora-Scientific-Vagrant-Rawhide-20180725.n.1.x86_64.vagrant-virtualbox.box to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/compose/Labs/x86_64/images/Fedora-Scientific-Vagrant-Rawhide-20180725.n.1.x86_64.vagrant-virtualbox.box
[INFO ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Minimal_Base) failed, but going on anyway. [IMAGE_BUILD ] [INFO ] Hardlinking /mnt/koji/packages/Fedora-Container-Minimal-Base/Rawhide/20180725.n.1/images/Fedora-Container-Minimal-Base-Rawhide-20180725.n.1.armhfp.tar.xz to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/compose/Container/armhfp/images/Fedora-Container-Minimal-Base-Rawhide-20180725.n.1.armhfp.tar.xz
[INFO ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Base) failed, but going on anyway. [INFO ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway.
[INFO ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway. [IMAGE_BUILD ] [INFO ] Hardlinking /mnt/koji/packages/Fedora-Container-Base/Rawhide/20180725.n.1/images/Fedora-Container-Base-Rawhide-20180725.n.1.aarch64.tar.xz to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/compose/Container/aarch64/images/Fedora-Container-Base-Rawhide-20180725.n.1.aarch64.tar.xz
[ERROR ] Compose run failed: Live media task failed: 28586097. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/i386-x86_64/livemedia-Spins-KDE.i386-x86_64.log for more details.
So far, @kevin has tracked this down to qt5-qtwebkit suddenly becoming huge in qt5-qtwebkit-5.212.0-0.26.alpha2.fc29 , which appears to be because the main library was missed by find-debuginfo.sh and did not have its debuginfo stripped, so the library file (and thus the main package) is huge (3.2GB on aarch64 and x86_64!) and the debuginfo package is tiny.
We do not know why this happened, yet, and are trying to figure it out. I did come across a very similar case a few days earlier, in llvm6.0: in the llvm6.0-6.0.1-3.fc29 build, similarly, stripping of libLLVM-6.0.so was skipped. In a straight up rebuild, llvm6.0-6.0.1-4.fc29, stripping of the same library succeeded. However, that build happened on 2018-07-23, before the problematic qt5-qtwebkit build, so we're not just dealing with some bug that was fixed between 2018-07-19 and 2018-07-23 here.
libLLVM-6.0.so
Basically we've got what looks like the same problem, missed stripping of a library, happening in builds that ran on 2018-07-19 and 2018-07-24...but not happening in a build that ran on 2018-07-23. So there definitely seems to be something unpredictable about the bug.
We haven't yet looked to see if this is a wider problem, it could well be the case that some other builds have run into this, but no-one noticed it yet.
See https://bugzilla.redhat.com/show_bug.cgi?id=1608549 for more.