#577 Fedora-Rawhide-20180725.n.1 DOOMED
Opened by dustymabe. Modified

pungi.global.log

[INFO    ] [FAIL] Buildinstall (variant Everything, arch s390x) failed, but going on anyway.
[INFO    ] Runroot task failed: 28583042. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/s390x/buildinstall-Everything.s390x.log for more details.
[INFO    ] [FAIL] Buildinstall (variant Server, arch s390x) failed, but going on anyway.
[INFO    ] Runroot task failed: 28583043. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/s390x/buildinstall-Server.s390x.log for more details.
[INFO    ] [FAIL] Ostree (variant AtomicWorkstation, arch x86_64) failed, but going on anyway.
[INFO    ] Runroot task failed: 28585189. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/x86_64/AtomicWorkstation/ostree-4/runroot.log for more details.
[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Server, arch armhfp, subvariant Server) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 28586027. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/armhfp/liveimage-Fedora-Server-armhfp-Rawhide-20180725.n.1.armhfp.log for more details.
[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Workstation, arch armhfp, subvariant Workstation) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 28586091. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/armhfp/liveimage-Fedora-Workstation-armhfp-Rawhide-20180725.n.1.armhfp.log for more details.
[LIVE_MEDIA      ] [INFO    ] [FAIL] Live media (variant Labs, arch *, subvariant Jam_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [INFO    ] Live media task failed: 28586067. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/i386-x86_64/livemedia-Labs-Jam_KDE.i386-x86_64.log for more details.
  • No Task ID, look at log statement
[INFO    ] [FAIL] Image build (variant Cloud, arch s390x, subvariant Cloud_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Cloud-Base/Rawhide/20180725.n.1/images/Fedora-Cloud-Base-Rawhide-20180725.n.1.ppc64.qcow2 to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/compose/Cloud/ppc64/images/Fedora-Cloud-Base-Rawhide-20180725.n.1.ppc64.qcow2
  • No Task ID, look at log statement
[INFO    ] [FAIL] Live media (variant Labs, arch x86_64, subvariant Scientific_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Scientific_KDE-Live/Rawhide/20180725.n.1/images/Fedora-Scientific_KDE-Live-i386-Rawhide-20180725.n.1.iso to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/compose/Labs/i386/iso/Fedora-Scientific_KDE-Live-i386-Rawhide-20180725.n.1.iso
[IMAGE_BUILD     ] [INFO    ] [FAIL] Image build (variant Workstation, arch *, subvariant Workstation) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] ImageBuild task failed: 28586104. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/aarch64/imagebuild-Workstation-Workstation-raw-xz.aarch64.log for more details.
  • No Task ID, look at log statement
[INFO    ] [FAIL] Image build (variant Labs, arch i386, subvariant Python_Classroom) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Python-Classroom-Vagrant/Rawhide/20180725.n.1/images/Fedora-Python-Classroom-Vagrant-Rawhide-20180725.n.1.x86_64.vagrant-virtualbox.box to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/compose/Labs/x86_64/images/Fedora-Python-Classroom-Vagrant-Rawhide-20180725.n.1.x86_64.vagrant-virtualbox.box
[IMAGE_BUILD     ] [INFO    ] [FAIL] Image build (variant Server, arch *, subvariant Server) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] ImageBuild task failed: 28586090. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/aarch64/imagebuild-Server-Server-raw-xz.aarch64.log for more details.
  • No Task ID, look at log statement
[INFO    ] [FAIL] Image build (variant Labs, arch i386, subvariant Scientific) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Scientific-Vagrant/Rawhide/20180725.n.1/images/Fedora-Scientific-Vagrant-Rawhide-20180725.n.1.x86_64.vagrant-virtualbox.box to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/compose/Labs/x86_64/images/Fedora-Scientific-Vagrant-Rawhide-20180725.n.1.x86_64.vagrant-virtualbox.box
  • No Task ID, look at log statement
[INFO    ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Minimal_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Container-Minimal-Base/Rawhide/20180725.n.1/images/Fedora-Container-Minimal-Base-Rawhide-20180725.n.1.armhfp.tar.xz to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/compose/Container/armhfp/images/Fedora-Container-Minimal-Base-Rawhide-20180725.n.1.armhfp.tar.xz
  • No Task ID, look at log statement
[INFO    ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Base) failed, but going on anyway.
[INFO    ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[INFO    ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Container-Base/Rawhide/20180725.n.1/images/Fedora-Container-Base-Rawhide-20180725.n.1.aarch64.tar.xz to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/compose/Container/aarch64/images/Fedora-Container-Base-Rawhide-20180725.n.1.aarch64.tar.xz
  • Compose run failed because: - 28586097
[ERROR   ] Compose run failed: Live media task failed: 28586097. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180725.n.1/logs/i386-x86_64/livemedia-Spins-KDE.i386-x86_64.log for more details.

So far, @kevin has tracked this down to qt5-qtwebkit suddenly becoming huge in qt5-qtwebkit-5.212.0-0.26.alpha2.fc29 , which appears to be because the main library was missed by find-debuginfo.sh and did not have its debuginfo stripped, so the library file (and thus the main package) is huge (3.2GB on aarch64 and x86_64!) and the debuginfo package is tiny.

We do not know why this happened, yet, and are trying to figure it out. I did come across a very similar case a few days earlier, in llvm6.0: in the llvm6.0-6.0.1-3.fc29 build, similarly, stripping of libLLVM-6.0.so was skipped. In a straight up rebuild, llvm6.0-6.0.1-4.fc29, stripping of the same library succeeded. However, that build happened on 2018-07-23, before the problematic qt5-qtwebkit build, so we're not just dealing with some bug that was fixed between 2018-07-19 and 2018-07-23 here.

Basically we've got what looks like the same problem, missed stripping of a library, happening in builds that ran on 2018-07-19 and 2018-07-24...but not happening in a build that ran on 2018-07-23. So there definitely seems to be something unpredictable about the bug.

We haven't yet looked to see if this is a wider problem, it could well be the case that some other builds have run into this, but no-one noticed it yet.

See https://bugzilla.redhat.com/show_bug.cgi?id=1608549 for more.

Metadata