dusty / failed-composes

#1179 Fedora-Rawhide-20181219.n.0 DOOMED
Opened by dustymabe. Modified

Open
Close Issue

pungi.global.log

  • No Task ID, look at log statement
[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant AtomicHost, arch x86_64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: None. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/x86_64/AtomicHost/ostree_installer-3/runroot.log for more details.
  • No Task ID, look at log statement
[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Silverblue, arch x86_64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: None. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/x86_64/Silverblue/ostree_installer-4/runroot.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Workstation, arch armhfp, subvariant Workstation) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 31531184. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/armhfp/liveimage-Fedora-Workstation-armhfp-Rawhide-20181219.n.0.armhfp.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Astronomy_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 31531146. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Labs-Astronomy_KDE.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Scientific_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 31531152. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Labs-Scientific_KDE.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Games) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 31531155. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Labs-Games.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Security) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 31531159. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Labs-Security.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Design_suite) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 31531149. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Labs-Design_suite.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Robotics) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 31531190. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Labs-Robotics.i386-x86_64.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant LXQt) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 31531161. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/armhfp/liveimage-Fedora-LXQt-armhfp-Rawhide-20181219.n.0.armhfp.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Jam_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 31531167. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Labs-Jam_KDE.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant SoaS) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 31531204. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Spins-SoaS.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Python_Classroom) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 31531196. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Labs-Python_Classroom.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant LXDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 31531207. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Spins-LXDE.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Xfce) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 31531202. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Spins-Xfce.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Cinnamon) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 31531206. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Spins-Cinnamon.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Mate) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 31531212. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Spins-Mate.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant LXQt) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 31531217. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Spins-LXQt.i386-x86_64.log for more details.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant AtomicHost, arch ppc64le, subvariant AtomicHost) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-AtomicHost/Rawhide/20181219.n.0/images/Fedora-AtomicHost-Rawhide-20181219.n.0.aarch64.qcow2 to /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/compose/AtomicHost/aarch64/images/Fedora-AtomicHost-Rawhide-20181219.n.0.aarch64.qcow2
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Cloud, arch s390x, subvariant Cloud_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Cloud, arch ppc64le, subvariant Cloud_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Cloud, arch ppc64le, subvariant Cloud_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Cloud-Base/Rawhide/20181219.n.0/images/Fedora-Cloud-Base-Rawhide-20181219.n.0.x86_64.qcow2 to /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/compose/Cloud/x86_64/images/Fedora-Cloud-Base-Rawhide-20181219.n.0.x86_64.qcow2

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Labs, arch *, subvariant Scientific) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 31531175. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/imagebuild-Labs-Scientific-vagrant-libvirt-vagrant-virtualbox.i386-x86_64.log for more details.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Container-Base/Rawhide/20181219.n.0/images/Fedora-Container-Base-Rawhide-20181219.n.0.aarch64.tar.xz to /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/compose/Container/aarch64/images/Fedora-Container-Base-Rawhide-20181219.n.0.aarch64.tar.xz
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Minimal_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Minimal_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Minimal_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Minimal_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Minimal_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Container-Minimal-Base/Rawhide/20181219.n.0/images/Fedora-Container-Minimal-Base-Rawhide-20181219.n.0.aarch64.tar.xz to /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/compose/Container/aarch64/images/Fedora-Container-Minimal-Base-Rawhide-20181219.n.0.aarch64.tar.xz
  • Compose run failed because: - 31531218
[ERROR   ] Compose run failed: Live media task failed: 31531218. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20181219.n.0/logs/i386-x86_64/livemedia-Workstation-Workstation.i386-x86_64.log for more details.

Lives are failing due to:

DEBUG util.py:490:  BUILDSTDERR: 2018-12-19 10:45:46,098: Service org.fedoraproject.Anaconda.Modules.Localization is not available..

I'll try and figure out what's going on there.

It seems like the dbus stuff is going wrong, but I'm not sure why. The anaconda.log has these errors:

10:42:10,743 ERR keyboard: Failed to get safe_dbus connection: g-io-error-quark: Could not connect: No such file or directory (1)
10:42:10,748 ERR keyboard: Failed to set keymap: Unable to connect to system bus: g-io-error-quark: Could not connect: No such file or directory (1)
10:42:10,749 ERR keyboard: Failed to set keymap: Unable to connect to system bus: g-io-error-quark: Could not connect: No such file or directory (1)

but I can't really see why that would suddenly be happening since 20181216 (the last successful compose) - neither anaconda nor dbus packages have changed in that time...

Here's the diff between packages used for the (successful) 20181216.n.1 Workstation live and the (failed) 20181218.n.0 Workstation live:

--- /tmp/1216n1 2018-12-19 09:36:36.358420455 -0800
+++ /tmp/1218n0 2018-12-19 09:37:28.136355429 -0800
@@ -9,7 +9,7 @@
 bind-export-libs-9.11.4-11.P2.fc30.x86_64
 blivet-data-1:3.1.2-1.fc30.noarch
 blivet-data-3.1.2-1.fc30.noarch
-brotli-1.0.7-1.fc30.x86_64
+brotli-1.0.7-2.fc30.x86_64
 bzip2-libs-1.0.6-28.fc29.x86_64
 ca-certificates-2018.2.26-2.fc30.noarch
 chkconfig-1.11-2.fc30.x86_64
@@ -70,10 +70,10 @@
 gdbm-libs-1.18-1.fc30.x86_64
 genisoimage-1.1.11-40.fc29.x86_64
 glib2-2.58.1-2.fc30.x86_64
-glibc-2.28.9000-27.fc30.x86_64
-glibc-all-langpacks-2.28.9000-27.fc30.x86_64
-glibc-common-2.28.9000-27.fc30.x86_64
-glibc-langpack-en-2.28.9000-27.fc30.x86_64
+glibc-2.28.9000-28.fc30.x86_64
+glibc-all-langpacks-2.28.9000-28.fc30.x86_64
+glibc-common-2.28.9000-28.fc30.x86_64
+glibc-langpack-en-2.28.9000-28.fc30.x86_64
 gmp-1:6.1.2-8.fc29.x86_64
 gmp-6.1.2-8.fc29.x86_64
 gnupg2-2.2.11-2.fc30.x86_64
@@ -100,7 +100,7 @@
 kmod-25-4.fc30.x86_64
 kmod-libs-25-4.fc30.x86_64
 kpartx-0.7.9-1.fc30.x86_64
-krb5-libs-1.17-1.beta2.1.fc30.x86_64
+krb5-libs-1.17-1.beta2.2.fc30.x86_64
 langtable-0.0.39-2.fc30.noarch
 langtable-data-0.0.39-2.fc30.noarch
 libacl-2.2.53-2.fc29.x86_64
@@ -180,9 +180,9 @@
 libxml2-2.9.8-4.fc29.x86_64
 libyaml-0.2.1-2.fc29.x86_64
 libzstd-1.3.6-1.fc30.x86_64
-lorax-30.8-1.fc30.x86_64
-lorax-lmc-novirt-30.8-1.fc30.x86_64
-lorax-templates-generic-30.8-1.fc30.x86_64
+lorax-30.9-1.fc30.x86_64
+lorax-lmc-novirt-30.9-1.fc30.x86_64
+lorax-templates-generic-30.9-1.fc30.x86_64
 lsof-4.91-2.fc29.x86_64
 lua-libs-5.3.5-2.fc29.x86_64
 lz4-1.8.2-2.fc29.x86_64
@@ -201,7 +201,7 @@
 NetworkManager-team-1:1.14.4-2.fc30.x86_64
 NetworkManager-team-1.14.4-2.fc30.x86_64
 npth-1.6-1.fc30.x86_64
-openldap-2.4.46-10.fc30.x86_64
+openldap-2.4.46-11.fc30.x86_64
 openssl-libs-1:1.1.1-7.fc30.x86_64
 openssl-libs-1.1.1-7.fc30.x86_64
 p11-kit-0.23.14-1.fc30.x86_64
@@ -256,7 +256,7 @@
 python3-pytz-2018.5-1.fc30.noarch
 python3-pyudev-0.21.0-9.fc30.noarch
 python3-requests-2.21.0-1.fc30.noarch
-python3-requests-file-1.4.3-8.fc29.noarch
+python3-requests-file-1.4.3-9.fc30.noarch
 python3-requests-ftp-0.3.1-13.fc29.noarch
 python3-rpm-4.14.2.1-3.fc30.x86_64
 python3-setuptools-40.4.3-1.fc30.noarch
@@ -275,8 +275,8 @@
 rpm-sign-libs-4.14.2.1-3.fc30.x86_64
 satyr-0.27-1.fc30.x86_64
 sed-4.5-2.fc29.x86_64
-selinux-policy-3.14.3-14.fc30.noarch
-selinux-policy-targeted-3.14.3-14.fc30.noarch
+selinux-policy-3.14.3-15.fc30.noarch
+selinux-policy-targeted-3.14.3-15.fc30.noarch
 setup-2.12.6-1.fc30.noarch
 shadow-utils-2:4.6-5.fc30.x86_64
 shadow-utils-4.6-5.fc30.x86_64
@@ -284,11 +284,11 @@
 squashfs-tools-4.3-16.fc28.x86_64
 syslinux-6.04-0.8.fc28.x86_64
 syslinux-nonlinux-6.04-0.8.fc28.noarch
-systemd-239-9.git9f3aed1.fc30.x86_64
-systemd-libs-239-9.git9f3aed1.fc30.x86_64
-systemd-pam-239-9.git9f3aed1.fc30.x86_64
-systemd-rpm-macros-239-9.git9f3aed1.fc30.noarch
-systemd-udev-239-9.git9f3aed1.fc30.x86_64
+systemd-239-10.git3bf819c.fc30.x86_64
+systemd-libs-239-10.git3bf819c.fc30.x86_64
+systemd-pam-239-10.git3bf819c.fc30.x86_64
+systemd-rpm-macros-239-10.git3bf819c.fc30.noarch
+systemd-udev-239-10.git3bf819c.fc30.x86_64
 teamd-1.28-1.fc30.x86_64
 tzdata-2018g-1.fc30.noarch
 usermode-1.112-3.fc29.x86_64

Quite short, but several plausible candidates: glibc, lorax, selinux-policy and systemd.

Interesting: the 20181216.n.1 anaconda.log actually shows the same keyboard / dbus errors as 20181218.n.0, but the compose succeeds...

Oh, just noticed that 20181218.n.0 failed a bit differently from 20181219.n.0 - it failed in post-install (though the failure still looks somehow dbus related)...

@bcl @m4rtink any ideas?
Edited by adamwill

@adamwill - thanks for posting your investigation here :)

So, @bcl suspects this might be due to the new lorax, as it has some code that uses pkill to try and clean up anaconda on exit, and we think this may be killing other composes running on the same build host - so some of the tasks fail for other reasons, then they hit pkill, then they kill all the other tasks on the same build host. Or something like that, anyway. It's a bit fuzzy, but we think it'd be a good idea to untag lorax-30.9-1.fc30 and try another compose.

Done so and fired off a new compose.

And it worked...

So it turns out 30.9 has a second bug which always causes live composes to fail near the end with 'NoneType' object is not callable - so @bcl and I figure that on any given buildhost, the first live media task it's assigned will fail that way, and then all others will be killed by the pkill from that task.

I guess he'll look at fixing this stuff up with a 30.10 after shutdown, for now we'll just roll with 30.8.
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.