dusty / failed-composes

#1534 Fedora-Rawhide-20190228.n.0 DOOMED
Opened by dustymabe. Modified

Open
Close Issue

pungi.global.log

[ERROR   ] [FAIL] Ostree (variant Silverblue, arch x86_64) failed, but going on anyway.
[ERROR   ] Runroot task failed: 33097085. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/x86_64/Silverblue/ostree-1/runroot.log for more details.

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Silverblue, arch x86_64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 33097098. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/x86_64/Silverblue/ostree_installer-1/runroot.log for more details.

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Spins, arch *, subvariant Minimal) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 33099721. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/aarch64/imagebuild-Spins-Minimal-raw-xz.aarch64.log for more details.
  • No Task ID, look at log statement
[CREATEISO       ] [ERROR   ] [FAIL] Iso (variant Server, arch ppc64le) failed, but going on anyway.
[CREATEISO       ] [ERROR   ] Runroot task failed: None. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/ppc64le/createiso-Fedora-Server-dvd-ppc64le-Rawhide-20190228.n.0.iso.ppc64le.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Server, arch armhfp, subvariant Server) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33099684. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/armhfp/liveimage-Fedora-Server-armhfp-Rawhide-20190228.n.0.armhfp.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Workstation, arch armhfp, subvariant Workstation) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33099716. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/armhfp/liveimage-Fedora-Workstation-armhfp-Rawhide-20190228.n.0.armhfp.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Astronomy_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33099685. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/livemedia-Labs-Astronomy_KDE.i386-x86_64.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Labs, arch armhfp, subvariant Python_Classroom) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33099682. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/armhfp/liveimage-Fedora-Python-Classroom-armhfp-Rawhide-20190228.n.0.armhfp.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant Mate) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33099698. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/armhfp/liveimage-Fedora-Mate-armhfp-Rawhide-20190228.n.0.armhfp.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Design_suite) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33099687. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/livemedia-Labs-Design_suite.i386-x86_64.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant KDE) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33099686. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/armhfp/liveimage-Fedora-KDE-armhfp-Rawhide-20190228.n.0.armhfp.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Scientific_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33099690. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/livemedia-Labs-Scientific_KDE.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Games) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33099699. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/livemedia-Labs-Games.i386-x86_64.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant LXDE) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33099689. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/armhfp/liveimage-Fedora-LXDE-armhfp-Rawhide-20190228.n.0.armhfp.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Security) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33099703. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/livemedia-Labs-Security.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Jam_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33099710. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/livemedia-Labs-Jam_KDE.i386-x86_64.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant SoaS) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33099705. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/armhfp/liveimage-Fedora-SoaS-armhfp-Rawhide-20190228.n.0.armhfp.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant LXQt) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33099714. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/armhfp/liveimage-Fedora-LXQt-armhfp-Rawhide-20190228.n.0.armhfp.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Robotics) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33099712. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/livemedia-Labs-Robotics.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Python_Classroom) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33099715. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/livemedia-Labs-Python_Classroom.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Xfce) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33099722. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/livemedia-Spins-Xfce.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant SoaS) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33099730. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/livemedia-Spins-SoaS.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Cinnamon) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33099736. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/livemedia-Spins-Cinnamon.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant LXDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33099737. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/livemedia-Spins-LXDE.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Mate) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33099741. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/livemedia-Spins-Mate.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant LXQt) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33099744. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/livemedia-Spins-LXQt.i386-x86_64.log for more details.

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Cloud, arch *, subvariant Cloud_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 33099696. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/x86_64/imagebuild-Cloud-Cloud_Base-vagrant-libvirt-vagrant-virtualbox.x86_64.log for more details.

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Workstation, arch *, subvariant Workstation) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 33099723. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/aarch64/imagebuild-Workstation-Workstation-raw-xz.aarch64.log for more details.

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Labs, arch *, subvariant Python_Classroom) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 33099708. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/imagebuild-Labs-Python_Classroom-vagrant-libvirt-vagrant-virtualbox.i386-x86_64.log for more details.

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Labs, arch *, subvariant Scientific) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 33099713. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/i386-x86_64/imagebuild-Labs-Scientific-vagrant-libvirt-vagrant-virtualbox.i386-x86_64.log for more details.

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Server, arch *, subvariant Server) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 33099717. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/aarch64/imagebuild-Server-Server-raw-xz.aarch64.log for more details.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch aarch64, subvariant Container_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch aarch64, subvariant Container_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Container-Base/Rawhide/20190228.n.0/images/Fedora-Container-Base-Rawhide-20190228.n.0.x86_64.tar.xz to /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/compose/Container/x86_64/images/Fedora-Container-Base-Rawhide-20190228.n.0.x86_64.tar.xz
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Minimal_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Minimal_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Minimal_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Minimal_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Minimal_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Container-Minimal-Base/Rawhide/20190228.n.0/images/Fedora-Container-Minimal-Base-Rawhide-20190228.n.0.x86_64.tar.xz to /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/compose/Container/x86_64/images/Fedora-Container-Minimal-Base-Rawhide-20190228.n.0.x86_64.tar.xz
  • Compose run failed because: - 33099701
[ERROR   ] Compose run failed: LiveImage task failed: 33099701. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190228.n.0/logs/armhfp/liveimage-Fedora-Minimal-armhfp-Rawhide-20190228.n.0.armhfp.log for more details.

This time it's connection timeout errors from from zchunk code. It doesn't look like downgrading createrepo_c on the builders did anything here because it's still trying to access comps zchunk file (our theory yesterday was that downgrading createrepo_c would cause the zchunk to not get created as part of the repodata, which would then work around zchunk download failures).

DEBUG util.py:490:  BUILDSTDERR: repo: downloading from remote: koji-override-0
DEBUG util.py:490:  BUILDSTDERR: error: Curl error (28): Timeout was reached for https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190228.n.0/compose/Everything/armhfp/os/repodata/8622dd5223370d24d64c505f6744d2cedbf69c0c3a6924fe09c8e58b10161d48-comps-Everything.armhfp.xml.zck [SSL connection timeout] (https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190228.n.0/compose/Everything/armhfp/os/repodata/8622dd5223370d24d64c505f6744d2cedbf69c0c3a6924fe09c8e58b10161d48-comps-Everything.armhfp.xml.zck).
DEBUG util.py:490:  BUILDSTDERR: error: Zchunk header checksum didn't match expected checksum (https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190228.n.0/compose/Everything/armhfp/os/repodata/8622dd5223370d24d64c505f6744d2cedbf69c0c3a6924fe09c8e58b10161d48-comps-Everything.armhfp.xml.zck).
DEBUG util.py:490:  BUILDSTDERR: koji-override-0                                 730 kB/s |  67 MB     01:33    
DEBUG util.py:490:  BUILDSTDERR: Cannot download 'http://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190228.n.0/compose/Everything/armhfp/os': Yum repo downloading error: Downloading error(s): repodata/8622dd5223370d24d64c505f6744d2cedbf69c0c3a6924fe09c8e58b10161d48-comps-Everything.armhfp.xml.zck - Cannot download, all mirrors were already tried without success.
DEBUG util.py:490:  BUILDSTDERR: Failed to synchronize cache for repo 'koji-override-0'

I think its the timing, dont how but the rawhide-composer got updated and got the latest createrepo_c at Thu 28 Feb 2019 01:12:48 AM UTC.

$ dnf history info 56
Transaction ID : 56
Begin time : Thu 28 Feb 2019 01:12:48 AM UTC Begin rpmdb : 954:84bf6cf2fe30515d80792a82a7e156b940260615 End time : Thu 28 Feb 2019 01:13:40 AM UTC (52 seconds)
End rpmdb : 954:9b58ef2f44f2a843e8a64192e7996671e675f6df
User : System
Return-Code : Success
Releasever : 29
Command Line :
Packages Altered:
Install kernel-4.20.11-200.fc29.x86_64 @updates
Install kernel-core-4.20.11-200.fc29.x86_64 @updates
Install kernel-modules-4.20.11-200.fc29.x86_64 @updates
Upgrade bind-export-libs-32:9.11.5-4.P4.fc29.x86_64 @updates
Upgraded bind-export-libs-32:9.11.4-13.P2.fc29.x86_64 @@System
Upgrade bind-libs-32:9.11.5-4.P4.fc29.x86_64 @updates
Upgraded bind-libs-32:9.11.4-13.P2.fc29.x86_64 @@System
Upgrade bind-libs-lite-32:9.11.5-4.P4.fc29.x86_64 @updates
Upgraded bind-libs-lite-32:9.11.4-13.P2.fc29.x86_64 @@System
Upgrade bind-license-32:9.11.5-4.P4.fc29.noarch @updates
Upgraded bind-license-32:9.11.4-13.P2.fc29.noarch @@System
Upgrade bind-utils-32:9.11.5-4.P4.fc29.x86_64 @updates
Upgraded bind-utils-32:9.11.4-13.P2.fc29.x86_64 @@System
Upgrade createrepo_c-0.12.1-1.fc29.x86_64 @updates
Upgraded createrepo_c-0.12.0-1.fc29.1.x86_64 @@System
Upgrade createrepo_c-libs-0.12.1-1.fc29.x86_64 @updates
Upgraded createrepo_c-libs-0.12.0-1.fc29.1.x86_64 @@System
Upgrade dhcp-client-12:4.3.6-29.fc29.x86_64 @updates
Upgraded dhcp-client-12:4.3.6-28.fc29.x86_64 @@System
Upgrade dhcp-common-12:4.3.6-29.fc29.noarch @updates
Upgraded dhcp-common-12:4.3.6-28.fc29.noarch @@System
Upgrade dhcp-libs-12:4.3.6-29.fc29.x86_64 @updates
Upgraded dhcp-libs-12:4.3.6-28.fc29.x86_64 @@System
Upgrade python3-bind-32:9.11.5-4.P4.fc29.noarch @updates
Upgraded python3-bind-32:9.11.4-13.P2.fc29.noarch @@System
Upgrade python3-createrepo_c-0.12.1-1.fc29.x86_64 @updates
Upgraded python3-createrepo_c-0.12.0-1.fc29.1.x86_64 @@System
Removed kernel-4.20.4-200.fc29.x86_64 @@System
Removed kernel-core-4.20.4-200.fc29.x86_64 @@System
Removed kernel-modules-4.20.4-200.fc29.x86_64 @@System

But the zck file got created at Feb 28 04:17

-rw-r--r--. 1 root root 456848 Feb 28 04:17 8622dd5223370d24d64c505f6744d2cedbf69c0c3a6924fe09c8e58b10161d48-comps-Everything.armhfp.xml.zck

I want to try it again with downgrading createrepo_c but I want to know if its a planned upgrade or some cron job running somewhere, so that we can disable it. I couldn't find any cron jobs, but just want to make sure before I rerun it again.
Edited by mohanboddu
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.