dusty / failed-composes

#1505 Fedora-Rawhide-20190225.n.0 DOOMED
Opened by dustymabe. Modified

Open
Close Issue

pungi.global.log

  • No Task ID, look at log statement
[CREATEISO       ] [ERROR   ] [FAIL] Iso (variant Server, arch ppc64le) failed, but going on anyway.
[CREATEISO       ] [ERROR   ] Runroot task failed: None. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/ppc64le/createiso-Fedora-Server-dvd-ppc64le-Rawhide-20190225.n.0.iso.ppc64le.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Server, arch armhfp, subvariant Server) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33030460. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/armhfp/liveimage-Fedora-Server-armhfp-Rawhide-20190225.n.0.armhfp.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant LXDE) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33030469. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/armhfp/liveimage-Fedora-LXDE-armhfp-Rawhide-20190225.n.0.armhfp.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Labs, arch armhfp, subvariant Python_Classroom) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 33030459. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/armhfp/liveimage-Fedora-Python-Classroom-armhfp-Rawhide-20190225.n.0.armhfp.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Design_suite) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33030472. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/i386-x86_64/livemedia-Labs-Design_suite.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Astronomy_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33030468. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/i386-x86_64/livemedia-Labs-Astronomy_KDE.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Scientific_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33030476. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/i386-x86_64/livemedia-Labs-Scientific_KDE.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Games) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33030480. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/i386-x86_64/livemedia-Labs-Games.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Jam_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33030492. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/i386-x86_64/livemedia-Labs-Jam_KDE.i386-x86_64.log for more details.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Container-Base/Rawhide/20190225.n.0/images/Fedora-Container-Base-Rawhide-20190225.n.0.x86_64.tar.xz to /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/compose/Container/x86_64/images/Fedora-Container-Base-Rawhide-20190225.n.0.x86_64.tar.xz

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Robotics) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33030496. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/i386-x86_64/livemedia-Labs-Robotics.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Python_Classroom) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 33030504. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/i386-x86_64/livemedia-Labs-Python_Classroom.i386-x86_64.log for more details.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Cloud, arch s390x, subvariant Cloud_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Cloud, arch ppc64le, subvariant Cloud_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Cloud, arch ppc64le, subvariant Cloud_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Cloud-Base/Rawhide/20190225.n.0/images/Fedora-Cloud-Base-Rawhide-20190225.n.0.aarch64.qcow2 to /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/compose/Cloud/aarch64/images/Fedora-Cloud-Base-Rawhide-20190225.n.0.aarch64.qcow2

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Labs, arch *, subvariant Scientific) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 33030486. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/i386-x86_64/imagebuild-Labs-Scientific-vagrant-libvirt-vagrant-virtualbox.i386-x86_64.log for more details.

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Server, arch *, subvariant Server) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 33030491. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/aarch64/imagebuild-Server-Server-raw-xz.aarch64.log for more details.

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Labs, arch *, subvariant Python_Classroom) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 33030478. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/i386-x86_64/imagebuild-Labs-Python_Classroom-vagrant-libvirt-vagrant-virtualbox.i386-x86_64.log for more details.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Minimal_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Minimal_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Minimal_Base) failed, but going on anyway.
[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Minimal_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Minimal_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Container-Minimal-Base/Rawhide/20190225.n.0/images/Fedora-Container-Minimal-Base-Rawhide-20190225.n.0.aarch64.tar.xz to /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/compose/Container/aarch64/images/Fedora-Container-Minimal-Base-Rawhide-20190225.n.0.aarch64.tar.xz
  • Compose run failed because: - 33030477
[ERROR   ] Compose run failed: LiveImage task failed: 33030477. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190225.n.0/logs/armhfp/liveimage-Fedora-Minimal-armhfp-Rawhide-20190225.n.0.armhfp.log for more details.

Looks like some kind of networking issue? https://koji.fedoraproject.org/koji/taskinfo?taskID=33030477 has this:

DEBUG util.py:490:  BUILDSTDERR: repo: downloading from remote: koji-override-0
DEBUG util.py:490:  BUILDSTDERR: error: Curl error (16): Error in the HTTP2 framing layer for https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190225.n.0/compose/Everything/armhfp/os/repodata/680a24dafe44150232912d9d57a2c43f1465980457a389b628a5923ca82a0e37-primary.xml.zck [] (https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190225.n.0/compose/Everything/armhfp/os/repodata/680a24dafe44150232912d9d57a2c43f1465980457a389b628a5923ca82a0e37-primary.xml.zck).
DEBUG util.py:490:  BUILDSTDERR: koji-override-0                                 678 kB/s |  40 MB     01:00    
DEBUG util.py:490:  BUILDSTDERR: Cannot download 'http://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190225.n.0/compose/Everything/armhfp/os': Yum repo downloading error: Downloading error(s): repodata/680a24dafe44150232912d9d57a2c43f1465980457a389b628a5923ca82a0e37-primary.xml.zck - Cannot download, all mirrors were already tried without success.
DEBUG util.py:490:  BUILDSTDERR: Failed to synchronize cache for repo 'koji-override-0'

well, perhaps, but it's hit that 2 days in a row. That exact same call. ;(

I note this is .zck, I wonder if we should not turn off zchunk somehow for this?

@jdieter Any ideas?

@kalev I ran a scratch build taking the inputs from this failed task -https://koji.fedoraproject.org/koji/taskinfo?taskID=33036719 and it worked. So, I am guessing its a network error.

This is not a networking error, but a bug in Curl's HTTP/2 handling.
If it were a networking issue, it would have caught failures in either the TCP or TLS layers, not all the way up to the HTTP/2 layer: error: Curl error (16): Error in the HTTP2 framing layer for.

I have a guess that this has been caused by connection reuse, which I'm currently trying to make no longer an issue by just trying to enable h2 on all proxies.

@mohanboddu it is sort of random which proxy was being hit, and half of them had h2 enabled, half had them disabled.
If you retry the job now, it should always hit a proxy with h2, and we can see whether it still hits it.

@puiterwijk Resubmiited - https://koji.fedoraproject.org/koji/taskinfo?taskID=33038330

@puiterwijk Resubmiited - https://koji.fedoraproject.org/koji/taskinfo?taskID=33038330

It successfully completed. Should I go ahead and kick off another rawhide compose?

Sure, let's see if this helps.

@kalev, sorry, just got off work and saw this. Is everything good now?

For future reference, if you need to disable downloading zchunk metadata, it should be sufficient to set zchunk=False in the dnf config file.

@jdieter I think we are good. Scratch build worked fine.

For future reference, if you need to disable downloading zchunk metadata, it should be sufficient to set zchunk=False in the dnf config file.

Thanks for the info.
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.