We have recently discovered quite a few problems when building Pythons, after Koji builders got an updated kernel. Could we please also have the kernel updated on Test Machine Resources For Package Maintainers for easier reproducers?
Namely, I've tried to dnf update kernel and reboot aarch64-test02.fedorainfracloud.org, but the kernel stayed on 5.4.18-200.fc31.aarch64.
dnf update kernel
aarch64-test02.fedorainfracloud.org
Thanks
Cc @vstinner
Metadata Update from @smooge: - Issue assigned to kevin - Issue priority set to: Waiting on Assignee (was: Needs Review)
Note that builders were updated, but havent been rebooted yet... so they are on a mix of 5.4 and 5.3 kernels...
What kernel do you need? Are you sure the kernel is causing the problems?
Not sure what version we need exactly but yes, the problems we see are kernel related. Having "latest" would work fine for our use case.
https://bugzilla.redhat.com/show_bug.cgi?id=1797052#c31 https://bugzilla.redhat.com/show_bug.cgi?id=1795576#c5 (this contains some kernel version info)
Here's the kernel version on all the builders:
bkernel03.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.18-200.fc31.x86_64 bkernel04.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.18-200.fc31.x86_64 buildhw-01.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildhw-02.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildhw-03.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildhw-04.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildhw-05.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildhw-06.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildhw-07.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildhw-08.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildhw-09.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildhw-10.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildhw-11.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildhw-12.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildhw-aarch64-01.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildhw-aarch64-02.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildhw-aarch64-03.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildhw-aarch64-04.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildhw-aarch64-05.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildhw-aarch64-06.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildhw-aarch64-07.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildhw-aarch64-08.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-01.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-02.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-03.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-04.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-05.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-06.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.12-200.fc31.x86_64 buildvm-07.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-08.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-09.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-10.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-11.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-12.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-13.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-14.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-15.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-16.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-17.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-18.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-19.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-20.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-21.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-22.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-23.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-24.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-25.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-26.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-27.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-28.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-29.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-30.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-31.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-32.phx2.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.x86_64 buildvm-aarch64-01.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.12-200.fc31.aarch64 buildvm-aarch64-02.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.12-200.fc31.aarch64 buildvm-aarch64-03.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.12-200.fc31.aarch64 buildvm-aarch64-04.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-05.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-06.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-07.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-08.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-09.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-10.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-11.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-12.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-13.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-14.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-15.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-16.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-17.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-18.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-19.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-20.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-21.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-22.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.18-200.fc31.aarch64 buildvm-aarch64-23.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.18-200.fc31.aarch64 buildvm-aarch64-24.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-aarch64-25.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.aarch64 buildvm-armv7-01.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.12-300.fc31.armv7hl+lpae buildvm-armv7-02.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.12-300.fc31.armv7hl+lpae buildvm-armv7-03.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.12-300.fc31.armv7hl+lpae buildvm-armv7-04.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.12-300.fc31.armv7hl+lpae buildvm-armv7-05.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.12-300.fc31.armv7hl+lpae buildvm-armv7-06.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.12-300.fc31.armv7hl+lpae buildvm-armv7-07.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-08.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-09.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-10.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-11.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-12.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-13.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-14.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-15.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-16.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-17.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-18.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-19.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-20.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-21.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.3.13-300.fc31.armv7hl+lpae buildvm-armv7-22.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.18-200.fc31.armv7hl+lpae buildvm-armv7-23.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.18-200.fc31.armv7hl+lpae buildvm-armv7-24.arm.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.18-200.fc31.armv7hl+lpae buildvm-ppc64le-01.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-02.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-03.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-04.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-05.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-06.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-07.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-08.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-09.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-10.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.10-200.fc31.ppc64le buildvm-ppc64le-11.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.10-200.fc31.ppc64le buildvm-ppc64le-12.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.10-200.fc31.ppc64le buildvm-ppc64le-13.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.10-200.fc31.ppc64le buildvm-ppc64le-14.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.10-200.fc31.ppc64le buildvm-ppc64le-15.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.10-200.fc31.ppc64le buildvm-ppc64le-16.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.10-200.fc31.ppc64le buildvm-ppc64le-17.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.10-200.fc31.ppc64le buildvm-ppc64le-18.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.10-200.fc31.ppc64le buildvm-ppc64le-19.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.10-200.fc31.ppc64le buildvm-ppc64le-20.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-21.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-22.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-23.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-24.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-25.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-26.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-27.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-28.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-29.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-30.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-ppc64le-31.ppc.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.ppc64le buildvm-s390x-01.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.12-200.fc31.s390x buildvm-s390x-02.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-03.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-04.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-05.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-06.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-07.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-08.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.12-200.fc31.s390x buildvm-s390x-09.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-10.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.12-200.fc31.s390x buildvm-s390x-11.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.12-200.fc31.s390x buildvm-s390x-12.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-13.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-14.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.12-200.fc31.s390x buildvm-s390x-15.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-16.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-17.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-18.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-19.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-20.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.12-200.fc31.s390x buildvm-s390x-21.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-22.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-23.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.8-200.fc31.s390x buildvm-s390x-24.s390.fedoraproject.org | CHANGED | rc=0 | (stdout) 5.4.12-200.fc31.s390x
I have moved the aarch64-test instances to aws and they both now have:
5.4.19-200.fc31.aarch64
Is that sufficent for your needs?
For our immediate needs, yes it is. For the general "get a reproducer" case, updating all of them would IMHO be beneficial, but I am OK to open a new ticket once we need a specific one. Thanks
Metadata Update from @churchyard: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)