I think the new proposal is OK, even though I'm not particularly happy about it.

The fact that it's hard to determine which hardware to get (and then manage to actually buy it) to do Fedora-on-aarch64 testing doesn't help. With Asahi defaulting to KDE, keeping KDE-on-aarch64 blocking sounds like a good compromise.

Yeah, the hardware question is in #3466, and I think resolving that question gives a better pathway to supporting desktop ARM across the board.

Insofar as this goes, it's okay to me.

I think we would still want to go ahead with this regardless the outcome of #3466 . That is, even if we do reduce the 'blocking' ARM hardware to a sensible small set, we don't want Workstation on ARM to be blocking if the desktop team is not interested in at least helping out with the testing. We definitely appreciate that KDE team has stepped up to help out with this.

So, if someone volunteers to run openQA on aarch64 fedora workstation it will continue to be release blocking? Is there a formal process for that?

To be honest, if we could, we'd be running OpenQA on AArch64 for Fedora KDE. We don't know how and currently lack infrastructure to do it.

So, if someone volunteers to run openQA on aarch64 fedora workstation it will continue to be release blocking? Is there a formal process for that?

There's also someone has to run triage and participate in blocker review and a few other processes.

It's not just about automated tests. Those might actually constitute a smaller part of the work. If I'm not forgetting something, the work would include:

1. Maintaining the OpenQA automated tests to pass, inspecting failures, etc
2. Running all required manual test cases regularly (desktop; base; installation - just selected tables with aarch64, not all), and performing them very fast especially for any release candidate composes
3. Reporting any bugs found during automated or manual tests
4. Debugging issues, especially those which are proposed or accepted as blockers, common issues, or other noteworthy ones. Testing fixes.
5. Participating in blocker review discussions
6. Helping to write common issues entries, when needed

I think the most time-consuming parts are 2-4.

The KDE team was doing mostly all of this during F42 cycle already (except the automation tests, which don't exist for KDE aarch64 in OpenQA at the moment - maybe we'll find a way in the future). We would need the same (and more - handling the automation as well) for Workstation aarch64, if we were to keep it release-blocking.

There is zero reason for running manual testing though, except to engage the community in one off device testing. People have been automating board farms for decades now. IP/KVM/testing setups can not only drive mouse/keyboard but also virtual USB disks plugged into USB ports, firmware, etc. Even on 'difficult' devices like phones. And AFAIK largely we really aren't interested in one off device failures. If that weren't the case I would have nominated blocker bugs for the last 3 fedora releases where KDE doesn't work properly on my x86 nvidia optimus/thunderbolt connected laptop with wayland.

This all comes down to the fact that fedora shouldn't be shipping high severity bugs. And the severity is at least partially dictated by whether or not anyone notices that bug. AKA, can't boot fedora workstation on arm, and no one notices? No problem. Someone notices F43 fails to boot because grub has a giant bug, or sddm crashes, and leaves a black screen on boot up then yes, block and get it fixed it shouldn't really matter if its server or workstation because there really isn't a difference between the two, because I can group install the workstation components off most of the install images and reproduce those kinds of bugs. At the end of the day, it is the same pixman bug crashing a large fraction of GUI applications on ashai and workstation. Which it turns out happened with F42 on aarch64.

To be honest, if we could, we'd be running OpenQA on AArch64 for Fedora KDE. We don't know how and currently lack infrastructure to do it.

I can look at enabling KDE tests in official openQA now, actually. We never did before because a) capacity and b) the old workers were super slow and flaky. In rdu3 we have a lot more capacity on much beefier and less flaky workers. (And @kevin was saying something earlier today about giving me two more of them, which would be great). I'm already looking at enabling more graphical update tests on aarch64.

There is zero reason for running manual testing though

I disagree. One, I think we should always have a human in the loop at least for smoke testing. Humans do stuff differently from robots. Two, some things are very hard to automate, like audio and printer testing. Three, see below:

People have been automating board farms for decades now.

Unfortunately none of those people are doing it on Fedora. Who is going to do it for us? We've got a couple of PiKVMs lying around the place. We have zero time to make them do anything useful. Any time we spend on that is time we have to take away from doing something else useful. So we only have automated virtual testing, which is quite different from hardware. The graphics rendering stuff goes down pretty different codepaths on a VM without 3D acceleration compared to real hardware with a 3D accelerator. And, of course, virt testing doesn't tell you anything much about all the devicetree mess on the real ARM hardware we currently claim to support/block on. So we need people to test on real hardware, and we will continue to need this as long as we want to block on desktop functionality, unless somebody gives us a couple of full-time bodies and some money to actually set up and maintain a good board farm.

This all comes down to the fact that fedora shouldn't be shipping high severity bugs. And the severity is at least partially dictated by whether or not anyone notices that bug. AKA, can't boot fedora workstation on arm, and no one notices?

The problem is the set of people who may notice changes. A lot of people only install stable releases. Just because nobody noticed the problem at random during the pre-release cycle doesn't mean nobody will notice it after release.

We had a much less structured, more haphazard release process and philosophy before. There wasn't much structured testing. There were no release criteria. Blocker decisions were made based on vibes, and blocker status was not strictly adhered to. This didn't work very well, and is why Fedora used to have a terrible reputation for quality. We now have a fairly good reputation for quality, because we have proper procedures and criteria and we respect blocker status. I don't think we want to go back to deciding blockers based on vibes.

Server vs. Workstation is significant because there is no release-blocking GUI functionality in Server. If we didn't block on Workstation or KDE we would not care about any pixman bugs.

If that weren't the case I would have nominated blocker bugs for the last 3 fedora releases where KDE doesn't work properly on my x86 nvidia optimus/thunderbolt connected laptop with wayland.

As Kamil noted somewhere, we have a very well-established process for evaluating hardware-specific bugs on x86_64, it's written down in the process docs. See https://fedoraproject.org/wiki/Blocker_Bug_FAQ#What_about_hardware_and_local_configuration_dependent_issues? .

To be honest, if we could, we'd be running OpenQA on AArch64 for Fedora KDE. We don't know how and currently lack infrastructure to do it.

I can look at enabling KDE tests in official openQA now, actually. We never did before because a) capacity and b) the old workers were super slow and flaky. In rdu3 we have a lot more capacity on much beefier and less flaky workers. (And @kevin was saying something earlier today about giving me two more of them, which would be great). I'm already looking at enabling more graphical update tests on aarch64.

That would be great, but now we're in a situation that if it is enabled, it risks Fedora QA coming back and saying we need to drop release-blocking status for KDE on AArch64. I really don't want to have this discussion again when Fedora KDE is putting in the effort to support the release-blocking status just because now Fedora OpenQA runs the KDE tests on AArch64 too.

There is zero reason for running manual testing though

I disagree. One, I think we should always have a human in the loop at least for smoke testing. Humans do stuff differently from robots. Two, some things are very hard to automate, like audio and printer testing. Three, see below:

I concur. Having been a SQA automation engineer before, I've found that it's the human testing that informs what kind of automated testing to have. The best use of automated testing (in my opinion) is to ensure regressions do not occur. And while we're not awesome at capturing fixed bugs and turning them into regression tests, we're not terrible at it either.

People have been automating board farms for decades now.

Unfortunately none of those people are doing it on Fedora. Who is going to do it for us? We've got a couple of PiKVMs lying around the place. We have zero time to make them do anything useful. Any time we spend on that is time we have to take away from doing something else useful. So we only have automated virtual testing, which is quite different from hardware. The graphics rendering stuff goes down pretty different codepaths on a VM without 3D acceleration compared to real hardware with a 3D accelerator. And, of course, virt testing doesn't tell you anything much about all the devicetree mess on the real ARM hardware we currently claim to support/block on. So we need people to test on real hardware, and we will continue to need this as long as we want to block on desktop functionality, unless somebody gives us a couple of full-time bodies and some money to actually set up and maintain a good board farm.

I don't know if I've ever heard anyone say good things about the various board automation systems (e.g. LAVA). But it is a current weakness of ours that we don't do automated hardware testing. We don't even do it for server class hardware which OpenQA supports with IPMI, much less with IPKVM equipment using the generalhw backend that OpenQA has.

I know that openSUSE does have automated real-hardware testing with OpenQA, as they have physical laptops wired into OpenQA and those tests do run on every openSUSE Tumbleweed snapshot. These tests exist in openSUSE OpenQA as ones with the suffix beginning with @laptop. They also do have RPi tests with RPi boards wired in for their AArch64 tests, but I don't know how reliable those are. They at least seem to be green most OpenQA runs, so that is something...

This all comes down to the fact that fedora shouldn't be shipping high severity bugs. And the severity is at least partially dictated by whether or not anyone notices that bug. AKA, can't boot fedora workstation on arm, and no one notices?

This is easy to say but hard to execute on. For one, we can't make anyone care about anything. Fedora KDE cares about AArch64 because it's important to us to work on ARM platforms. We are also fortunate that we have people in upstream KDE that care about ARM platforms and adjacent to us in the Asahi SIG that support us when we need to do deeper work in Fedora KDE on ARM because Fedora Asahi Remix cares about Fedora KDE for its primary variant.

The problem is the set of people who may notice changes. A lot of people only install stable releases. Just because nobody noticed the problem at random during the pre-release cycle doesn't mean nobody will notice it after release.

We had a much less structured, more haphazard release process and philosophy before. There wasn't much structured testing. There were no release criteria. Blocker decisions were made based on vibes, and blocker status was not strictly adhered to. This didn't work very well, and is why Fedora used to have a terrible reputation for quality. We now have a fairly good reputation for quality, because we have proper procedures and criteria and we respect blocker status. I don't think we want to go back to deciding blockers based on vibes.

Server vs. Workstation is significant because there is no release-blocking GUI functionality in Server. If we didn't block on Workstation or KDE we would not care about any pixman bugs.

I think this is difficult for everyone. Most teams do not have the capacity to support alternative architectures as well as x86_64. Even those that do may not necessarily want to (Fedora Workstation) or have the knowledge to do so (Fedora Server).

And some of this comes down to how much specialized knowledge is required to support various ARM platforms. You can't just not know how these different boards work and expect everything to be fine. We've historically not been able to distribute that knowledge and grow the engineering community around ARM on Fedora. While that is slowly changing now, it's still sufficiently byzantine that we need to consider how to structure support for this to be sustainable across the various Fedora teams.

That would be great, but now we're in a situation that if it is enabled, it risks Fedora QA coming back and saying we need to drop release-blocking status for KDE on AArch64. I really don't want to have this discussion again when Fedora KDE is putting in the effort to support the release-blocking status just because now Fedora OpenQA runs the KDE tests on AArch64 too.

I don't think this would happen. Honestly most of the trouble with ARM is the manual testing of crappy hardware. Running openQA tests on two arches isn't a lot harder than running them on one. Investigating the failures is time consuming, but somebody is going to have to do that whether the testing is done manually or automatically. It would certainly be great if the KDE team can help with that work.

I know that openSUSE does have automated real-hardware testing with OpenQA, as they have physical laptops wired into OpenQA and those tests do run on every openSUSE Tumbleweed snapshot. These tests exist in openSUSE OpenQA as ones with the suffix beginning with @laptop. They also do have RPi tests with RPi boards wired in for their AArch64 tests, but I don't know how reliable those are. They at least seem to be green most OpenQA runs, so that is something...

SUSE, frankly, put more resources into it. At some point, AIUI, SUSE basically decided openQA was all their testing and put all their testing money into it.. We have me and @lruzicka and that's it. The two of us can only do so much. One day I will get the damn PiKVM out of its box and do something with it! One day.

That would be great, but now we're in a situation that if it is enabled, it risks Fedora QA coming back and saying we need to drop release-blocking status for KDE on AArch64. I really don't want to have this discussion again when Fedora KDE is putting in the effort to support the release-blocking status just because now Fedora OpenQA runs the KDE tests on AArch64 too.

I don't think this would happen. Honestly most of the trouble with ARM is the manual testing of crappy hardware. Running openQA tests on two arches isn't a lot harder than running them on one. Investigating the failures is time consuming, but somebody is going to have to do that whether the testing is done manually or automatically. It would certainly be great if the KDE team can help with that work.

We (Fedora KDE) are happy to help in that regard. We already are trying to be more proactive with this stuff, and OpenQA runs would help us on that front.

I know that openSUSE does have automated real-hardware testing with OpenQA, as they have physical laptops wired into OpenQA and those tests do run on every openSUSE Tumbleweed snapshot. These tests exist in openSUSE OpenQA as ones with the suffix beginning with @laptop. They also do have RPi tests with RPi boards wired in for their AArch64 tests, but I don't know how reliable those are. They at least seem to be green most OpenQA runs, so that is something...

SUSE, frankly, put more resources into it. At some point, AIUI, SUSE basically decided openQA was all their testing and put all their testing money into it.. We have me and @lruzicka and that's it. The two of us can only do so much. One day I will get the damn PiKVM out of its box and do something with it! One day.

Yeah. This is a common story here: build systems, testing infrastructure, etc. They put more into automation because historically they had fewer people. We can learn and benefit from it, though. :grinning:

Yeah. This is a common story here: build systems, testing infrastructure, etc. They put more into automation because historically they had fewer people. We can learn and benefit from it, though. 😀

I don't say they put more into automation, they put more into openQA. They bet the farm. Red Hat, in typical fashion, hasn't bet the farm on anything; RH's resources are spread around on a bunch of different test automation systems run by different teams at different levels for different purposes with different strengths and weaknesses, glued together in various ways and to differing extents...

Man, when I joined RH we were still talking about the Fedora Beaker lab. I'm sure it's coming any day now!

They also bet the farm on OBS prior to that. The bet the farm on Spacewalk, kiwi, etc. They do this a lot because they can't afford to waffle on things, because they don't have the people to spread around for that.

I think we've moved away a bit from the primary purpose of this topic - to get approval for dropping the release-blocking status of F43 Workstation aarch64.

So far, I've seen approval from Fabio and Neal. We have Beta Go/NoGo on 2025-09-11, where this decision needs to be known.

How do we handle this at this moment? Is ACK from two FESCo members good enough, can we ask Fedora Program Manager to implement this? Do we wait a few more days for more FESCo members to voice their opinion? Do you intend to discuss it in a FESCo meeting and vote on it? Please tell me what the next step is, thank you :)

This all comes down to the fact that fedora shouldn't be shipping high severity bugs. And the severity is at least partially dictated by whether or not anyone notices that bug. AKA, can't boot fedora workstation on arm, and no one notices?

The problem is the set of people who may notice changes. A lot of people only install stable releases. Just because nobody noticed the problem at random during the pre-release cycle doesn't mean nobody will notice it after release.

Right, I think your making my argument for me..

We had a much less structured, more haphazard release process and philosophy before. There wasn't much structured testing. There were no release criteria. Blocker decisions were made based on vibes, and blocker status was not strictly adhered to. This didn't work very well, and is why Fedora used to have a terrible reputation for quality. We now have a fairly good reputation for quality, because we have proper procedures and criteria and we respect blocker status. I don't think we want to go back to deciding blockers based on vibes.

Server vs. Workstation is significant because there is no release-blocking GUI functionality in Server. If we didn't block on Workstation or KDE we would not care about any pixman bugs.

Cockpit is explicitly listed in the server blocking criteria, and it has KVM functionality. Which along with "server with GUI" which is explicitly in the formal documentation too, seem that its a bit of which side of the argument you fall on.

And to clarify, much of my interest in this topic stems from a "I want the BMC on my server to work properly" perspective.

For purposes of the FESCo meeting today, lets assume that I will pickup most of the workstation testing. If Neal can test KDE, I can test workstation.. And I've attended blocker meetings/etc, having nominated a few in the past.

I have, or have access to pretty much all the actual workstation HW, be that the x13s, m1, orion6, various repurposed server boards with GPUs, etc. As well as pi3/4/5, although I don't consider them really to be candidates for actual workstation usage. IoT style signage apps, yes, actual workstation, no.

I need to get some things lined up, over the next couple days to clarify this, which includes making sure the latest fedora tests run sorta work on the suse openqa instance I have running in a container, since my attempts at duplicating the fedora infra are often a timesink.

So, I would request a punt today.

How do we handle this at this moment? Is ACK from two FESCo members good enough

No, by FESCo rules, it needs to have +3 after a week or +1 after two weeks with no -1 votes. If anyone votes -1, it goes to a meeting where it's discussed and then voted on there.

I myself have been following this discussion but explicitly not voting, because the debate is still raging. It hasn't calmed enough that I think it's worth voting on yet.

can we ask Fedora Program Manager to implement this? Do we wait a few more days for more FESCo members to voice their opinion? Do you intend to discuss it in a FESCo meeting and vote on it? Please tell me what the next step is, thank you :)

Given the ongoing debate, I'd normally be inclined to punt and wait until we figure out if there's a consensus coming. It sounds like @jlinton is going to attempt to mirror the KDE SIG's efforts to maintain aarch64 for Workstation and I'd be in favor of deferring the vote until we see the plan come to the table.

On the other hand, the Go/No-Go decision is nine days away; if we defer our vote to the next FESCo meeting, that's only two days to adapt the QE policy for Beta.

Since FESCo policy is to maintain the status quo unless a proposal reaches a majority, the current expectation would then have to be that we continue to treat aarch64 Workstation as blocking for F43 Beta. Can QE tolerate that, with the possibility of it being revoked for F43 Final?

This was discussed during the FESCo meeting today:
* AGREED: FESCo will leave aarch64 as blocking on both Workstation and KDE media, under the condition that blocking issues are maintained by Jeremy Linton and the KDE SIG, respectively (+6, 0, 0)