I think its worthwhile to clarify, that second link (https://fedoraproject.org/wiki/Architectures/ARM/Supported_Platforms) is super stale and probably should be deleted except for historic info. In the F35 timeframe, as linked from the supported_hardware_and_devices page, it was replaced by https://fedoraproject.org/wiki/Architectures/ARM/Fedora_Linux_35. But, since these pages are outdated nearly as soon as they are published I think everyone is just updating the supported_hardware_and_device Wiki pages as they test hardware they are using. It would be nice to have all that hw in an automated lab and the test matrix updated, but frankly it costs $$$$$$+ and then arm would be special because no one is doing that for random x86/etc boards either. From the perspective of someone fixing things in various upstreams and trying to assure that fedora works as expected, my goal is 'just works' for most of the systemready (SR class) platforms and the few stragglers that are 'popular' but non-standard like the x13s, rpi4, etc.

This differs from other's goals (pbrobinson can clarify what he looks after, for example, as can many other people who are in the trenches just assuring parts of fedora work properly on arm based platforms) in complementary ways. But both sides of that equation represent a lot of deployed HW, and while https://fedoraproject.org/cloud/download#cloud_launch is easy, there is maybe another dozen+ cloud providers each with a few platforms that fedora runs on, usually with little more than building a custom image with imagebuilder/whatever.

To save a google search, some of them can be found from here: https://www.arm.com/markets/computing-infrastructure/cloud-computing

I think it's fairly reasonable to treat ARM this way given the much lower degree of standardization and how difficult supporting accessible hardware has been.

SystemReady SR platforms are rare, and ones that are affordable are even rarer. For example, the Radxa Orion O6 is basically unavailable in North America, and in low supply in Europe. The most affordable SR platform I can purchase is >US$2500 from System76.

Non-SR platforms are commonplace, and it's a question of what non-SR platforms we can reasonably support given that every single one requires independent manual testing, since we lack automation infrastructure for testing these things.

I would personally like to see us support WoA devices and the RPi 4 series and newer. But WoA devices are still a mess at the moment, and I am unsure how complete our Raspberry Pi platform support is. I know we don't have RPi 5 series, and I think we have most of the RPi 4 series.

But at the moment, PC-class ARM hardware seems out of reach on mainline Fedora. Of course, I'd love to be wrong and would appreciate any information to the contrary that would help us better support ARM in Fedora.

that second link is super stale

We've been trying to consolidate both lists for the past 5 years. Our current proposal is just the latest attempt. The "super stale" list is still referenced from our release criteria, so it's the only official hw list that matters, and nobody seems to care enough to change it. I don't even know who's supposed to be leading this area. Sure, Peter is the main development person, but way too busy, and Paul is probably mostly involved in IoT and not ARM in general, I think? No idea. The ARM page is not helpful either, no names, it still links to IRC, the arm list is super quiet, there's no ticket tracker. It all feels dead, and frustrating, to me.

Currently QA is in a situation where we believe that we need to do something about it now, and not wait another 5 years, hence this ticket. I'll consider a success if we manage to do any RPi4 testing this cycle, or it'll be just automated VM testing only, and relying on community and SIGs to do the job. The proposed unification would give us a clear guidance on what platforms are important when dealing with proposed blockers.

Can we stop conflating what QE is testing on with what is actually supported please.

We don't limit the blocking devices on x86 to the devices QE has sitting on their desks.

If QE want to limit their testing to a set of devices that is fine, it will give them a good overview of things like packages running or crashing, if not a good overview of the overall ecosystem. But please don't limit that to the things we block on.

For example, across the 100s of devices we support the most widely used (and well supported) GPU is the MALI GPU, it's used across a number of vendors and overall would be much more widely deployed than the rpi specific v3d/vc4 device. Similarly on other devices the most widely used NIC uses the sttmac driver, the most widely used USB is dwc3 etc, none of this IP is used on the RPI and if testers in the community find problems with them we should be blocking on it. This is like we'd block on a Intel GPU or Intel/Atheros wireless issue even though they aren't explicitly listed as blocking devices because they're widely deployed.

I have no issues if QE want to limit what they test on, what I have issues with is using that to define blocking lists which is the wording which QE is using. Please DO NOT conflate the two problems.

A few notes on the discussion thread (the forums are really bad when you don't have connectivity, which is why I didn't comment on them, and we were asked not to use the list even though a message was sent there).

1) in cloud arm is quickly becoming the most widely deployed there according to various proivder numbers
2) probably the most popular, based on bugs/RFEs/queries I get) are rockchip based devices (rk33xx/rk35xx etc)
3) there's lots of interest QCom laptops
4) there's a lot more interesting stuff coming down the pipe

We've been trying to consolidate both lists for the past 5 years. Our current proposal is just the latest attempt. The "super stale" list is still referenced from our release criteria, so it's the only official hw list that matters, and

As I have stated before in every time we have this conversation I don't believe an explicit device list is the way to do blocking, we don't do it for x86, and for that reason I have not updated the device list. The device list goes back to original armv7 early days when we really did support just 6 devices. The world has long moved on. We should just drop an explicit list.

As I have also previously stated before, I am also happy to work with QE for them to have a handful of devices to test with, that will evolve over time, but that absolutely should not be a release blocking list of hardware.

We don't limit the blocking devices on x86 to the devices QE has sitting on their desks.

I think the difference here is the standardization, ease of acquisition, and reproducibility. For x86 hardware, how the hardware is put together is independent of how hardware is activated in Linux. Driver issues can be reproduced across multiple hardware platforms reliably, and in cases where specific platforms are required, it's much easier to acquire them (either first-hand or second-hand).

With ARM platforms, non-SR hardware doesn't work that way at all. And in practice, we should have an HCL for DT based platforms because DeviceTrees are system specific.

I think the difference here is the standardization, ease of acquisition, and reproducibility. For x86 hardware, how the hardware is put together is independent of how hardware is

Yes, you've missed my point though, separating testing from blocking.

Just like on x86 if someone files a blocking bug there's a process for them testing and dealing with it.

Of course if QE choose specific platforms it's by definition it's more likely to have blocking bugs because they'll be testing it but we should no be changing the blocking process to be explicit to hardware. If QE wants to choose explicit it should be something closer to the SystemReady standards than RPi because at least then testing wider things like the standardised power interfaces (PSCI/SPMI etc) that are now basically used everywhere else.

I'm going to attempt to rephrase things, so bear with me if I get it wrong, please.

Due to significant personnel and hardware shortages, QA needs to reduce the amount of testing done. Testing of aarch64 hardware is a significant drain on those resources for limited return: very little of that hardware is actually used in the community.

Peter's counterargument is that we don't have to drop blocking status for untested hardware, we can treat it akin to how we've treated Fiber Channel support for years: we don't actively test it, but if someone discovers a major issue and reports it, we might block on it. (Sidebar: this is probably a lie, since any such blocking issues on hardware we don't have is likely to get the "difficult to fix" fudge in blocker review meetings).

Neal's counter-counterargument is that, unlike with x86_64, the aarch64 hardware available rarely have any commonalities, so that in most cases we cannot reproduce bugs without that exact hardware (sometimes even the same revision of that hardware, such as Raspberry Pi 3 B+). So if we agree to block on general "aarch64 hardware", in order to verify the fix, we will need someone with that exact hardware to do so. This gets us back into a similar situation to where we started, which is dealing with resource-constraints.

The proposed middle-ground (as I understand it), is for QA to select a very small subset of readily-available hardware that we will treat as both testable and blocking. Issues that come in that are not reproducible on that subset of hardware will not block a Fedora release.

So, if I report a hypothetical issue on a piece of non-blocking aarch64 hardware where Emacs crashes performing some function and I can reproduce this same behavior on Fedora Asahi or on a Raspberry Pi 400, then it's a blocking issue. If the issue is unique to that hardware... maybe it gets a Freeze Exception if someone figures it out and the reporter verifies it.

Peter's counterargument is that we don't have to drop blocking status for untested hardware, we can treat it akin to how we've treated Fiber Channel support for years: we don't actively test it, but if someone discovers a major issue and reports it, we might block on it. (Sidebar: this is probably a lie, since any such blocking issues on hardware we don't have is likely to get the "difficult to fix" fudge in blocker review meetings).

That is not what I said, I said:
1) we don't have to define blocking HW (we don't do that on x86 even on HW we would block on)
2) QE is not the only group doing testing

Neal's counter-counterargument is that, unlike with x86_64, the aarch64 hardware available rarely have any commonalities

That statement is also incorrect, there's lots on commonalities, GPU, wireless, network, USB etc etc

The proposed middle-ground (as I understand it), is for QA to select a very small subset of readily-available hardware that we will treat as both testable and blocking. Issues that come in that are not reproducible on that subset of hardware will not block a Fedora release.

So, if I report a hypothetical issue on a piece of non-blocking aarch64 hardware where Emacs crashes performing some function and I can reproduce this same behavior on Fedora Asahi or on a Raspberry Pi 400, then it's a blocking issue. If the issue is unique to that hardware... maybe it gets a Freeze Exception if someone figures it out and the reporter verifies it.

Where does Asahi come in here, they're a fork of Fedora with a different kernel and all sorts of other stuff. In most cases things like emacs would be the same issue every where, it's things like drivers where the problem comes

Peter's counterargument is that we don't have to drop blocking status for untested hardware, we can treat it akin to how we've treated Fiber Channel support for years: we don't actively test it, but if someone discovers a major issue and reports it, we might block on it. (Sidebar: this is probably a lie, since any such blocking issues on hardware we don't have is likely to get the "difficult to fix" fudge in blocker review meetings).

That is not what I said, I said:
1) we don't have to define blocking HW (we don't do that on x86 even on HW we would block on)

We sort of do, though it's not by brand and model. It's not really explicitly stated, but the "between the lines" reading is "You could expect an x86_64 Windows install media to run on this machine". There does exist rare x86_64 (appliance) hardware that doesn't meet that requirement and I am absolutely sure we wouldn't block Fedora on it.

2) QE is not the only group doing testing

I don't think I said otherwise. But QA is the only group whose testing we can rely on (in the sense of being able to explicitly say: "I need this tested by Thursday in time for the Go/No-Go meeting"). A third-party may be able to do that... or they might not.

Neal's counter-counterargument is that, unlike with x86_64, the aarch64 hardware available rarely have any commonalities

That statement is also incorrect, there's lots on commonalities, GPU, wireless, network, USB etc etc

I probably overstated that. My apologies.

I mostly meant that, unlike x86_64, I can't reasonably expect that the same install media would work on two different models of the hardware.

The proposed middle-ground (as I understand it), is for QA to select a very small subset of readily-available hardware that we will treat as both testable and blocking. Issues that come in that are not reproducible on that subset of hardware will not block a Fedora release.

So, if I report a hypothetical issue on a piece of non-blocking aarch64 hardware where Emacs crashes performing some function and I can reproduce this same behavior on Fedora Asahi or on a Raspberry Pi 400, then it's a blocking issue. If the issue is unique to that hardware... maybe it gets a Freeze Exception if someone figures it out and the reporter verifies it.

Where does Asahi come in here, they're a fork of Fedora with a different kernel and all sorts of other stuff. In most cases things like emacs would be the same issue every where, it's things like drivers where the problem comes

I debated mentioning Asahi, but I kept it because it's probably our most-tested platform for the aarch64 runtimes. Yes, it's a remix, but it's probably the easiest physical (non-VM) device we have access to for testing aarch64 issues that have nothing to do with the hardware itself. (For example, I used it to test a fix for a qemu-system-x86_64 bug a while back since at the time I didn't have another non-headless aarch64 system available and the bug wasn't reproducible on a VM).

So FWIW, aside from the resource constraints, a problem with the approach we've been using for a while is that it kinda becomes the @pbrobinson show. Peter nominates bugs as blockers; because the hardware support lists are antiquated and he knows way more than anyone else about ARM, the process of deciding whether the bug is a blocker basically becomes "well, Peter said it was, so I guess it is"; then we all sit around waiting for Peter to fix it. We probably can't do anything to investigate the bug or verify the fix because we probably don't have whatever platform the bug exists on.

On the one hand, hey, we want to reduce work, right? OTOH, that feels like a bit of a wacky process. There's also a lot of assertions in the above comments without much data - on how popular various pieces of hardware are, etc. It would be good to have some kind of objective numbers, here. @mattdm , do we have any stats on arch use across editions?

Looking at the ARM mailing list archives, there are two emails this year from people using Fedora on non-Raspberry ARM systems: one asking for a 32-bit build of a package to rescue an ancient 32-bit install (not really relevant), and @pboy asking about his Rock Pi. So, basically one person.

Looking at https://discussion.fedoraproject.org/tag/arm , this year we have two threads about Snapdragon X Elite (which we don't really support), one general tablet inquiry, and one thread about a Pi system. That's it.

Looking at bug reports, ~~there have been 59 Fedora bugs reported on aarch64 this year. There have been 2928 reported on x86_64. I just went through nearly every one of those 59 bugs (looks like I lost one, somehow). Out of them, 22 are on Asahi, or at least identifiably on an Apple M* CPU (so Asahi or virt on a Mac). 10 are identifiably on virtualization. 5 are on Raspberry. 6 were people reporting build or packaging issues with no identifiable tie to a hardware platform. 7 were unknown. Only 8 reports are identifiably on a hardware system that's not a Mac or a Raspberry: one Thinkpad X13s, one XPS 13 9345 (Snapdragon), one Orange Pi 5, three Pinephone reports (all from the same person), and two I filed as "other HW".~~ SEE UPDATED BUG NUMBERS BELOW

Where is all the usage/interest/bug reporting claimed above coming from? Where can we see them?

Cloud is a different situation entirely. It's much easier and faster to test on cloud, the set of supported functionality is massively reduced (no worrying about 3D or wifi); it's much more susceptible to automation (Cloud SIG is currently working on this). We already test and block on aarch64 on AWS and we're not proposing to change that; we can consider adding additional blocking cloud providers along with x86_64.

To try and come up with a way forward here: just considering the "QA resource constraint" problem, that would definitely be helped if we dropped the explicit lists of supported hardware (all of them!), the criteria ties to those, and the hardware test matrices in the validation pages (except for any that other teams can commit to testing on, e.g. whatever the KDE team is going to test ARM on since they've committed to doing that, and whatever anyone can commit to testing IoT on).

In theory I think we could buy saying something about SystemReady being the equivalent of 'generic' x86_64 hardware, but it's complicated by the practical issues around availability of SystemReady-compliant hardware. I don't have any (I don't think; the latest thing Peter told me to buy was the Particle Tachyon, AFAICS it's not SR). I don't think we have any in Brno. As discussed above, none seems to be readily available on the open market at reasonable prices. So in practice I don't think we can do this unless someone can point to a readily-available SR 'reference system' at less than, say, US$1000.

That still leaves the "pbrobinson show" problem: we'd still - in fact, probably to an even greater extent - be in the position of relying on Peter to nominate blockers, guide our acceptance decisions, and fix the problems. But that is kinda a separate problem, I guess.

[...]

Of course if QE choose specific platforms it's by definition it's more likely to have blocking bugs because they'll be testing it but we should no be changing the blocking process to be explicit to hardware. If QE wants to choose explicit it should be something closer to the SystemReady standards than RPi because at least then testing wider things like the standardised power interfaces (PSCI/SPMI etc) that are now basically used everywhere else.

I see that the SystemReady-compliant hardware ensures standard interfaces for UEFI boot, ACPI support and other goodness. Testing standardized interfaces sounds great (I prefer this too!), as you can avoid dealing with custom device tree and other boot process complications.

Based on my very limited hands-on experience with server-grade Arm hardware, some machines that seem to be SystemReady:

• Nvidia "Grace" and "GraceHopper" series
• AWS "Graviton" series
• "Radxa Orion O6" (which claims to be the "world's first open source Arm v9 motherboard"), but is only available in two countries, along with "Global". (Neal also mentioned this hardware above)

Now, who has access to one of these that can consistently test on Fedora and reliably report back?

The Tachyon 5G is supposedly a UEFI board, but it doesn't mention SystemReady, so it may be in the same class as WoA devices where it uses UEFI but requires some DeviceTree finagling anyway to work.

D'oh, it just occurred to me, of course Bugzilla search was default filtering to only open bug reports. I'll re-do the numbers including closed ones, please stand by. sorry...

"Global" means "you can get it from AliExpress", which, yeah, will work in most countries I guess. Delivery is a couple of weeks, which isn't awful. There will be import taxes/customs in most countries. Still, if that thing actually works, it might be a viable test platform. But for how long? This is the other thing with ARM - there always seems to be another one around the corner. I have giant teetering pile of ARM SBCs here which were The Hot Thing for like six months each.

The price is basically more than double for me because of US tariffs. I don't think it's reasonable to spend that kind of cash for it.

Updated bug numbers:

• x86_64 - 4475 total
• aarch64 - 111 total

Updated aarch64 breakdown:

• 27 asahi/mac
• 23 build
• 20 unknown
• 14 virt
• 14 raspberry
• 3 pinephone
• 2 i.MX8MP-EVK
• 1 x13s
• 1 xps13 9345
• 1 snapdragon x
• 1 orange pi 5
• 1 pinebook pro
• 1 ampere

The 'snapdragon X' report is from Dennis Gilmore, I forget what actual laptop model he was testing on (I think a Thinkpad). pinephone and i.MX8MP-EVK have multiple reports each from a single person, so they're not really any different from the 1s. The total isn't 111 because I lost a report and another wasn't counted because it was actually for x86_64.

'unknown' are cases where it's definitely not a 'build' issue (those are ones where the report is clearly just about some kind of build/packaging issue and don't indicate the reporter was using an ARM system outside of infra at all), but the report doesn't have sufficient info to determine what platform the reporter was using. These could be (and many likely are) virt, or asahi, or raspberry, as well as being something 'else'; I guess the most valid thing to do would be to assume they break down approximately along the same lines as the 'known' cases.

BTW, I dug a bit into:

1) in cloud arm is quickly becoming the most widely deployed there according to various proivder numbers

I found a lot of fluff, but the only solid numbers I got were:

• A claim from an AWS re:Invent 2024 keynote where Dave Brown stood in front of a screen saying "Over the last two years 50% of new CPU capacity was on AWS Graviton" and said more or less the same thing, with a lot of poorly-written and unsourced articles written around that time which say something like "Half of all instances spun up on Amazon Web Services over the past two years used Graviton" - like this one - but that's not what that slide says, or what Dave Brown said (you can see the video at https://reinvent.awsevents.com/on-demand/ , it's the "Monday Night Live with Peter DeSantis" video, this bit comes at about 17:25)
• A 2023 article which - without sourcing - reports that "Arm now claims to hold a 10.1% share of the cloud computing market", but in context that seems to be mostly about Graviton in AWS as well
• A report saying "Arm Holdings expects its share of the global market for data center central processing units to surge to 50% by the end of the year, up from about 15% in 2024 with gains driven by the boom in artificial intelligence, a senior executive said" - in context this again looks to be mainly about AWS and also LLM DCs using NVIDIA's chips, which I don't think we support or test or block on

So mostly this is about Graviton in AWS, which is a very specific case, and one we don't really have any problem supporting/testing; we will absolutely continue to test and support aarch64 on AWS (though it would be great if we could finally get it automated). But it's a custom CPU you can't get outside of AWS, running through a (very nicely) abstracted interface. It's not really related to testing/blocking on graphical desktops and wifi on unreliable SBCs. The LLM datacenter case is an even more esoteric and specialized one which doesn't have a whole lot to do with us.

Thanks, Adam, for digging up those numbers. I feel they actually support our proposal. Both the communication channels and bug numbers seem to match the graphs that we shared in our ARM desktop proposal (note: that was desktops), showing ARM as having a very low user base - a few percent at most of all our users. Of those users, the vast majority of them either run on a Macbook (which Fedora doesn't support) or a Rasperry Pi, and any other boards are just outliers. Which is why we suggested RPi as the only blocking board that makes sense.

I don't understand why Fedora should block on exotic boards that our users don't use. It's nice that they're standardized and everything, I wish them luck, and I believe Fedora should definitely try to support them, but until they're widespread among our users, why would we block on them? We also don't turn a Spin into an Edition until it has gained a significant userbase and importance.

The same goes for QA testing. We're limited on time, but have endless work to do. We need to prioritize heavily, and that's why we focus on the mass market, the most popular options, because that affects most of our users. Having a SystemReady board for testing is theoretically nice, but how do we justify spending time on it rather than on something more important to our users? Especially now with our limited manpower, we might not be able to even do any RPi testing, the most popular board out there, and just rely on other teams/volunteers to ping us with bugs, and maintain our automated virtualized tests, and nothing else. I actually think that if somebody wanted to designate some SR board as the only blocking board instead of a RPi, it would be much better for Fedora to not block on any ARM board at all, and just block on virtual tests - the community impact would be the same, and QA would have more time to look at other things.

Peter keeps claiming that we block on all x86_64 hardware in general, and therefore we should block on all aarch64 hardware as well. It's true and false at the same time. We heavily discuss the popularity of the broken hardware when deciding a blocker status. This is codified in release criteria, and we even have a dedicated FAQ entry for it. We can use the same approach, but that again leads us to the same result - the Rasperry Pi. We prefer having it defined in a document (and update it from time to time), because while we can do at least some reasonable guesses in the x86_64 world, we have no idea about the aarch64 world (because the hw is not widespread among our users and in our homes), and we can't do numbers investigations (as Adam did) every time. In practice, we rely on what Peter says in each case, and we're not aware of anyone else who is also knowledgeable in this area (maybe Neal, he knows everything). Petr loves ARM, and I love his passion, but this is not a healthy way to run the release process.

Please note that everything above is seen from the "aarch64 is a tinkerer board, maybe a laptop one day" perspective. There are also other important areas:

• Cloud is completely separate from this, because it's operated through cloud provider environments. As Adam said, we already block on aarch64 for Cloud images in AWS, and don't request any changes there. Sorry for not being clear about this from the beginning.
• Server aarch64 support might be important for our users, but even Peter Boy couldn't supply any information about what hardware is probably the most popular for Fedora Server. The server-grade hardware is probably not something that we would be able to test frequently, even if we had access to it, but if we had more information, we could include some specifications in our criteria and it would work akin to the FibreChannel etc stuff - blocking if somebody reports it broken, but not generally tested (at least by QA, but maybe by the Server team).
• IoT might share the general RPi popularity, or might not, and we have no idea. We hoped IoT folks would supply us with some data or guesses, but we didn't hear anything back. Simply for convenience and as our best shot, we proposed the RPi4 for it as well.

This was discussed during the FESCo meeting today:
AGREED: Fedora on AArch64 release-blocking hardware moves from a dedicated hardware compatibility list to a generic architecture status where issues are decided on a case-by-case basis with similar factors to how x86_64 blocker bugs are decided. (+6, 0, 0)

Wiki draft changes and discussion: https://discussion.fedoraproject.org/t/release-criteria-proposal-drop-arm-and-iot-hardware-compatibility-lists/163414