For the record, I am -1 to no-go. We have already setup everything for the release. Large piles of people have downloaded the images from the staging area or torrents (which we open a day ahead).
I agree this is bad, but stopping everything now would cause more confusion.

There's a lot of problems from the releng side to stop and try and make another rc.

"respinning" also presents a bunch of problems. Perhaps they could be overcome, but at least for the immediate issue, -1 for no-go for me.

I would prefer we not reverse the GO decision either. -1 to no-go.

I think this is probably a case where we develop a process to officially respin all the deliverables and update the available media on the website. It's been a long-running thing for a while now where it's come up that we should have some way to refresh the images, especially to deal with commonbugs that get fixed later.

I think this is probably a case where we develop a process to officially respin all the deliverables and update the available media on the website.

:point_up: yes to this.

I can see both sides of this argument, especially "don't mess with people's systems from live media without explicit user action" ... but with the release literally being just hours away, I don't think pulling the emergency breaks is great, either.

That said, I am leaning towards a very weak and very-likely-inconsequential-given-prior-negative-votes +1 to no-go for April 15 (and wouldn't 04/22 for 42 be nicer anyway? :cry: )

We at least should communicate in any announcement saying do not use the live media yet, I think. Or pull the live media ISOs from the mirrors for now until they are fixed.

People who download something marked RC -- the expectations for QA is lower. If they download an ISO marked final and hit this issue... reputational damage indeed.

So - can't we do both? GO in general for people upgrading, but pull the affected live ISOs.

We at least should communicate in any announcement saying do not use the live media yet, I think. Or pull the live media ISOs from the mirrors for now until they are fixed.

People who download something marked RC -- the expectations for QA is lower. If they download an ISO marked final and hit this issue... reputational damage indeed.

So - can't we do both? GO in general for people upgrading, but pull the affected live ISOs.

We don't differentiate RCs from the GA release; they're not renamed or anything. They're the exact same medium.

This is... a really unfortunate situation. I think reputational damage can be mitigated if we make a public statement as part of the release, though.

I'm -1 to trying to abort the GA release, but I think we may definitely want to amend the release announcement to include something like the following:

"The Fedora Project has been made aware of an issue with some of the Live images in the release. Unfortunately, a bug was discovered after the distribution of the release to the mirrors that may result in the use of Live media adding an unexpected entry to the UEFI boot loader even when Fedora is not installed to the local system. While this may look alarming, the practical effect is largely benign: it will result in the Live media being the boot target when inserted into the system and should not affect the boot process as long as the media is not present. The Fedora Project apologizes for the issue and instructions can be found here to manually remove this extra entry."

Please feel free to wordsmith this as appropriate (or correct me if I'm misunderstanding BZ 2358785)

As unfortunate as this is, I am -1 on delaying the GA release.

Document this as a known issue.

Speaking as releng: I’m also -1 to no-go.

Yes, the bug is real and not great — live media should never touch UEFI config on the host like that. But we’re way too close to release to pull the brakes now. Torrents are seeded, mirrors are syncing, and backing out would just create more noise and confusion for users than it solves.

We also don’t have any tested process for a coordinated live media respin this late, and trying to improvise one under time pressure feels like a bigger risk. That said, I 100% agree we should figure out a post-release path for refreshing live media when issues like this come up — this is the first time, and it might not be the last.

For now, clear messaging on the download page and in the release notes is the way to go. Let users know to avoid booting the live image on real hardware if they don’t have to, especially with TPM systems.

-1 to the no-go.

But we need to document this issue.

I too see both sides of the argument. I'll defer to people with more experience and participation in the releng processes. If Kevin and others think that respinning would be more trouble than it's worth, I accept that.

0 to the no-go

I think this is probably a case where we develop a process to officially respin all the deliverables and update the available media on the website.

Yeah, we should. This has come up before, and if we have a process to update the installation media, we would be able to deal with bugs that are discovered very late or even after a release in a more graceful way.

-1 to no-go

But it should be documented in known issues.

I'm -1 to no-go, but I think we should really create a respinning process and start to use it frequently, so that we are always sure it works :)

For now, clear messaging on the download page and in the release notes is the way to go. Let users know to avoid booting the live image on real hardware if they don’t have to, especially with TPM systems.

We should not tell people this because we primarily produce lives and they're not fundamentally broken.

For now, clear messaging on the download page and in the release notes is the way to go. Let users know to avoid booting the live image on real hardware if they don’t have to, especially with TPM systems.

For me this is not a reasonable thing to expect from our users. Booting the Live ISO on systems with Secure Boot and a TPM enabled is the expected path for most of the people installing Fedora. We can not say: "Don't do it".

For me, it's a +1 to no-go. I wouldn't want to use a Live ISO that messed with the EFI configuration on my system. This is what Live ISOs are used for.

I'm really sorry that this is going to cause a lot of work but I think we should pull this release.

Note: I'm not a member of FESCo, so my vote "does not count". I'm a Fedora Atomic Desktops maintainer but we are not impacted by this bug as we are not building the installers with kiwi yet.

By current count of FESCo votes, we have (1, 0, 6) on this decision.
@salimma your vote is not super clear, I think its similar to @zbyszek who is 0. We traditionally announce the release in around ~2/3 hours and the release 'train' technically is in motion, so while I understand this is a huge decision to make, and not one anyone wants to, we do need to make a decision very soon so we are prepared for whatever the outcome.

CC @mattdm , I feel FPL should have say here too, or at least ack the decision.

By current count of FESCo votes, we have (1, 0, 6) on this decision.

Isn't it (+1, 1, -6)? Not that it makes a difference.

Thanks all. Let's goooooo!

@churchyard you're correct, apologies the vote is (1, 1, 6) right now and 7 fesco votes are needed to accept a decision on the fast track policy docs

With about 1 hour to go from our traditional release time and a majority vote in favour of releasing today and FPL ack, can we confirm what FESCo's decision is here please?
@humaton @sgallagh @kevin @salimma @zbyszek @dcantrel @fale @decathorpe @ngompa

I don't think technically we have met the criteria for a formal decision, so lacking a decision for making today's release a no-go, things are still on for "go" IMO

Even without enough fast track votes to affirmatively decide not to reverse the "go" decision, "not reversing the go" is the same as "go", which is the status quo.

Some of the 6 would have to change their vote to make it possible for "hold the presses!" to win, which seems unlikely in the next two hours.

So I think we're sticking to go. Am I missing something? It is early in the morning for me! :)

Fedora CoreOS will wait another 10m or so before running some of our final release processes (that take a little bit of time). We are assuming Go still.

I've written a patch for kiwi to give us a hook to fix this issue: https://github.com/OSInside/kiwi/pull/2778

It needs some testing to verify it resolves the problem.

we also have the option of just using xorriso to edit the already-built ISOs, since all we have to do is remove a couple of files from each. This might be 'safer' than rebuilding them all from scratch, in some ways. we'd have to re-checksum them all, of course.

The mediacheck checksum will also need to be edited, but yes.

We released Fedora Linux 42, so this ticket is moot.

For the final count, I'm a 0 - reporting this in the next meeting schedule as Rejected (+1, 2, -6)