#3437 Draft of Formalised Spins and Labs Keepalive Process - Feedback and Discussion welcome
Closed: Accepted by zbyszek. Opened by amoloney.

Hi FESCo members,

I've drafted a formalised process for the spins and labs keepalive requests. I havent been doing a great job on keeping to the deadline or on top of responses by spin and lab maintainers, and there were two labs missing from the F43 release that were listed officially (Neuro & Astronomy). When reviewing what happened in F43, I found there was actually no process that I was following, it was more of a 'oh crap that deadline has passed I better open tickets' which is neither scalable nor suitable :(

So, Ive created a light process to follow and I would like your feedback on it. I've put it in a hackmd, let me know if theres a better format though and Ill swap it to that.

Id like to get this process trialled for F43 which means getting it underway in about a weeks time to see how it holds up in real life and to get feedback from the spins and labs too. Its process improvement but not release blocking so please weight it against other responsibilities you have appropriately :)

Thanks!
Aoife

https://hackmd.io/@Ap8CkTlpSfmjb44UGV-kWA/BJLoyg2Bgx/edit


The process looks reasonable, but I think we probably want to gate things at Beta rather than Final, barring extraordinary circumstances.

In general, if something is broken at Beta and there is no one to fix it, then it's probably not getting fixed for Final either, and we might as well discontinue it at that point.

Should we also note that a "Yes" response only means that Release Engineering will continue to attempt to compose the Spin, not that it guarantees that the Spin will actually ship? Spins aren't release-blocking, so I have concerns about the phrasing "ensure that all necessary artifacts for the Spin or Lab are built, tested, and made available for download on release day".

And I agree with @ngompa about the timing: this keepalive needs to happen aligned with the Beta Freeze date as the deadline (since we start trying to produce RCs at that point). I think it might be reasonable to file the tickets a little earlier though; one week is a bit short when it could correspond to a vacation.

How would you rephrase that to avoid the problem of having a spin or lab listed on the main website as available when we release, if its not actually available and general people are confused as to why something is there when its not. Thats one of the main problems Im trying to solve with this :-/

How would you rephrase that to avoid the problem of having a spin or lab listed on the main website as available when we release, if its not actually available and general people are confused as to why something is there when its not. Thats one of the main problems Im trying to solve with this :-/

Unfortunately, there are technical limitations we have to deal with there: even if a Spin has received its "Keepalive" notification, it's entirely possible that the Final RC won't build and thus include it (such as a package only included on that Spin is uninstallable). So the finalization of the website design needs to happen in the five days between the Go decision and the release announcement. To that end, we should probably ask that the website team look into ways to make the Spins page easily modular, so Spins that don't make the cut can be dropped from it easily, without requiring a major rework.

That could work!

Who must open the tickets?

Well... we actually do already have a policy?

https://docs.fedoraproject.org/en-US/fesco/Spins/

So I guess this is an expansion/clarification on process there?

For the images that don't build, I think websites already can drop things that were not produced. Although that adds to confusion in the end... I think it might be better to have the spin/lab page, but instead say 'sorry due to last minute issues this image wasn't composed for the XYZ cycle" or something.

The PgM (or FOA - me nowadays) would open the tickets as is current practice, and use the ticket open for the keepalive as the source of truth. If a spin or lab comments they want to discontinue, it probably does make the most sense for the PgM/FOA to be the one to open the tickets in the righr places notifying folks of the change.

I can reflect that better in the doc if that makes sense to ppl?

@kevin yeah I think this is definitely a clarification of the linked process.
I like the suggestion of the spins/labs page being updated in the event a spin or lab is late for the release for whatever the reason, but I think having a basic process to follow would be helpful so that information makes it to the web page before we release and not weeks later so I still see the need for a deadline by which things should be ready, and if not a next step is a ticket filed (by FOA/PgM) with websites on the spin or labs behalf using teh info they provide from their keepalive ticket, etc so the right updates can be made in advance of release. Would that still make sense?
FWIW this is for PgM/FOA to follow when working with the labs and spins per release so some are not accidentally missed.

Continued inclusion
Website Ticket: A ticket must be filed with the Websites team to ensure the Spin or Lab is included on the official release day website and associated pages.

That is unexpected. I'd expect that once a spin is included, it'll continue to be included until something changes. I don't think we want to have a dozen extra tickets each release.

reach out to the maintainer to check if they have seen the ticket and wish to keep the spin or lab active for F43.

This probably should say "next fedora release" not "f43".

Attempted summary of feedback to edit the proposal with:

  • Continued inclusion - remove the requirement to file a ticket as this is inherent from the ack the spin/lab maintainer gives in the keepalive ticket the FOA opens

  • Change wording to 'next release' where it gives a specific fedora release

  • Make explicit that the tickets (where required) will be opened by the FOA, unless clearly stated otherwise

  • Loop in websites team for their feedback on how best to handle discontinued and 'late' spins/labs for release day. This is to avoid having spins or labs listed that are not actually available.

  • Beta release is the preferred deadline to gate spins and labs against, even though they are not release blocking. This just gives everyone more time to get release ready.

Have I missed anything or summarised any points wrong?

Also to clarify the intention of this process - this is primarily for the FOA role to follow as an SOP. However, some parts are probably relevant to use to update the existing spins & labs policy to add a little more detail on how things work per release as an official spin or lab and what may be expected of spins or labs that wish to be continued in each release.

I'd like to start with an SOP though, see how that goes and get some feedback from the spins and labs as I am following it for F43 before proposing an official change to the spin & labs policy. Would that make sense to do?

  • Beta release is the preferred deadline to gate spins and labs against, even though they are not release blocking. This just gives everyone more time to get release ready.

It's the deadline to express intent to deliver those non-blocking deliverables. Because if they aren't out in Beta, there's no public mechanism to deliver testing content to users either. If no one has confirmed them prior to Beta Freeze, we drop them from the compose process so we won't even attempt to include them. This means that the compose may finish earlier, which means a tighter feedback loop if we need to do multiple RC spins.

Could it mean that there is just better tracking on status for spins or labs that dont make the beta to check in on their progress more during the beta -> final release time to make sure they are likely to make the final release date? Would that make more sense? And the 'tracking' can be absorbed by the FOA. Im thinking simple communication via ticket with the spin or lab so we know whats happening with them for the upcoming release. Maybe they wont make beta but they definitely will make final. Or maybe they wont make final but they should have something available 1 or 2 weeks post release. Then I (and others ) know the status and we can make sure thats reflected in our public comms on release day.

Is any further feedback from the side of FESCo needed?

I think all that feedback sounds reasonable to me.

Perhaps we could adjust the fesco policy page based on this then and have a sop on what FOA needs to do when?

I can make a PR to the FESCo docs that explains what we/me is trying to do with spins and labs tracking during the release cycle, and I can directly create an SOP in the FOA/PgM section too. I think this ticket can be closed then if agreeable to everyone?

Yeah, any adjustments can be made in the pr I would think...

  • Beta release is the preferred deadline to gate spins and labs against, even though they are not release blocking. This just gives everyone more time to get release ready.

It's the deadline to express intent to deliver those non-blocking deliverables. Because if they aren't out in Beta, there's no public mechanism to deliver testing content to users either. If no one has confirmed them prior to Beta Freeze, we drop them from the compose process so we won't even attempt to include them. This means that the compose may finish earlier, which means a tighter feedback loop if we need to do multiple RC spins.

Most images are built in parallel, so I don't think this is really an issue. From a time perspective, only the biggest one is the issue as that's the longest to build.

I think there's nothing left to discuss. I'll close this following the last few comments.

Metadata Update from @zbyszek:
- Issue close_status updated to: Accepted
- Issue status updated to: Closed (was: Open)

Just following up on this. I updated the PgM SOP to include some more detailed steps in the keepalive process https://docs.fedoraproject.org/en-US/program_management/pgm_guide/sop/spins-keepalive/. This seems to compliment the existing FESCo policy, and I cant see any changes required for that page https://docs.fedoraproject.org/en-US/fesco/Spins/.

I think the more thorough steps outlined in the PgM page should be sufficient to improve the process. We will see for F44/F45 though :)

Thanks for your feedback!

Metadata