Am I understand this correctly that this would be the situation you envision?

• a rawhide update is submitted
• the update goes through gating tests
• all required tests pass, but (yet) non-blocking CoreOS tests fail
• update is submitted to stable by bodhi
• untagged from rawhide after the fact due to failed CoreOS tests

Assuming this is correct, then I don't like it - and would rather see CoreOS gating tests made required :)

Untagging things from rawhide after a bodhi update hit "stable" leaves things in inconsistent states forever - bodhi will record the update as having been pushed to "stable" since it doesn't know about it having been "unpushed" after the fact. That means the update couldn't be resubmitted from the same build, which is now possible for builds that were unpushed by bodhi (without circumventing bodhi).

In the mailing list thread I've been told this is a normal workflow for how things are prevented from going to rawhide. i.e. as long as you untag before the next rawhide compose happens then it won't actually go into the rawhide repos.

I'm not sure what the state in bodhi looks like if this workflow is applied. @adamwill or @zbyszek might be able to comment on that.

when we manually untag something that already made it to 'stable' so far as bodhi is concerned, bodhi knows nothing about it. it still shows as stable in bodhi. you can find the true state in koji (which is always the canonical reference).

we used a process similar to the one dusty is proposing for Rawhide, unofficially, for several months when we were running the openQA tests on Rawhide but not gating on them. When the tests found something that looked like a serious problem I would ask releng to untag the build in question. we would then usually work to get the problem resolved and the same, or a newer, build tagged back in.

Thanks @adamwill for the extra insight.

Of course, if this process were ever used, a comment would be added to the bodhi update to mention what happened. So if the user looked past the stable status at the top they may see.

I guess if you think this will happen rarely and to not cause issues, then it should be fine.

My guess would be average once a month (some months none, some months more; like when heavy development is happening), but that's just a gut feeling.

So, when we have done this, we have used a releng ticket to track / allow for a 'tail' as to what happened.

Do coreos folks directly need to untag things? Or could that just be part of the releng process... file ticket, releng processes it (although to be clear, I'd be happy to have more folks helping in releng, and if some coreos folks wanted to do so we could look at adding them there)

I think a ticket is important because it gives visibility to whats happened and allows people affected to coordinate.

Also, permissions to untag are in koji (which is seperate from anything else). (Although I have a plan to move this to fas groups, but I need to test some things before I can say if that will work).

Anyhow, I'm fine with the idea, just need to sort the process details.

I don't think we should do this. Why should we allow this when no other Fedora deliverable team has that capability?

If I were to request to have the same powers on behalf of Fedora Cloud or Fedora KDE, I'm pretty sure the answer would be "no" even though we also have gating tests.

Both groups file tickets with release engineering to get stuff untagged when things break, and I don't see a compelling reason to give Fedora CoreOS the ability to bypass this.

I think I am in line with @ngompa , I don't think it should be something specific to CoreOS but follow the normal releng process.

For example, I am a member of the @releng group (maybe I shouldn't since I don't contribute much anymore) but would having someone from the CoreOS WG join the releng group grant these permissions? That way it is not so much about being an exception for CoreOS but just helping to participate in the bigger releng ecosystem.

@kevin
I think a ticket is important because it gives visibility to whats happened and allows people affected to coordinate.

Yes. The proposal in the description mentions that a ticket would still be filed.

Do coreos folks directly need to untag things? Or could that just be part of the releng process... file ticket, releng processes it (although to be clear, I'd be happy to have more folks helping in releng, and if some coreos folks wanted to do so we could look at adding them there)

To be clear what I would prefer here is what we asked for on the list, which is the permission to mark CoreOS tests as blocking for a subset of packages. However, that was rejected on the list so I'm asking for the permissions that were suggested as a compromise on the list.

What I'm asking for is the ability to act without having to bug someone. In all likelihood I probably would still ping in the releng channel after opening a ticket, but if no one is around, would still like to be able to do something.

I don't personally care whether it's an exception made for CoreOS folks, or if some CoreOS folks can get onboarded into releng processes without it really being much responsibility.

I don't personally care whether it's an exception made for CoreOS folks, or if some CoreOS folks can get onboarded into releng processes without it really being much responsibility.

I'd prefer to onboard CoreOS folks to do this as part of releng, but I guess it's all the same difference in the end.

Anyhow, +1 to doing so and trying this out...

I don't think we should do this. Why should we allow this when no other Fedora deliverable team has that capability?

If I were to request to have the same powers on behalf of Fedora Cloud or Fedora KDE, I'm pretty sure the answer would be "no" even though we also have gating tests.

One difference, I think, is that the CoreOS tests are arguably generally applicable to all deliverables, it's just that we are not convinced yet that they are

Both groups file tickets with release engineering to get stuff untagged when things break, and I don't see a compelling reason to give Fedora CoreOS the ability to bypass this.

Agreed - I think I'd also prefer onboarding some CoreOS people who want to help out to releng. Looks like dusty and kevin are both on board with that too, so ... +1 for trying that out.

Onboarding CoreOS members in releng seems to be the most sensible approach, imho

+1 too.

APPROVED some CoreOS SIG members will join releng (6,0,-0)