Issue #2152: "backport" branches in src.fp.o for backports to coreos stable streams - fesco

fesco

#2152 "backport" branches in src.fp.o for backports to coreos stable streams

Closed: Accepted Jun 28, 2019 by psabata. Opened Jun 20, 2019 by dustymabe.

[note] context for this request lives in https://github.com/coreos/fedora-coreos-tracker/issues/79

Fedora CoreOS will have several release streams. The stable release stream will trail fedora stable (bodhi stable updates) by some time (approximately two weeks). Generally Fedora will output a security fix but only for the latest version of a package. For packages that move to new versions within a major release of Fedora (i.e. packages like the kernel), this can be problematic as we'd like to deliver the smallest delta possible to users on the existing stable release. Therefore we'd like to develop a method in which we can build a version of the package that represents the latest rpm in the affecting Fedora CoreOS stream + fix.

Rather than ask rpm maintainers to take on that role we'd like to use other branches in src.fp.o that is designated for backports for Fedora CoreOS. Thus, we'd like to propose if we need a backport for a package we:

create a coreos-backport-A branch to use. If we ever need two backports for a given package at the same time then we create a coreos-backport-B branch.
when we run a build we create a tag in the git repo (auditability for where the backport rpms come from)
we re-usecoreos-backport-A whenever we need a new backport (this way we don't pollute the git repo with a bunch of branches that people may not like)

It is worth mentioning that we don't anticipate using this workflow often, but feel it is necessary considering our plans for release streams.

Metadata Update from @jforbes:
- Issue tagged with: meeting

Jun 20, 2019

bowlofeggs commented Jun 20, 2019

Keep in mind that branches live forever, so this may result in a lot of branches with confusing names (A, B, C, etc.) that packagers may not understand the purpose of.

ngompa commented Jun 21, 2019

I'll be honest, this seems like you're doing it wrong. You guys are creating all kinds of weird workflows that not only don't make sense, but add confusion to packaging and pressure on the infrastructure.

I'd actually suggest banning this entirely. Instead, if you need updates, get them the same way everyone else does and spin a new OSTree and push it out.

Isn't that the point of RPM-OSTree?

jlanda commented Jun 21, 2019

What is the lifecycle for those backports and who will keep them updated?

Continuing with the kernel example, let say we have kernel-backport-a and kernel-backport-b and appears a new speculative attack or so, who will keep those backpprts updated? And until when? Do we have to eforce maintainers to keep updating n more branches for y more time?

bgilbert commented Jun 21, 2019

To @ngompa's point, there was some followup discussion in the original bug that might help explain the thinking here. The current plan is: every two weeks, the Fedora CoreOS testing stream snapshots Bodhi updates-testing, and the previous testing snapshot (plus any fixes) promotes to stable. During one of those two-week periods, in order to minimize risk, we don't want to take large changes to either the testing or stable streams. Thus, if a security update or bugfix is needed, we might need to backport fixes rather than taking a new upstream release of a package.

@jlanda The lifecycle for these backports would generally be 2-4 weeks, and they'd be maintained by the Fedora CoreOS team. There should be no burden on package maintainers other than seeing the extra branches in dist-git.

If anyone has better ideas for the Git branch structure, I'd love to hear them. I'm not thrilled with the coreos-backport-A nomenclature and would prefer short-lived branches, but of course that won't work with dist-git.

Edited Jun 21, 2019 by bgilbert

jlanda commented Jun 21, 2019

@jlanda The lifecycle for these backports would generally be 2-4 weeks, and they'd be maintained by the Fedora CoreOS team. There should be no burden on package maintainers other than seeing the extra branches in dist-git.

Nice to hear this on one hand, but I think I missing something on the other.

~~The goal of this branches is to minimize delta upgrade sizes, but they have short lifecycles. Aren't we just deferring the big delta size? Or you want to divide the big delta size due to many big package updates on a certain fortnight?~~

~~Let suppose that the kernel had a big update and needs a backport branch, so fortnight 1 upgrades goes with the first update using that branch instead of going with the updated kernel. Then the next fortnight we continue with the same branched package, and now we reach the lyfecycle end of that branch. What will we do? Just revert to standard stable branch? Does not the big delta that we wanted to avoid on fortnight 1 appear on fortnight 3? with additional updates and maintenance work during this 6 weeks? Is not the overall delta bigger in this case?~~

I wrote all that due to the "we'd like to deliver the smallest delta possible to users" but after reading the github issue the underlying reason seems to be another

Edited Jun 21, 2019 by jlanda

dustymabe commented Jun 21, 2019

I'll try to address some questions/points that weren't already covered by @bgilbert:

@bowlofeggs
Keep in mind that branches live forever, so this may result in a lot of branches with confusing names (A, B, C, etc.) that packagers may not understand the purpose of.

"Keep in mind that branches live forever"
This is why we're suggesting to only use the coreos-backport-A and coreos-backport-B branches and re-use them whenever we need new backports. Realistically we'd only ever need coreos-backport-A but we've come up with cases where we may need more than one backport at a time so we compromised and would like to name the branches with A and B.
"that packagers may not understand the purpose of"
This concern is exactly why we've discussed this as a solution and are now here talking about it with FESCO. We want to find the best solution and also raise awareness of these new branches if that is what we end up doing.

@jlanda
I wrote all that due to the "we'd like to deliver the smallest delta possible to users" but after reading the github issue the underlying reason seems to be another

I hope I wasn't too confusing with that wording. My intent was to express that if there is a fix to be delivered to a stable stream that we deliver an update to that rpm that only includes the fix and not a bump in version. For example if the stable FCOS stream is still on kernel-1.1 and kernel-2.2 is currently in latest stable in fedora with the fix we'd like to backport the fix to kernel-1.1 and ship that as an update. When we do our next regularly scheduled release we'll ship kernel-2.2.

jforbes commented Jun 21, 2019

I would feel better if it were a single coreos-backport branch. If you need to pull in a 2nd patch, it would still be on the same branch wouldn't it? I don't see much benefit to multiple branches here.

bgilbert commented Jun 21, 2019

@jforbes For fast-moving packages, there might be a separate base for each stream. In the worst case, each of stable, testing, next, testing-devel, and next-devel would be distinct. That seems most likely with the kernel, and less likely with other packages.

churchyard commented Jun 21, 2019

Quick idea: is it possible to push a tag and build in Koji from that commit/tag, without having a branch?

churchyard commented Jun 24, 2019

Here is my alternate proposal:

do a commit on top of the specified release branch such as f30
tag the commit as f30-coreos-backport- (or similar)
push the tag only
build from the tag or its commit hash
tag remains forever (if not, make the necessary changes in dist git to make it so)

If building from commit hashes/tags is impossible, create a temporary branch to be deleted by releng after the build (and since the tag preserves the commit, it should be allowed).

churchyard commented Jun 24, 2019

If we don't want the branches to be deleted at all, we could move their heads to refs/archive/, so that they will remain in the repo, will not be garbage collected by git, but will not be cloned by git by default. As described in https://pagure.io/releng/issue/7265#comment-499371

dustymabe commented Jun 24, 2019

Quick idea: is it possible to push a tag and build in Koji from that commit/tag, without having a branch?

Indeed that is a preferable solution from my perspective as well. The proposal we came up with was admittedly constrained to what we know can already be done with dist-git/koji. Can we get some experts to comment on the feasibility of https://pagure.io/fesco/issue/2152#comment-579010 ?

dustymabe commented Jun 25, 2019

Can we get some experts to comment on the feasibility of https://pagure.io/fesco/issue/2152#comment-579010 ?

tagging a few people who may be able to give us more information: @kevin @mohanboddu @sgallagh @lsedlar

zbyszek commented Jun 25, 2019

coreos-backport-A and coreos-backport-B

This was already mentioned in the IRC discussion, but not in the ticket here:
it is not possible to "rewrite history" in dist-git, so if coreos-backport-X is forked from fY branch, before coreos-backport-X reused, if there were any changes on fY , it'll be necessary to merge the fY into coreos-backport-X. This will result in painfully ugly history and conflicts in the merge (for example very frequently in %changelog).

Because of this, I think a solution that uses "fresh" tags or branches for the coreos builds will work better. Also, I think a meaningful name for the branch or tag is more important than conserving the number of names.

dustymabe commented Jun 25, 2019

coreos-backport-A and coreos-backport-B

This will result in painfully ugly history and conflicts in the merge (for example very frequently in %changelog).

Indeed. This is one drawback of the original proposal in this ticket. We were trying to live within the restrictions of dist-git where we can't delete branches, which means we could get a pile of branches over time that maintainers would not like.

Because of this, I think a solution that uses "fresh" tags or branches for the coreos builds will work better. Also, I think a meaningful name for the branch or tag is more important than conserving the number of names.

I agree. If we could use tags with meaningful names and possibly even dates in the name, it would make me happier. For example a tag of 20190625-fedora-coreos-backport-cveAABBCC or we could even make the tag be the NVR where we modify R to include coreos-backport.

dustymabe commented Jun 25, 2019

or we could even make the tag be the NVR where we modify R to include coreos-backport.

A bit of a tangent, but I think it would be really nice if the tooling around dist-git/koji would automatically create a tag for an NVR (unique in koji) in a dist-git repo. Would make it easier to browse the repos without having to inspect that information from koji.

bookwar commented Jun 25, 2019

Also, I think a meaningful name for the branch or tag is more important than conserving the number of names.

+1 here

I don't understand the problem with "polluting repo with branches". What exactly are we polluting? Afaik for git additional branch costs about the same as additional tag.

And if the workflow is about creating a branch for each backport - then we should be honest to ourselves about it an create those branches in the repo.

I would create namespaces for branches of the same type, like here coreos-backport-*. Or backport/coreos-* or whatever for easy filtering. But that's it.

And if Pagure UX is an issue here, then we should file a feature request to Pagure to handle branch lists in a better way.

dustymabe commented Jun 25, 2019

I don't understand the problem with "polluting repo with branches". What exactly are we polluting? Afaik for git additional branch costs about the same as additional tag.

As a packager I find it confusing when there are more branches in a repo than the fxx branches. The mental model maps very nicely to releases. I've definitely gone to repos where maintainers accidentally pushed new branches and been thrown for a loop momentarily. Our original proposal was assuming if we have many coreos-backport-* branches it would be undesirable to the maintainers of those packages.

bookwar commented Jun 25, 2019

If I were the maintainer, I'd say branches are more desirable then hidden workarounds. git has autocompletion for branch names and if there is a naming convention in place then it should be manageable.

I understand that old branches that aren't updated in years are annoying, but for me it feels like a generic dist-git problem, and trying to solve it it one particular case via very special project-specific approach is wrong.

I think we should get a procedure to archive unused branches. The idea from SO for example is to replace refs/branch/<name> with refs/archive/<name>. This will allow us to keep branch name, but to hide the branch from the obvious refs list into a separate namespace.

But then it should be a common dist-git policy.

dustymabe commented Jun 25, 2019

I'd say branches are more desirable then hidden workarounds.

Just want to clarify, no one is proposing any hidden workarounds. If we choose to use coreos-backport-A and coreos-backport-B we'd be using two branches, but the history would be ugly. It's perfectly within the capabilities of git to make this happen, even if it is ugly.

Alternatively if we choose to make a new branch for each backport, it's not any less of a hidden workaround, it's just ugly in a different way (multiple branches vs ugly branch history).

bookwar commented Jun 25, 2019

I don't really agree here. I mean we can call it both ugly, no problem with that, but I think multiple branches approach is a straightforward implementation of the idea of adding patch to a certain state(this is a definition of the branch), while intertwined branches is an additional layer on top of that. It is created to overcome the technical difficulties (too many branches to manage) which appear when you implement idea as is. So the first one is a direct solution and the second is workaround.

It would be just a wording issue, but there is more to it:

Intertwined branches require advanced levels of git-fu when resolving merge conflicts or navigate history. It is not smth you can ask random contributor to handle easily. There are all sort of weirdness happening with such long lived branches, simple git operations won't work on them. For example it is quite painful to look for common ancestors of two patches, when you have a branch which merged in itself several times. And then someone will surely merge it in a wrong way, removing half of the changes done in the main branch while resolving a conflict.

Multiple branches require more typing when doing git checkout and git pull. It is annoying, but it is simple. There are no additional knowledge requirements. There are no loops, no merge conflicts, no complexity. It is just a large tree.

dustymabe commented Jun 25, 2019

Intertwined branches

This term might be a bit overloaded. We'd only ever create/use the B branch if we needed two backports at the same time. The A and B branches have nothing to do with one another.

Intertwined branches require advanced levels of git-fu when resolving merge conflicts or navigate history. It is not smth you can ask random contributor to handle easily.

We'd never ask any users other than Fedora CoreOS team members to touch these branches. In fact we should probably have an ACL that prohibits it.

Either way let's table this particular tangent for now and wait to see what we get back from https://pagure.io/fesco/issue/2152#comment-579344

Edited Jun 25, 2019 by dustymabe

jforbes commented Jun 25, 2019

Intertwined branches

This term might be a bit overloaded. We'd only ever create/use the B branch if we needed two backports at the same time. The A and B branches have nothing to do with one another.

I guess I am still not seeing where the B branch would come in. These are for critical backports. If you backport A as a stop gap until the next rebase, and issue B comes up, under what circumstances would you want a B without A? Seems from a source branch, you only need A, and when B comes out, it is backported on top of A. I realize the binary build of A may not be done pushing, but that doesn't change the source.

dustymabe commented Jun 25, 2019

@jforbes
I guess I am still not seeing where the B branch would come in. These are for critical backports. If you backport A as a stop gap until the next rebase, and issue B comes up, under what circumstances would you want a B without A? Seems from a source branch, you only need A, and when B comes out, it is backported on top of A. I realize the binary build of A may not be done pushing, but that doesn't change the source.

How we ended up here started at https://github.com/coreos/fedora-coreos-tracker/issues/79#issuecomment-473426063 . Mind taking a look to see if it helps clear things up? If not, maybe we could carry that discussion forward in the ticket.

dustymabe commented Jun 26, 2019

@churchyard
Quick idea: is it possible to push a tag and build in Koji from that commit/tag, without having a branch?

@dustymabe
Indeed that is a preferable solution from my perspective as well. The proposal we came up with was admittedly constrained to what we know can already be done with dist-git/koji. Can we get some experts to comment on the feasibility of https://pagure.io/fesco/issue/2152#comment-579010 ?

So it seems like this isn't too far off from being able to be done I can push a tag to a repo. I chose an old dead repo and did:

git checkout HEAD^
git commit
git tag test
git push HEAD:refs/tags/test
fedpkg --release f30 --debug --verbose build

I got a TypeError: HEAD is a detached symbolic reference error:

[dustymabe@media fedora-productimg-cloud ((test))]$ fedpkg --release f30 --debug --verbose build                                                                                                                                                                                                                                                                          
Creating repo object from /var/b/shared/code/src.fedoraproject.org/rpms/fedora-productimg-cloud                                                                                      
Could not determine the remote name: Cmd('git') failed due to: exit code(1)                                                                                                          
  cmdline: git config --get branch.f30.remote                                                                                                                                                                                                                                                                                                                             
Falling back to default remote name 'origin'                                                                                                                                         
Package fedora-productimg-cloud has stream branches: []                                                                                                                              
Logout kojisession                                                                                                                                                                   
Initiating a koji session to https://koji.fedoraproject.org/kojihub                                                                                                                                                                                                                                                                                                       
Logging into https://koji.fedoraproject.org/kojihub with Kerberos authentication.                                                                                                    
Could not execute build: HEAD is a detached symbolic reference as it points to 'a44d8d305dfdb376e2e7c0eebeb341f94b662255'                                                                                                                                                                                                                                                 
Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                        
  File "/usr/bin/fedpkg", line 11, in <module>                                                                                                                                       
    load_entry_point('fedpkg==1.37', 'console_scripts', 'fedpkg')()                                                                                                                                                                                                                                                                                                       
  File "/usr/lib/python3.7/site-packages/fedpkg/__main__.py", line 86, in main                                                                                                       
    sys.exit(client.args.command())                                                                                                                                                                                                                                                                                                                                       
  File "/usr/lib/python3.7/site-packages/pyrpkg/cli.py", line 1731, in build                                                                                                         
    task_id = self._build(sets=sets)                                                                                                                                                                                                                                                                                                                                      
  File "/usr/lib/python3.7/site-packages/fedpkg/cli.py", line 1090, in _build                                                                                                        
    return super(fedpkgClient, self)._build(sets)                                         
  File "/usr/lib/python3.7/site-packages/pyrpkg/cli.py", line 1788, in _build                                                                                                        
    fail_fast=self.args.fail_fast)                                                                                                                                                   
  File "/usr/lib/python3.7/site-packages/pyrpkg/__init__.py", line 2242, in build                                                                                                                                                                                                                                                                                         
    self.check_repo()                                                                                                                                                                                                                                                                                                                                                     
  File "/usr/lib/python3.7/site-packages/pyrpkg/__init__.py", line 2171, in check_repo                                                                                                                                                                                                                                                                                    
    branch = self.repo.active_branch                                                                                                                                                 
  File "/usr/lib/python3.7/site-packages/git/repo/base.py", line 696, in active_branch                                                                                               
    return self.head.reference                                                                                                                                                                                                                                                                                                                                            
  File "/usr/lib/python3.7/site-packages/git/refs/symbolic.py", line 275, in _get_reference                                                                                          
    raise TypeError("%s is a detached symbolic reference as it points to %r" % (self, sha))                                                                                                                                                                                                                                                                               
TypeError: HEAD is a detached symbolic reference as it points to 'a44d8d305dfdb376e2e7c0eebeb341f94b662255'

If I disable the logic that does the check in the code, though I get a successful build.

If we go this route we'd need to modify rhpkg to support checking if the commit is in a tag. We'd also need to establish some sort of policy around pushing tags and who has access to them. i.e. if we need a backport against the foo package we need an acl to be able to push tags to the foo package.

Edited Jun 26, 2019 by dustymabe

churchyard commented Jun 28, 2019

FTR deleting tags is not allowed.

[fedora-productimg-cloud (master)]$ git push origin  --delete test
remote: Branch deletion is not allowed
remote: Denied push for ref 'refs/tags/test' for user 'churchyard'
remote: All changes have been rejected
To ssh://pkgs.fedoraproject.org/rpms/fedora-productimg-cloud
 ! [remote rejected] test (pre-receive hook declined)
error: failed to push some refs to 'ssh://churchyard@pkgs.fedoraproject.org/rpms/fedora-productimg-cloud'

(I'm a provenpackager and cvsadmin.)

churchyard commented Jun 28, 2019

[fedora-productimg-cloud ((test))]$ git push origin test 
To ssh://pkgs.fedoraproject.org/rpms/fedora-productimg-cloud
 ! [rejected]        test -> test (already exists)
error: failed to push some refs to 'ssh://churchyard@pkgs.fedoraproject.org/rpms/fedora-productimg-cloud'
hint: Updates were rejected because the tag already exists in the remote.

psabata commented Jun 28, 2019

From today's FESCo meeting:

  * AGREED: Let's use tags. The exact workflow is up to the CoreOS team.
    (+8, 0, -0)  (contyk, 15:47:52)

Metadata Update from @psabata:
- Issue untagged with: meeting
- Issue close_status updated to: Accepted
- Issue status updated to: Closed (was: Open)

Jun 28, 2019

dustymabe commented Jun 28, 2019

Thanks everyone for the discussion.. Our plans going forward are laid out here

Metadata

Assignee

None

Tags

None

Blocking

None

Depending on

None

Priority

None

fesco

#2152 "backport" branches in src.fp.o for backports to coreos stable streams Closed: Accepted Jun 28, 2019 by psabata. Opened Jun 20, 2019 by dustymabe.

Metadata

#2152 "backport" branches in src.fp.o for backports to coreos stable streams

Closed: Accepted Jun 28, 2019 by psabata. Opened Jun 20, 2019 by dustymabe.