As a practical matter, it's not very hard to do this. However, it's a horrific idea since Git is not a stable source of truth.

But sure, it can be done easily enough:

1. Devise a URL scheme for VCS checkouts to use with Source
2. Create a prep macro that uses that URL scheme to do checkouts from source control properly

You're pretty much done after that. But as I noted earlier, it's a horrid idea because Git is a highly mutable change set tracker.

You'll also lose an important aspect of Source RPMs: archiving all the inputs of the build.

The goal of the brainstorming session would be to see if we address these points :)

Note, I'm ok with killing the idea from the get go if we see no value in it

@pingou what would be the dream outcome of this session for you?

I’d say it’s similarly horrible as having Source point to some random URL and leave it at that. Which we don’t :wink:!

To me the problem boils down to providing that stability ourselves: We need information about all artifacts necessary to produce a built package and keep (copies of) these artifacts under our control.

Traditionally this means something like (not telling you any news here, just to illustrate): a package maintainer downloads the source tarball, uses fedpkg new-sources to upload it into the lookaside cache (now it’s under our control) and store a hash of this copy in the sources file (which “blesses“ this copy as what should be used for building). This allows us to retain reproducibility even with upstreams swapping tarballs of a version with one with different content (oops).

My thoughts on this are not more than half-baked, so there might be holes – but I don’t see that we can’t make a virtue out of necessity and give people a fedpkg new-sources equivalent which a) takes care of that the current state of where the git URL points to in this moment(*) is and remains available to the build system (essentially telling koji to pull from the repo plus hash to a "lookaside repo"), and b) declaring this state as what should be built from in sources. This obviously needs a way to cast the worktree of that state of the git repo into an archive of some kind so rebuilding SRPMS works, but my gut feeling is that this is solvable.

(*): I guess this should be a local worktree for practical purposes, I don’t imagine people want to specify hashes when doing that, and blindly trusting a remote repo doesn’t feel safe to me.

@ngompa

You'll also lose an important aspect of Source RPMs: archiving all the inputs of the build.

I think the good start will be to list more requirements like this and ask ourselves why we need them.

So that we can switch from "building from git is not possible because you lose a feature X" to "if we are to build from git, we need to find a way how to keep this feature X".

And for example, archiving all inputs for the build can be done: if you build from git, it doesn't necessarily mean you skip the SRPM stage.

@pingou what would be the dream outcome of this session for you?

I think it will be multi-fold:

• a list of requirements for such system/idea to be viable (ie: we need to be
  able to reproduce a build in 6 months)
• a list of potential solutions (and thus a list of remaining challenges)
• a start of understanding of the fall out of this idea (ie: what would be the
  impact in working with git repos instead of tarball be for packit/source-git?)

Maybe we'll need to keep a copy of the source git repo for build
reproducibility, maybe we can just update that copy when working with the
source-git workflow, maybe we create a tarball of the git repo and upload that
to the lookaside cache, thus giving us a near-today experience.
I'm sure we can find a bunch of ideas, not all of them would be good, but that's
the goal of a brainstorming session: finding all the ideas first and then
filtering them to see which stand out.

Our build system archives the SRPMs. I don't think any approach in this area would change that at all. In fact, if we generate Git bundles on the fly instead of tarballs, we can preserve more information in the archival system, and make that also available quasi-indefinitely to end users.

Internally, we already have a variant of fedpkg-simple that can run git archive to produce tarballs instead of downloading them from the lookaside cache. It uses a Git URL and a commit reference that it finds in a source-repos file. But I think this is a bit of a lose-lose situation:

• If we put a commit hash into that source-repos file, we have to update dist-git with the commit hash each time we push something to the source repository. That extra step seems unnecessary.
• If we use a symbolic reference, it's not really obvious what we are building from the Koji logs and dist-git contents that's referenced from Koji data. That can be fixed by putting git rev-parse output somewhat, I guess.

But I think it would be simpler to avoid the repository separation. But in that case, we really need to move gating.yaml and subdirectories such astests, plans under a magic, neutral subdirectory (such as _rpm), so that we don't step on upstream stuff. Maybe we need to do this for the RPM spec file as well. I don't think Koji needs to know about this (only fedpkg-simple), but other tools that extract data from dist-git will need to adapt. Separate repositories for upstream-like sources would avoid the need for this work, but I really dislike the complexity that comes with that.

I think there is merit to what @pingou is proposing here. I also think @bookwar has a great point in that we would still have SRPMs.

As a package maintainer and upstream maintainer, I would like to see tighter integration with upstream repos.

Slightly tangential, but I get jealous of the kernel guys in that upstream changes are streamlined to rawhide rpms pretty quickly via kernel-ark. Sometimes I hear about an enhancement to the kernel that got merged 6 months ago and after a "uname -r" I realise it has already perculated to my machine.

I think we could learn a lot from kernel-ark. There's a bit of overhead to updating a package in comparison to contributing something upstream. This overhead is enough to cause fedora rpms to be years behind upstream in some cases. Of course, something like this should not be forced on the package maintainer, they should have the autonomy to opt-in to a system like this if they see fit for a particular package.

The flip side here is that Debian already does it this way and they're still slow at updating software. It's not a question of tooling or workflow to have something released faster, it's entirely a question of whether the maintainer wants to.

I was not able to find tickets at a reasonable price, so I will not be able to attend hatch in Cork :(