From my perspective, Koji is already difficult enough to get up and running. Introducing Kubernetes would make this considerably worse.

Not to mention, changing to Kubernetes means that we no longer have a model for scaling up from the simple case (local developer builds with mock) to the distro case (doing lots of these at a time).

The more complex you make Koji, the less adoption among shops for building software you will have. Finally, Kubernetes introduces a resource floor that is ridiculously high for useful deployments. For example, with OpenShift Kubernetes, there are no useful deployments below mid-range server class hardware (>24 cores of CPU, >128GB of RAM, terabytes of storage, etc.)

I would rather not do this.

Not to mention, changing to Kubernetes means that we no longer have a model for scaling up from the simple case (local developer builds with mock) to the distro case (doing lots of these at a time).

Not at all, one would just use a tool that runs inside a podman container, or for convenience mock could become a simple unprivileged wrapper script for doing this.

While I certainly appreciate the appeal here, I do not think that such a drastic redesign is feasible.

Not all Koji tasks run in containers, and not all can run in containers. While I do think that Koji might be able to learn to make use of a Kubernetes cluster for some work, I really don't think we can handle all tasks that way. I also share Neil's concern about the increased setup requirements.

It is important that Koji continue to be able to replicate previous build environments. "Stop using mock" is a non-starter in that regard, as it is the primary tool Koji has used for this since day 1. That said, adding new ways to perform builds is viable.

"Stop using mock" is a non-starter in that regard, as it is the primary tool Koji has used for this since day 1.

I think it'd be quite doable to have legacy build environments run mock inside a privileged container. But bear in mind, mock is insecure - because it constructs the buildroot outside of a container, all package scripts run as real root in the cluster.

Another way to look at this - by far the dominant way most people do upstream CI is yum install foo-devel bar-devel inside a podman/docker container (or in Kubernetes) today. That is "securable" (although, there have and will continue to be container breakouts, it's way, way better than just running package scripts as root outside of any sandboxing at all).

Edit: And to flesh this out a bit more, for maximum security, one can deploy e.g. kata containers - which then makes the architecture similar to COPR, which is also a bespoke Linux container and VM scheduling system specialized to RPM that could also just use Kubernetes (and Kata).

Securing Mock is also possible without going through this, because we could easily extend Mock around our usage of systemd-nspawn. And unlike the current OCI tools, Nspawn is considerably easier to use for doing a lot of the types of builds Koji does.

And unlike the current OCI tools, Nspawn is considerably easier to use for doing a lot of the types of builds Koji does.

You've provided no references or data, so this seems like a baseless assertion.

And unlike the current OCI tools, Nspawn is considerably easier to use for doing a lot of the types of builds Koji does.

You've provided no references or data, so this seems like a baseless assertion.

Nspawn handles working with directory-based bootstrap as well as rootfs based bootstrap. Nspawn also handles exposing the necessary devfs stuff for image builds properly. Nspawn can automatically create private/fake subset fstrees for home, tmp, run, proc, etc. that doesn't break package and image builds.

I have had numerous problems trying to replicate our image builds in OCI-style containerized environments, and frankly it's not worth my time to dig into the mess that is the OCI circus to make it work.

I'm interested in building and testing a proof-of-concept for this, at least. I think this could be a good future for Koji.

Colin, it would be worthwhile to start discussion with the mock team, too. I've been wondering for a while what Mock's future is w.r.t OpenShift. In https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahosted.org/thread/NWJU3MS422DYD2WQAEWYOO7BVWHIPNSY/ Leonardo had to implement kubevirt for builders, since mock could not run unprivileged.

(Also, we need to figure out how this system would build RPMs for multiple arches. From what I understand this has been a problem with OSBS, since one OpenShift cluster cannot be multi-arch)

I'm interested in building and testing a proof-of-concept for this, at least. I think this could be a good future for Koji.

👍

I think it's:

• "stub out code build an rpm in a pod" (low difficulty, modulo nfs configurability)
• Add option to have koji hub connect to a kubernetes cluster (or, possibly multiple; this intersects with the multi-arch thread) (seems like low difficulty, though I have no experience with python clients)
• Dive into the mechanics of scheduling the build as a pod, monitoring it and getting results back (this is the bulk of the work, but...in the end what's going on here is not complicated)
• Then it's tying up various loose ends to "productize"

Colin, it would be worthwhile to start discussion with the mock team, too. I've been wondering for a while what Mock's future is w.r.t OpenShift.

I don't think it makes much sense to carry forward long term with hard requiring mock-the-bespoke-container-engine-wrapper to build RPMs. Some stuff like the definition of the buildroot lives in executable Python code in the mock source - that stuff would need to be extracted and put in a saner text format.

since one OpenShift cluster cannot be multi-arch

https://github.com/openshift/enhancements/pull/1014

I have had numerous problems trying to replicate our image builds in OCI-style containerized environments, and frankly it's not worth my time to dig into the mess that is the OCI circus to make it work.

I have extensive experience in this; the production Fedora CoreOS disk images are generated in a fully unprivileged OpenShift pod, just with /dev/kvm set up. Now, it's certainly true that these types of image build tools can get tied to (and tested in) a specific execution environment or framework. Some of them often want to use container features on their own - in fact, rpm-ostree wants this. So what we end up doing is running it in a transient VM too.

The short term out is to just have these things run as privileged containers, or keep the existing koji builder flow targeting VMs. I think about 70% of the benefit is moving all the RPM workload into Kubernetes then that same hardware/clusters can be shared to build things that are not RPMs (e.g. native container images).

I've also been thinking of RPM building recently, though haven't really been ready to make a formal proposal. However, I thought I'd throw this out here now that the topic is under discussion:

How about using hosted image builder to build RPMs? We are already building images, ostree commits, and containers - RPMs aren't really very different.

The benefits would be that no changes would be required to koji, and there would be no new service to deploy/host, as everything would be hosted by the image builder team in api.openshift.com. Image builder already needs to deal with building in VMs, so whether or not we can build RPMs in containers is not a concern, and we already need to run across all the four main architectures, so that is also not a concern.

What exists today:
- Image builder has koji integration (as a content generator), which is used by RHEL and currently being onboarded by Fedora
- There are koji plugins to enable a new osbuildImage command along side the existing image one.

What is on the roadmap:
- The same, but for base containers.

What we would need:
- osbuild would need to learn how to build RPMs in a mock-compatible way
- we would need to extend our plugins to add a new osbuildRPM command

• osbuild would need to learn how to build RPMs in a mock-compatible way
• we would need to extend our plugins to add a new osbuildRPM command

This seems to me like mixing up two completely separate layers - osbuild is for generating OS images, which are composed of RPMs, while koji is for building RPMs. What benefit would we get from using a single system for both?

This seems to me like mixing up two completely separate layers - osbuild is for generating OS images, which are composed of RPMs

osbuild is a generic framework for building operating system artefacts. The plugins (stages) we created so far were mostly focused on building images from RPMs, but we also support building from Arch packages, and we support building ostree commits and OCI containers. Technically speaking RPMs are not really different. We can build virtually anything as long as the inputs are content addressable and the stages are (reasonably) deterministic.

while koji is for building RPMs.

koji also builds images (and other artefacts), though in RHEL this is outsourced to image builder (and similar work for Fedora is in the works). My proposal would be to simply extend the logic to cover RPMs as well.

What benefit would we get from using a single system for both?

What I can imagine from the top of my head:
- outsourcing RPM building would make running koji itself simpler
- sharing the code and infrastructure of image builder would mean all the performance / scalability work that goes into building images would benefit RPM building as well
- having a service outside of koji build RPMs would make it possible to use the same service for other services that need to build RPMs but cannot go through koji.

Koji itself is rather simple to deploy, provided you don't want to solve the authentication problem. Koji's currently supported authentication methods are not very easy to setup. If you skip auth, it's pretty easy. Beyond that, pretty much every proposal here makes Koji quite a bit more difficult to set up and maintain.

It doesn't make sense to authenticate to koji at all. A build system should just react to commits to git, and commits to git should trigger builds. In very unusual circumstances a buildsystem administrator could pause/hold things out of band (also ideally done via committing that desired state into a git repository).

It doesn't make sense to authenticate to koji at all. A build system should just react to commits to git, and commits to git should trigger builds. In very unusual circumstances a buildsystem administrator could pause/hold things out of band (also ideally done via committing that desired state into a git repository).

This process is now outsourced to various CI systems using koji (e.g. koschei in fedora). They are monitoring git repos, enforcing possible rules, monitoring changed dependencies and starting builds (they know better if they should just start scratch build or regular build, etc.) In this workflow CI system is normal koji user, so it authenticates against koji and spawning builds on their behalf or on behalf of user who did the commit. Anyway, build is always "owned" by somebody, so authentication is required from security/audit reasons.

We stop using e.g. mock, and the generated pods just do yum builddep && rpmbuild inside them.

The very idea that this is achievable is IMHO quite flawed. Have you ever tried this? It's so much more complex than that. Consider for example %generate_buildrequires, but there is more, so much more. It is certainly possible to implement machinery that does this, but you will most likely end up reimplementing (a substantial part of) mock from scratch.

It doesn't make sense to authenticate to koji at all. A build system should just react to commits to git, and commits to git should trigger builds. In very unusual circumstances a buildsystem administrator could pause/hold things out of band (also ideally done via committing that desired state into a git repository).

This process is now outsourced to various CI systems using koji (e.g. koschei in fedora). They are monitoring git repos, enforcing possible rules, monitoring changed dependencies and starting builds (they know better if they should just start scratch build or regular build, etc.) In this workflow CI system is normal koji user, so it authenticates against koji and spawning builds on their behalf or on behalf of user who did the commit. Anyway, build is always "owned" by somebody, so authentication is required from security/audit reasons.

Most of the interaction with Koji is human, not machine. In general, Fedora and Red Hat have rejected the idea of autobuilding and autorebuilding and managing that grunt work through machines, so this will continue to need to be handled by humans, so authn/authz is required for that.

The very idea that this is achievable is IMHO quite flawed. Have you ever tried this?

That's exactly the the thing. Take rpm upstream's own CI Dockerfile.

This pattern is extremely common. Today when one hires a new engineer, it's almost certain they have at least seen a Dockerfile before and have some familiarity with now industry standard container flows. It's much, much less likely that they have any familiarity with mock.

And the benefits go beyond this - if Koji generates buildroots as standard containers, it becomes easy to e.g. pull that exact container locally to replicate a problem. I have wanted the ability to do this so many times.

It's so much more complex than that. Consider for example %generate_buildrequires,

Sure, that just maps to running some code and running yum install again.

but there is more, so much more.

What else? Now don't get me wrong, I am definitely sure there are load-bearing parts of mock that would need to be carried into this world. But the point is that there's a lot of other parts of mock in terms of being its own bespoke Linux container tool that would go away.

In general, Fedora and Red Hat have rejected the idea of autobuilding and autorebuilding

Fedora perhaps, but not Red Hat. Remember, I wear two hats - I also work on OpenShift. And to ship code to the product today, all an engineer needs to know is Go, Dockerfile (and containers), and the ability to submit PRs to GitHub and merge them. There are no spec files involved. And there is no manual invocations of koji build required once a change merges to e.g. a git repository, that's all handled by automation.
(Now of course, I and others regularly have to transition between these two worlds, and it is jarring)

OpenShift is the only product that works that way. All the other layered products, as well as RHEL itself, do not. So no, I disagree with your assertion that Red Hat has embraced the idea.

outsourcing RPM building would make running koji itself simpler

It took me quite a while to think about this. The "outsource all builds to a remote hosted service" is a pretty profound change. (Is this true in CentOS stream today?) It has a lot of implications; for example, a major one is that today the Fedora-derivative family of distributions is defined to be self-hosting. It seems (at least potentially) harder to accomplish changes when builds are handled by a separate service operating on its own lifecycle.

The overlap between this osbuild proposal and koji seems really high across the board. For example osbuild also has its own container flow (AIUI) - would it use mock or try to do mock inside its own containers?

Bigger picture though, my interest here is in having our build tooling work inside standard container flows (podman/Kubernetes/OpenShift) for the reasons mentioned above.

It took me quite a while to think about this. The "outsource all builds to a remote hosted service" is a pretty profound change. (Is this true in CentOS stream today?)

We don't do this anywhere.

Outsourcing builds to remote services:
- We have content generators for that (exactly what osbuild does for images)
- https://koji-planning.readthedocs.io/en/latest/Proposals/kbi.html - this is the idea of decoupling mock from koji itself to behave more like "plugin". This could be replaced by container building (it works +- as I've written PoC few years ago). But there was not that much call for such behaviour in those days, so we've abandoned it in that time. Container was basically preinstalled buildroot. Nevertheless, it was rough work and we would have to solve all the issues mock already solved (binding devices, kernel building issues, ...). So it would work for "basic" packages but not for every of them without (probably) significant work. Whole ABI work is postponed after the scheduler rewrite (which should finally happen in few months) as it a) will affect design b) there is simply not enough time to work on that.
- https://pagure.io/kube-sig/koji-operator is place where this meets together. Containerized deployment of koji. It uses standard builder with mock (but we've discussed ABI approach) for now but it means spawning VM (via kubevirt) just to run mock inside (as mock is still not able to run in unprivileged container). @lrossett can provide more details here. demo

It's so much more complex than that. Consider for example %generate_buildrequires,

Sure, that just maps to running some code and running yum install again.

Everything just maps to running some code, but the code needs to be quite robust. E.g. dynamic buildrequires are more complex than "running yum install again", see:

1. just running yum install again with a fixup
2. how it actually needed to be changed with a fixup

but there is more, so much more.

What else? Now don't get me wrong, I am definitely sure there are load-bearing parts of mock that would need to be carried into this world. But the point is that there's a lot of other parts of mock in terms of being its own bespoke Linux container tool that would go away.

What else? See e.g. https://github.com/rpm-software-management/mock/blob/mock-2.16-1/mock/py/mockbuild/backend.py#L257 and see what it does, except ignore all the container things.

BTW Mock can run with --isolation=simpleto not be container-based. https://rpm-software-management.github.io/mock/#mock-inside-podman-fedora-toolbox-or-docker-container

Last time I tried to run mock with --isolation=simple it still required a privileged pod/container but I don't think it makes much sense to run mock in a container, even if it is running in "container-less mode", when it could run those tasks in a pod.

As @tkopecek mentioned, we have a Koji operator that is WIP but is already able to build a package by provisioning VM based builders using kubevirt.

What I had in my mind is that implementing KBI would decouple mock from kojid (making its usage optional), so we would have a "mock kojid plugin" to keep things running as it is today and a new Kubernetes builder plugin that would not run mock at all during the build process.

We definitely can't build everything in a Pod but the operator can always provision a VM based builder if needed, providing different types of kojid deployments.

All operator issues are being tracked in this repository: https://pagure.io/kube-sig/kube-sig/issues.

As @tkopecek mentioned, we have a Koji operator that is WIP but is already able to build a package by provisioning VM based builders using kubevirt.

I was completely unaware of this. Nice! So...from my PoV I think we can probably just close out this issue in favor of tracking things over there?

I think so, as long we keep all Koji specific discussions (such as the Koji Builder Implementation) in this repository.

I've been trying to get more attention/contributors to the project for some time now, I would be more than happy :-)

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/3273

Please continue any further discussion there.