FYI, this is a result of releng#7260.

Hm, thinking how to fix that... Some thoughts:

• waitrepo takes long time, because we always create brand new repo for the module build. This is caused by a fact we create new koji tag X for each module, which inherits different koji tags. I think Koji generates single new repo for that koji tag X with packages from all the tags the X inherits. Therefore the repo is generated everytime.

Ways to fix that? I have just one sane idea right now:

• Change/Configure koji or MBS to enable base module (platform in our case) as a repo in a buildroot without creating new repo, but use the latest one. Let it create the repo just for modular dependencies (either remove platfrom koji tag from the inheritance list or do some koji code change).

Replacing inheritance with external repo would make new repos much faster - Koji won't need to read all RPMs from NFS, but only mergerepo f28-build repodata.

That sounds sane.

Perhaps we could mark modules for which this should be done in PDC under xmd somehow, to avoid adding platform-specific hacks to the code.

Just further info, @mizdebsk is using this setup in his private Koji instance: https://koji.lbzip2.org/koji/taginfo?tagID=3

This is how fast createrepo is when doing the repo this way: https://koji.lbzip2.org/koji/taskinfo?taskID=1

I'll recommend against koji code changes for this one. We should be able to get MBS to exclude platform from inheritance, but pull in the associated tag f28-build as an external repo instead.

/cc @mohanboddu, @tmlcoch, @rashmin, @ausil - I wanted you all to see this before we get further along. For releng:

• How does the proposed approach above sound to you?
• Any other ideas on how we could approach the problem?

Do you trigger regen-repo when the new tag is done and before you run the wait-repo?

If not then regeneration of the repo is done by a kojira service that periodically cycles through all repos and regen repos that have been updated. If there are a lot of repos, it might take some time to kojira to get to your repo = longer waiting on wait-repo.

An external repo might speed it up a bit, but I'm not sure if koji is prepared for massive usage of external repos because afaik, external repos couldn't be removed (you can remove them from a tag, but afaik a definition repo itself stays in koji forever even if it's not attached to any tag).

It would worth to consult that with guys from koji team.

/cc @mikem @tkopecek @breilly @mikeb ^^ koji wizards - do you have any input on the above?

As @tmlcoch mentions - kojira may be blamed here. If you want to make it faster, you've to trigger regen-repo manually, otherwise you'll have to wait for kojira to pick it.

For external-repos forever in db - I would say, it is similar as with semi-random tags used for modules. They are also in db forever, so cluttering db would be very similar.

How much time takes creatrepo task itself in this setup now? /In other words, is task slow, or just long waiting before it starts?/

@tmlcoch: @tkopecek: MBS triggers regen-repo manually, it does not wait for Kojira. The createrepo_c run just takes almost one hour to generate the repo with the koji tag we build every module in.

You can check the newRepo task started by MBS here: https://koji.fedoraproject.org/koji/taskinfo?taskID=24159853

You can compare that with the external-repo setup on @mizdebsk private Koji instance: https://koji.lbzip2.org/koji/taskinfo?taskID=1

This is becoming a blocking issue preventing us from reaching our deadlines. Is there any way we could hack something up quickly to get us going and solve it more correctly later?

The biggest issue here is that the way MBS works it reads all rpms from disk when creating new tags, which was fine when a module had a small set of RPMs, now that it has everything in them we have to redo how koji does repos or how MBS makes tags and have it work like traditional Fedora where we use a consistent tag for the buildroot

Consistent tag for the buildroot is not possible, because we need to be able to build some modules against different modules on top of that consistent tag, therefore the buildroot will have to inherit multiple tags (one can be common consistent tag, but others will be just different modular tags).

We could mitigate it by using the common tag per module stream, but it will still result in lot of big repositories.

@ausil: What do you think about the idea of using platform:f28 as external repository, so Koji just merges its metadata, but it does not have to read all RPMs and generate new repo from scratch?

@jkaluza that is clunky but could work.

I'm not concerned about the external repo definitions existing in the database, they're very small.

This looks like it will work, and should address the issue of slow repo regens. The downside is that it weakens the tracking of what's available in the buildroot for any given build. You now have to consider which repo was active at any given time for two tags, rather than one. There is also the possibility of the latest symlink in the external repo changing while the repos are being merged. The window for this is small, since all repodata is downloaded at once, but it's a possibility we should be aware of, since it could cause unexpected build failures.

But overall, +1, this sounds like a good solution.

So in the end we have found out with @mizdebsk that this might be just Koji configuration change:

We need the module-f28-build koji_tag to just have no inheritance at all, and one external repo configured to point to f28-build.

The same was as @mizdebsk is using here: https://koji.lbzip2.org/koji/taginfo?tagID=1 that's how our module-f28-build koji tag should look like.

Thanks all - I will make that koji configuration change.

Tag adjusted here: https://koji.fedoraproject.org/koji/taginfo?tagID=3307

One thing we forgot -> the package list. I'll modify the package list sync script to update this new tag independently.

On second thought, that script already updates the module-package-list tag.. so we should just establish inheritance with module-f28-build.

Should be all set now. Here's an example newRepo task: https://koji.fedoraproject.org/koji/taskinfo?taskID=24304274

Can we replicate this configuration in stage Koji and PDC?

Good catch @mprahl. Should be replicated in staging now.

(Technically, there is a platform-f27-4 module there which I had to use since there is no f28 in staging yet.. but it is close enough. I modified the testmodule yaml to reference this instead of bootstrap: master.)

@ralph can you please file a releng issue with the commands you ran so that they can get added to the mass branching sop

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/mbs/issues/830

Please continue any further discussion there.