CC: @breilly

Do you have an example for the EPEL8 failures?

I see a recent failed ELN build, as you say it times out, during the wait repo specifically: https://release-engineering.github.io/mbs-ui/module/14762/components
https://koji.fedoraproject.org/koji/taskinfo?taskID=89783434
@kevin any known issue with koji's createrepo channel, or kojira, at the moment that could cause this?

We have a mass rebuild running right now, so koji is pretty busy, but kojira should be running and processing as normal...

ELN builds are failing because F34 was EOL-ed and module-eln-build Koji tag still refers to f34-build external repo:

$ koji taginfo module-eln-build
Tag: module-eln-build [31229]
Arches: 
Groups: 
Tag options:
External repos:
    5 f34-build (https://kojipkgs.fedoraproject.org/repos/f34-build/latest/$arch/, merge mode: koji), arches: inherited from tag
Inheritance:
  0    .... module-package-list [412]

Koji can't create modular buildroots because external repo f34-build is no longer available.

Here are the latest 3 EL8 builds that failed:
https://release-engineering.github.io/mbs-ui/module/14769
https://release-engineering.github.io/mbs-ui/module/14759
https://release-engineering.github.io/mbs-ui/module/14753

They failed due to:

An unknown error occurred while validating the modulemd

EL8 failures are caused by a different issue that ELN failures.
I've fixed an issue in MBS (incorrect filesystem permissions on mbs-backend), please retry your failed EL8 module build(s).

@mizdebsk what permission was it? can we/should we add it to ansible?

@mizdebsk what permission was it? can we/should we add it to ansible?

On mbs-backend01 some (but not all) files under /tmp/mbs/dnf/module-el8-build-37/ directory were owned by root:root instead of fedmsg:fedmsg. I don't know how that happened. These files are not ansible-managed, so there is nothing to fix in ansible. I removed the whole directory manually and let MBS recreate it - now it has correct ownership.

This issue seems to be resolved, if the problems persist please reopen this ticket.