it would fail every night when it ran so we disabled it: https://pagure.io/pungi-fedora/pull-request/626# . If someone has done some work to make it not fail then it would be great to add it back, but I really don't want to watch a notification come through every night for something we know is going to fail.

That being said.. they've been failing every night anyway because armhfp is failing so I'd almost be in favor of removing that one too. It might be a simple problem, but I haven't had the time to look into it.

https://pagure.io/dusty/failed-composes/issues?status=Open&search_pattern=Container

@dustymabe For the s390x I would propose to try re-enabling that as rawhide seems to be building just fine. What is you opinion?

@dustymabe For the s390x I would propose to try re-enabling that as rawhide seems to be building just fine. What is you opinion?

I'm happy to re-enable if we have someone to look at failures and try to get it fixed. Are you willing to be that person?

Also if you know of anyone that can look at armhfp that would be great.

AFAIK the main issue with image creation on s390x is the use of nested virt (z/VM guest used as KVM host) and the slowness of this. The RHEL rel-eng team had to increase timeouts (total timeout, disk inactivity timeout) multiple times and at the end they switched the image builder to its own LPAR (so no nested virt).

I think I heard from @imcleod that there was a way to make this better. @imcleod is that true?

Also if you know of anyone that can look at armhfp that would be great.

Yeah, it would be great to figure out the armhfp problem.

I think I heard from @imcleod that there was a way to make this better. @imcleod is that true?

Indeed, but it will require some infra help to implement.

Running KVM on a host that it itself inside of zVM is terribly slow. Runing KVM directly on an LPAR is not. Also, running nested KVM on top of an LPAR works. In ASCII art form

LPAR -> zVM -> KVM - Bad - Very slow
LPAR -> KVM - Great - very fast (5x-10x better than the above)
LPAR -> KVM -> KVM - almost as good (though I have not personally tested this)

So, in short, one thing that should help a great deal is moving the koji image builder host for Fedora onto one of the last two configurations and away from what it has historically been on (which is the first one).

Thanks, was wondering how LPAR -> KVM -> KVM looks like, I think this is the way for Fedora infra, switch to KVM for the whole builder LPAR.

@sharkcz - can you open a ticket to track the move to a better nested virt setup for s390x ?

@jcajka - do you want to open a PR for adding back s390x to f28 container?

@jcajka - do you want to open a PR for adding back s390x to f28 container?

I am +1 on this.

Thanks @imcleod for the info :smile:

I have been talking to mainframe admins already about changing our s390x virt setup.

I didn't want to try and do anything right now because we are in final freeze and I don't want to disrupt the release.

It's on my list after freeze is over. Do we need to try and get it done before then?

I already have a ticket on my list. :)

@mohanboddu @dustymabe I have opened https://pagure.io/pungi-fedora/pull-request/662

It's on my list after freeze is over. Do we need to try and get it done before then?

probably fine to wait

I already have a ticket on my list. :)

can you link us to the ticket so I can subscribe to it ?

The ticket is in our planning instance (which we are seeing if it's useful before we widely advertise it): https://pagure.io/cpe-planning/issue/16

Well, this issue is solved by https://pagure.io/pungi-fedora/pull-request/662, but we still need to figure out armhfp issue.

Please follow https://pagure.io/cpe-planning/issue/16 for s390x virt setup.