fm-orchestrator

#1519 Stuck build in MBS
Closed: Fixed by mprahl. Opened by mbooth.

Closed: Fixed
Reopen Issue

This has been running for > 12 hours and only managed to build the module-build-macros package:

https://mbs.fedoraproject.org/module-build-service/1/module-builds/7085

What is MBS doing here?

I tried cancelling and resubmitting the build several times. My only option now is to make another empty commit and try fresh builds. It's impossible to do any modularity builds with any sense of urgency, eh? :-)

Build was still going 6 days later, so I attempted to cancel it one more time -- I don't want it to interfere in any other work.

Still no idea what caused this one to get stuck

time_modified "2019-11-26T10:40:42Z"
time_submitted "2019-11-20T14:25:26Z"
Edited by mbooth

The last entry in build log is:

2019-11-20 14:48:56,014 - PoolThread-twisted.internet.reactor-1 - MBS.builder.KojiModuleBuilder - DEBUG - Ensuring existence of tag='module-eclipse-latest-3020191120142149-1f72e1db'.

At this exact time fedmsg-hub service was restarted:

Nov 20 14:48:56 mbs-backend01.phx2.fedoraproject.org fedmsg-hub[2801]: [2019-11-20 14:48:56][MBS.builder.KojiModuleBuilder    INFO] <KojiModuleBuilder module: eclipse, tag: module-eclipse-latest-3020191120142149-1f72e1db> connecting buildroot.
Nov 20 14:48:56 mbs-backend01.phx2.fedoraproject.org systemd[1]: Stopping Generic fedmsg processing hub...
Nov 20 14:48:56 mbs-backend01.phx2.fedoraproject.org systemd[1]: Stopped Generic fedmsg processing hub.
Nov 20 14:48:56 mbs-backend01.phx2.fedoraproject.org systemd[1]: Started Generic fedmsg processing hub.

Around that time root user logged in from batcave01:

# last -w | grep 'Nov 20'
root     pts/0        batcave01.phx2.fedoraproject.org Wed Nov 20 14:48 - 14:49  (00:00)

There were no MBS-related Ansible playbook runs recorded around that time.

To me it seems someone logged into mbs-backend and restarted fedmsg-hub.
This caused build to be stuck.

We've been having issues with file descriptor leaks in MBS and it causes MBS to fail in odd ways. Some of these failures aren't recovered after a restart. We deployed the fix for https://pagure.io/koji/issue/1652 last week. We hope there aren't other file descriptor leaks which cause this odd behavior.

I'll close this ticket for now. Let us know if this happens again.

Metadata Update from @mprahl:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/mbs/issues/1519

Please continue any further discussion there.
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.