pungi

#625 pungi not exiting on failure
Closed: Fixed Opened by kevin.

Closed: Fixed
Reopen Issue

Rawhide has been failing for various reasons for the last 5 days.

Pungi is properly sending the fedmsg that the compose died.

However, on the compose machine it's not exiting the job:

root 41461 2.2 25.3 7154640 3899228 ? Sl May25 81:21 /usr/bin/python /usr/bin/pungi-koji --notification-script=/usr/bin/pungi-fedmsg-notification --config=fedora.conf --old-composes=/mnt/koji/compose/rawhide --nightly --skip-phase=productimg --target-dir=/mnt/koji/compose/rawhide

  |-crond,1102 -n
  |   |-crond,41385 -n
  |   |   |-sh,41386 -c...
  |   |   |   `-nightly.sh,41401 ./nightly.sh
  |   |   |       `-pungi-koji,41461 /usr/bin/pungi-koji --notification-script=/usr/bin/pungi-fedmsg-notification ...
  |   |   |           |-{pungi-koji},50990
  |   |   |           |-{pungi-koji},50991
  |   |   |           |-{pungi-koji},50992
  |   |   |           |-{pungi-koji},50993
  |   |   |           |-{pungi-koji},50994
  |   |   |           |-{pungi-koji},50995
  |   |   |           |-{pungi-koji},50996
  |   |   |           |-{pungi-koji},50997
  |   |   |           |-{pungi-koji},50998
  |   |   |           |-{pungi-koji},50999
  |   |   |           |-{pungi-koji},51007
  |   |   |           |-{pungi-koji},51008
  |   |   |           |-{pungi-koji},51009
  |   |   |           |-{pungi-koji},51010
  |   |   |           |-{pungi-koji},51011
  |   |   |           |-{pungi-koji},51012
  |   |   |           |-{pungi-koji},51013
  |   |   |           |-{pungi-koji},51015
  |   |   |           |-{pungi-koji},51016
  |   |   |           |-{pungi-koji},51017
  |   |   |           |-{pungi-koji},51018
  |   |   |           |-{pungi-koji},51019
  |   |   |           |-{pungi-koji},51020
  |   |   |           |-{pungi-koji},51021
  |   |   |           |-{pungi-koji},51022
  |   |   |           |-{pungi-koji},51023
  |   |   |           |-{pungi-koji},51032
  |   |   |           |-{pungi-koji},51033
  |   |   |           |-{pungi-koji},51034
  |   |   |           `-{pungi-koji},51035
  |   |   `-sendmail,41390 -FCronDaemon -i -odi -oem -oi -t -f root
  |   |       `-postdrop,41391 -r

This is a relatively known issue. The problem is the way how threads are implemented in Kobo library. Each phase in Pungi has it's own thread pool. Some phases run in parallel (because each thread is just waiting for external command).

The way the pools are implemented is that they have a start and stop methods. The first one just spawns the threads and returns almost immediately. The second blocks until all the queued work is finished. If a crash happens after start succeeded but before stop was called, the process will never exit.

So the fix from Pungi side would be to ensure that if start succeeded, we always call stop as well.

I think #626 might avoid this problem, but I don't have a nice way to test this.

I figured out a hacky way to test #626 and it seems to work fine in stage. I believe it should fix this issue. Let's reopen this if it still happens after patched pungi is deployed (should be available next week).

Metadata Update from @lsedlar:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

I have built pungi-4.1.15-2.fc25 with the patch to fix this that can be used until a proper release is out: https://koji.fedoraproject.org/koji/buildinfo?buildID=903059

Metadata Update from @lsedlar:
- Issue tagged with: 4.1.16

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/pungi/pungi/issues/625

Please continue any further discussion there.
Metadata
None

4.1.16

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.