koji

#4508 Stuck tagBuild operations on koji since upgrade to rhel9
Closed: Invalid by mikem. Opened by arrfab.

Closed: Invalid
Reopen Issue

Context: we recently upgraded from RHEL8 to RHEL9 (so 9.7 now) for https://cbs.centos.org (running koji 1.35.3)

When we imported postgresql, it was itself just a pg_dump for koji db, imported into new postgresql host (rhel9 too) so on postgresql 13

We started to suffer from multiple issues with db connection (see #4175) but that was "resolved" by switching httpd to mpm_prefork (it wasn't needed when we were on rhel8 though)

Then we have on regular basis "stuck" insert statements into postgresql, for tagBuild operations.

@mikem already briefly looked at this with me, and we tried already some "tuning" for postgresql but without any luck. Worth knowing that for same load as before we even have better/faster VM specs now, but that doesn't seem to be the issue either.

We even just upgraded postgresql to version 16 (by going first to 15) through modularity (available for rhel9 but again we suffer from same issue

A way to reproduce : I untagged 119 builds from our -testing tag (for Infra) and then tagged them back :

it works up to a point and then stops:

5349659 tagBuild (noarch): assigned (x86-08.cbs.centos.org) -> closed
  0 free  30 open  89 done  0 failed

and from pg_stat_activity we see remaining tasks as sql statements but "idle in transaction" : 

postgres=# select pid, query,state, query_start from pg_stat_activity where query_start < now() - '1 min'::interval order by query_start;
  pid   |                                                 query                                                  |        state        |          query_start          
--------+--------------------------------------------------------------------------------------------------------+---------------------+-------------------------------
 325478 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (28445, 2754988, 10, 2215) | idle in transaction | 2025-12-09 13:46:45.256391+00
 325242 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (37549, 2754990, 10, 2215) | idle in transaction | 2025-12-09 13:46:45.335397+00
 327124 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (27940, 2754991, 10, 2215) | idle in transaction | 2025-12-09 13:46:45.370958+00
 323935 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (34898, 2754994, 10, 2215) | idle in transaction | 2025-12-09 13:46:45.498193+00
 330403 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (33095, 2755003, 10, 2215) | idle in transaction | 2025-12-09 13:46:46.040326+00
 327090 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (33025, 2755004, 10, 2215) | idle in transaction | 2025-12-09 13:46:46.056925+00
 322904 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (31786, 2755007, 10, 2215) | idle in transaction | 2025-12-09 13:46:46.262307+00
 326103 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (31773, 2755014, 10, 2215) | idle in transaction | 2025-12-09 13:46:46.902548+00
 327011 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (28360, 2755018, 10, 2215) | idle in transaction | 2025-12-09 13:46:47.548585+00
 327635 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (27924, 2755020, 10, 2215) | idle in transaction | 2025-12-09 13:46:47.731239+00
 330917 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (27967, 2755023, 10, 2215) | idle in transaction | 2025-12-09 13:46:47.899367+00
 326398 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (28157, 2755026, 10, 2215) | idle in transaction | 2025-12-09 13:46:48.092703+00
 312602 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (30614, 2755032, 10, 2215) | idle in transaction | 2025-12-09 13:46:48.490481+00
 330936 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (27928, 2755034, 10, 2215) | idle in transaction | 2025-12-09 13:46:48.567943+00
 330904 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (30532, 2755035, 10, 2215) | idle in transaction | 2025-12-09 13:46:48.580106+00
 330937 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (31992, 2755037, 10, 2215) | idle in transaction | 2025-12-09 13:46:48.718186+00
 329265 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (31647, 2755039, 10, 2215) | idle in transaction | 2025-12-09 13:46:48.816316+00
 330967 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (34939, 2755041, 10, 2215) | idle in transaction | 2025-12-09 13:46:48.915073+00
 331027 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (34900, 2755045, 10, 2215) | idle in transaction | 2025-12-09 13:46:50.373698+00
 325310 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (33096, 2755046, 10, 2215) | idle in transaction | 2025-12-09 13:46:50.406841+00
 331028 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (28409, 2755048, 10, 2215) | idle in transaction | 2025-12-09 13:46:50.549679+00
 331139 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (31771, 2755053, 10, 2215) | idle in transaction | 2025-12-09 13:46:51.615381+00
 331179 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (31781, 2755060, 10, 2215) | idle in transaction | 2025-12-09 13:46:52.794963+00
 331201 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (34903, 2755064, 10, 2215) | idle in transaction | 2025-12-09 13:46:53.381364+00
 331254 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (37551, 2755080, 10, 2215) | idle in transaction | 2025-12-09 13:46:55.93126+00
 331184 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (27925, 2755089, 10, 2215) | idle in transaction | 2025-12-09 13:47:45.600998+00
 331639 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (31916, 2755090, 10, 2215) | idle in transaction | 2025-12-09 13:47:45.627249+00
 329008 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (61865, 2755091, 10, 2215) | idle in transaction | 2025-12-09 13:47:45.662621+00
 331516 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (36482, 2755093, 10, 2215) | idle in transaction | 2025-12-09 13:47:46.217785+00
 331133 | INSERT INTO tag_listing (build_id, create_event, creator_id, tag_id) VALUES (56412, 2755094, 10, 2215) | idle in transaction | 2025-12-09 13:47:46.245996+00
(30 rows)

After some debugging, it appears the issue is a stall in one of the messaging plugins during the postTag callback.

Metadata Update from @mikem:
- Custom field Size adjusted to None

Messagebus plugin authors might want to have a look at koji's protonmsg plugin:
https://pagure.io/koji/blob/master/f/plugins/hub/protonmsg.py

Metadata Update from @mikem:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/4508

Please continue any further discussion there.
Metadata
None
None
None
None
Low
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.