fedora-infrastructure

#12967 Bodhi messaging queue on staging has over 1 200 000 messages
Closed: Duplicate by zlopez. Opened by zlopez.

Closed: Duplicate
Reopen Issue

Describe what you would like us to do:

Last week I noticed that the message queue on staging bodhi is rising. It now has over 1 200 000 messages and it's still growing. I was checking what is happening there and the main problem is that processing of each message takes around 40 seconds. Which is much slower than the messages being published.

When do you need this to be done by? (YYYY/MM/DD)

Not urgent

I spent some time investigating this and I'm no wiser. Here is the log from message processing:

2025-12-08 16:10:11,121 INFO [fedora_messaging.twisted.consumer][MainThread] Consuming message from topic org.fedoraproject.stg.buildsys.tag (message id 1fb5b9ec-a958-448d-a258-abf172164c2b)
2025-12-08 16:10:11,122 INFO [bodhi][PoolThread-twisted.internet.reactor-0] Received message from fedora-messaging with topic: org.fedoraproject.stg.buildsys.tag
2025-12-08 16:10:11,122 INFO [bodhi][PoolThread-twisted.internet.reactor-0] vim-commentary-1.3-19.fc43 tagged into f43-signing-pending
2025-12-08 16:10:49,937 INFO [bodhi][PoolThread-twisted.internet.reactor-0] Tag is not testing side tag, skipping
2025-12-08 16:10:50,148 INFO [fedora_messaging.twisted.consumer][MainThread] Successfully consumed message from topic org.fedoraproject.stg.buildsys.tag (message id 1fb5b9ec-a958-448d-a258-abf172164c2b)

I noticed that the database db01.stg was under heavy load, so I bumped the number of vCPUs, but this doesn't seem to be related. I tried to enable debug output as well, but it didn't show anything new.

The processing is done by this code and I don't see anything that would take 40 seconds in there

        log.info("%s tagged into %s" % (build_nvr, tag))
        with self.db_factory() as dbsession:
            build = Build.get(build_nvr)
            if not build:
                log.info("Build was not submitted, skipping")
                return
            if not build.release:
                log.info('Build is not assigned to release, skipping')
                return
            if build.update \
                    and build.update.from_tag \
                    and not build.update.release.composed_by_bodhi:
                koji_testing_tag = build.release.get_pending_testing_side_tag(build.update.from_tag)
                if tag != koji_testing_tag:
                    log.info("Tag is not testing side tag, skipping")
                    return

Duplicate of https://pagure.io/fedora-infrastructure/issue/12932

Metadata Update from @zlopez:
- Issue close_status updated to: Duplicate
- Issue status updated to: Closed (was: Open)
Metadata

Needs investigation OpenShift bodhi low-gain ops staging

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.