fedora-infrastructure

#12617 [RFE] expose metrics from rabbitmq queues for centos infra consumption (or others)
Closed: Fixed with Explanation by gwmngilfen. Opened by arrfab.

Closed: Fixed with Explanation
Reopen Issue

Follow-up on DC move, where CentOS Stream infra, relying on rabbitmq (fedora-messaging) for automation (like pkgs signing, internally, and other tasks) was a little bit "in the dark" about queue length, etc .
Based on discussion with @abompard , it seems that consumers (client) can't retrieve number of tasks in the queue, but it's possible to graph that at rabbitmq side.

Ideas (proposal)
* Fedora infra exposes such metrics in a parsable way (through endpoint, can be just text file generated by some script running on rabbitmq => that way, anyone can use any tool to gather stats/metrics, and feed their own tool (zabbix for centos)
* Fedora infra accepts to use zabbix_sender directly on the host, but with a different config than the zabbix-agent running on/for fedora infra, and so using TLS (with PSK, as that we're using/enforcing between agents/proxies/server for our own zabbix infra for centos), and so sending these metrics directly to CentOS zabbix infra . From there I can graph things and also have template for alerts, etc, so that Stream team knows when there is a problem

Open to ideas and discussions but that would help us (centos) to gather these stats :)
/cc: @asamalik

I can do it, but if others are interested to learn how to get data from the rabbitmq REST API, I'm happy to assist :-)

Alright, could you please send me the certs / key / psk? It can be by mail, my gpg key id is 31584CFEB9BF64AD, or by putting the files on a server where only sysadmin-main folks have access.

Metadata Update from @phsmoura:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops

Commit 02abc93d fixes this issue

Metadata Update from @arrfab:
- Issue status updated to: Open (was: Closed)

@gwmngilfen : can you create a virtual host in centos zabbix, attach identity and psk (not the one for centos hosts) and hand these over to Fedora infra ? Also ensure that it wouldn't be clear-text values in fedora ansible roles ;-)

We have a private repo for private stuff.

Metadata Update from @gwmngilfen:
- Issue assigned to gwmngilfen

@arrfab I can look at this, yes. Got some things on the go right now, scheduling for tomorrow or Monday
Edited by gwmngilfen

I think we can close this:
- rabbit03 has a script to send queue values to the Centos Zabbix every 5min
- there's a preshared-key in ansible-private to secure the connection
- there's a virtual host in Zabbix (rabbitmq.fedora) to receive the data
- there's a template & hostgroup created just in case we want to expand this

Right now we're only sending the Robosignatory queue data, which I think is sufficient - we can create a new ticket if we want to add LLD item generation for arbitrary queues.

@arrfab I'll leave it to you to add the required triggers / dashboards on the CentOS side.

Metadata Update from @gwmngilfen:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

Sounds good !
Thanks @gwmngilfen and @abompard !
Metadata

medium-gain medium-trouble ops

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.