fedora-infrastructure

#12704 Update distrobuildsync-eln notifications
Closed: Will Not/Can Not fix by yselkowitz. Opened by yselkowitz.

Closed: Will Not/Can Not fix
Reopen Issue

Describe what you would like us to do:

Apparently there are notifications when the message queue for the distrobuildsync-eln service is large that are currently going to @sgallagh. These should come to my (@yselkowitz) instead, as Stephen is no longer working on ELN. If there are any other notifications around ELN services, these too should be directed to me.

When do you need this to be done by? (YYYY/MM/DD)

At your convenience.

Metadata Update from @james:
- Issue tagged with: low-gain, low-trouble

Metadata Update from @james:
- Issue priority set to: Waiting on Assignee (was: Needs Review)

I wanted to look at this today, but I'm not sure when this is even setup. I can't find anything in both ansible or nagios configuration.

I assume this is about nagios alerts for this check.

Yeah, I don't see where this is setup at all. ;(

@sgallagh what does this notification to you look like?

It comes from Nagios

On Tue, Sep 9, 2025 at 6:58=E2=80=AFPM Kevin Fenzi pagure@pagure.io wrote=
:

kevin added a new comment to an issue you are following:
``
Yeah, I don't see where this is setup at all. ;(

@sgallagh what does this notification to you look like?

``

To reply, visit the link below or just reply to this email
https://pagure.io/fedora-infrastructure/issue/12704

Looking at it more carefully, it appears that I'm actually getting ALL the NAGIOS messages to sysadmin-members@ and I have a filter that archives and marks as read all the ones that don't mention ELN.

An example message:

Delivered-To: sgallagh@gapps.redhat.com
Received: by 2002:a05:7108:3e93:b0:45e:34a1:60b4 with SMTP id hk19csp741867gdb;
        Sat, 16 Aug 2025 21:15:45 -0700 (PDT)
X-Forwarded-Encrypted: i=2; AJvYcCWyiaqfRpeM4hzm8fxuoNF3DHXwnlipyJmijZsMEeHA1Zqeje5opuySh/SzYJcGZAt+tfLh5uzrYg==@gapps.redhat.com
X-Google-Smtp-Source: AGHT+IGSUso58t38mFOdnKMjd5smSu+IPEm7ifsftitEA3JhElcHhrX5zh0VuGzY0LLDEuxklil0
X-Received: by 2002:a05:620a:2809:b0:7e3:60cf:c037 with SMTP id af79cd13be357-7e87e06bf0bmr935240885a.34.1755404145205;
        Sat, 16 Aug 2025 21:15:45 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1755404145; cv=none;
        d=google.com; s=arc-20240605;
        b=UUEUEtecQ32uuQVgXOOp2BeM7dP+h8ElqbG4DRe6Uedd3PpblSd6tVe3xSbNTDnbAY
         HY2fIchZlxc+BbW383zRe9bY2n7ITL0dHKpMh6i1ggU3t+BjkPPd7wzFlOpcqgFcN0Ev
         EVVDsSlJJs9MBwAAMa7se6xgUpvWc9sxSWzkCr+sIep5tZTvk5HjU1SQsD4Hd56IcZug
         IKgidls2UGpfRItrrRzu7WTK2Z42V5xkcV4qg3ugbS/dkXNttYqj1qWjDJQ6usohxfYw
         2o70k3r2nX/nALENw0Fo2TC6uA5e6pZ6jnjJB+kx+3JqmpamEDgYhF6hgFs0MOCbjzT5
         slAA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605;
        h=from:message-id:content-transfer-encoding:mime-version:user-agent
         :subject:to:date:dkim-signature:dkim-filter:delivered-to
         :delivered-to;
        bh=E2ady5nfAP8Ori3ZZbNtDB3mW295m5RVSEaxvL620oA=;
        fh=o497PrnkUKjCD0dWK2M6H0KtQEjEpCO9KzObNUyhPRA=;
        b=aY+PV0TLVoEZEOk4jcuLcIoAdYzt0MKuQ6wpfnGV/op+ERxtMn+Kegg6OXd+f3Th3I
         YdlENTgCuRh7RjkiNo0JLk592Oroiq1cFqslAf9An+ORRzOzV8irJ7mhTWR+x4GF5YQr
         to8Y+vjobir6ezNC2DwhZvEyvZX4slx8LlKmE/xjjmnSTvz5At0JCTTyMJXCg3aCPA9q
         cJZg1YcrlSt9Bq9Hxu5Edf9rL9WA/9pJIDiyYLoPvjRTW16u9CyOvAnBBU50309MGZJo
         wZKI8Abv1WPtTiq+PtpS/FOHC93PBxuF8RmBw+nd9wcxfL9uKk0zQpILOSHZIDnCL/Og
         FasA==;
        dara=google.com
ARC-Authentication-Results: i=1; mx.google.com;
       dkim=pass header.i=@fedoraproject.org header.s=bastion-iad header.b=LNahOO2t;
       gateway.spf=pass (google.com: domain gapps.redhat.com configured 170.10.132.61 as internal address) smtp.mailfrom=nagios@fedoraproject.org smtp.remote-ip=170.10.132.61 policy.d=gapps.redhat.com
Return-Path: <nagios@fedoraproject.org>
Received: from us-smtp-inbound-delivery-1.mimecast.com (us-smtp-inbound-delivery-1.mimecast.com. [170.10.132.61])
        by mx.google.com with ESMTPS id af79cd13be357-7e87e018f9bsi182849185a.218.2025.08.16.21.15.45
        for <sgallagh@gapps.redhat.com>
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Sat, 16 Aug 2025 21:15:45 -0700 (PDT)
Received-SPF: pass (google.com: domain gapps.redhat.com configured 170.10.132.61 as internal address)
Authentication-Results: mx.google.com;
       dkim=pass header.i=@fedoraproject.org header.s=bastion-iad header.b=LNahOO2t;
       gateway.spf=pass (google.com: domain gapps.redhat.com configured 170.10.132.61 as internal address) smtp.mailfrom=nagios@fedoraproject.org smtp.remote-ip=170.10.132.61 policy.d=gapps.redhat.com
Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-496-VxH39kmsM5WwbHxzE1e8mw-1; Sun, 17 Aug 2025 00:15:43 -0400
X-MC-Unique: VxH39kmsM5WwbHxzE1e8mw-1
X-Mimecast-MFC-AGG-ID: VxH39kmsM5WwbHxzE1e8mw_1755404142
Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
     key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 988D5180036E for <sgallagh@gapps.redhat.com>; Sun, 17 Aug 2025 04:15:40 +0000 (UTC)
Received: by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) id 7BD2F1800956; Sun, 17 Aug 2025 04:15:40 +0000 (UTC)
Delivered-To: sgallagh@redhat.com
Received: from bastion.fedoraproject.org (unknown [10.16.163.31]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 66A20180047F; Sun, 17 Aug 2025 04:15:39 +0000 (UTC)
Received: by bastion01.rdu3.fedoraproject.org (Postfix) id D30C4793A5; Sun, 17 Aug 2025 04:15:38 +0000 (UTC)
Delivered-To: sysadmin-members@fedoraproject.org
Received: from noc01.rdu3.fedoraproject.org (dhcp.rdu3.fedoraproject.org [10.16.163.10]) by bastion01.rdu3.fedoraproject.org (Postfix) with ESMTP id BEAAB793A1 for <sysadmin-members@fedoraproject.org>; Sun, 17 Aug 2025 04:15:38 +0000 (UTC)
DKIM-Filter: OpenDKIM Filter v2.11.0 bastion01.rdu3.fedoraproject.org BEAAB793A1
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fedoraproject.org; s=bastion-iad; t=1755404138; bh=E2ady5nfAP8Ori3ZZbNtDB3mW295m5RVSEaxvL620oA=; h=Date:To:Subject:From:From; b=LNahOO2tdWd+D12MIRBOAF0LzJg4H8qgYBYxNN0hxLDkezE06apeb66wZ9bUvMwnZ
     8Qyv4eTVAb2O/YVczppcfMPhBfJayz5cVM68ygfSQuP/yGY62FXl+wPWTF4ibrm+5H
     T1XX78DdxuBULrJ6QB1dOw6Uy4rAHIOBaTMgk6y0=
Received: by noc01.rdu3.fedoraproject.org (Postfix, from userid 420) id BB1BE75E51; Sun, 17 Aug 2025 04:15:38 +0000 (UTC)
Date: Sun, 17 Aug 2025 04:15:38 +0000
To: sysadmin-members@fedoraproject.org
Subject: ** PROBLEM alert - rabbitmq01.rdu3.fedoraproject.org/Check queue distrobuildsync-eln is WARNING **
User-Agent: s-nail v14.9.22
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Message-Id: <20250817041538.BB1BE75E51@noc01.rdu3.fedoraproject.org>
From: Nagios Monitoring User <nagios@fedoraproject.org>
X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111
***** Nagios  *****
Notification Type: PROBLEM
Service: Check queue distrobuildsync-eln
Host: rabbitmq01.rdu3.fedoraproject.org
Address: 10.16.163.78
State: WARNING
Source: noc01.rdu3.fedoraproject.org
Date/Time: Sun Aug 17 04:15:38 UTC 2025
Additional Info:
RABBITMQ_QUEUE WARNING - messages WARNING (11161), messages_ready OK (11136=
) messages_unacknowledged OK (25) consumers OK (1)

Ah, that would explain it. Yes, everyone in 'sysadmin' gets nagios alerts via email.

@yselkowitz shall I add you to that group? be ready tho... it gets a lot of email...

That sounds rather overwhelming and something of an overkill. How is this normally handled with openshift apps?

I am not sure what you mean... openshift apps have set 'app owners' and there's some small number of things that cause it to send email to all those if they happen (pod crash looping, deployment stuck, etc).

There's no way I know of to use notifications to get this currently, even tho nagios should send a message...

we are in process of switching to zabbix (and getting much closer). There may be a way to do this in zabbix... @gwmngilfen ?

Metadata Update from @yselkowitz:
- Issue close_status updated to: Will Not/Can Not fix
- Issue status updated to: Closed (was: Open)
Metadata
None

low-gain low-trouble

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.