koji

#3520 Database connection leak
Closed: Fixed by wart. Opened by wart.

Closed: Fixed
Reopen Issue

I recently installed koji-1.30.0 on a Rocky Linux 9 host. I also imported my production koji postgresdb from an earlier 1.20.1 install and ran through the sequence of db upgrade scripts.

After a while, koji stops being able to connect to the postgres db. Both koji and postgres complain about running out of connections:

httpd:
[Fri Sep 23 09:28:17.995204 2022] [wsgi:error] [pid 11762:tid 11899] [remote 10.13.5.16:57372] koji.ServerOffline: database outage

postgres:

2022-09-23 08:33:49.831 CDT [21856] FATAL: remaining connection slots are reserved for non-replication superuser connections

I've bumped max_connections in the postgres config up to 1000 (from the default of 100), but that only delays the issue.

Looking at the accumulated idle connections in postgres, it looks like they are all the result of a ROLLBACK command:

psql koji koji -c "SELECT * FROM pg_stat_activity WHERE state = 'idle' order by state_change;"

datid | datname | pid | leader_pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | xact_start | query_start | state_change | wait_event_type | wait_event | state | backend_xid | backend_xmin | query | backend_type
-------+---------+-------+------------+----------+---------+------------------+-------------+-----------------+-------------+-------------------------------+------------+-------------------------------+-------------------------------+-----------------+------------+-------+-------------+--------------+----------+----------------
17877 | koji | 27006 | | 16384 | koji | | | | -1 | 2022-09-23 11:43:59.935625-05 | | 2022-09-23 11:51:37.521096-05 | 2022-09-23 11:51:37.521109-05 | Client | ClientRead | idle | | | ROLLBACK | client backend

I've tried setting 'idle_in_transaction_session_timeout = 120000' in postgres.conf, but that doesn't stop the leak (it seems to slow it, but that could be my imagination).

The only places I see the ROLLBACK command being issued in the koji code are in these two files, and only the second one has a bare ROLLBACK command issued (as part of DBWrapper.close().

/usr/share/koji-hub/kojihub.py
/usr/lib/python3.9/site-packages/koji/db.py

Is there some configuration setting I may have overlooked to cause these connections to stick around?

Please check that apache ServerLimit and MaxRequestWorkers are lowed than max_connections in postgres. There is no connection pooling, so apache shouldn't spawn more clients than PG can handle.

ROLLBACK in idle state should be harmless per se (it already happened). Anyway, it means that connection is still not closed which looks weird. How clients are behaving? Are they waiting for some response or they finish without problems? Jump from 1.20.1 to 1.30 is a big one but I don't recall anything which should cause such behaviour.

Metadata Update from @tkopecek:
- Custom field Size adjusted to None

I had bumped max_connections to 1000, and ServerLimit/MaxRequestWorkers to 256. However, this still caused the database connection leak.

Clients work as expected, no hanging, no slow response. They only show errors once the database connections are exhausted.

Looking more closely at my production koji instance (still at 1.20.1), I found that the default MPM module for apache has changed from mpm_prefork (EL7) to mpm_event (EL9). Changing it back to mpm_prefork on the new RL9 koji server appears to have stabilized the number of database connections.

Ah, it makes sense. I'll investigate how it behaves on EL9.

With default values (centos9) it stops on 15 idle connections which seems to correspond to MaxServers (16 = 15 + 1 superuser reservation). It seems that I'm not able to get more idle connections. I've tried some stress-testing with 1000 connections in parallel, but nothing weird is happening.

Can you verify which mpm module is being used by your apache server? I just did a second install using Rocky Linux 9, and it defaulted to mpm_event, and showed the connection leak again. Switching it back to mpm_prefork stabilized the number of db connections.

Yes, it is for mpm_event. I can try Rocky, if it makes the difference.

Hmm, same there - It will raise to 15 and stop (with mod_mpm_event).

Well I'm stumped. But since I've got an acceptable workaround, I'll consider this resolved. If you have any ideas on how to debug further, I'd be happy to give it a go.

Metadata Update from @wart:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/3520

Please continue any further discussion there.
Metadata
None
None
None
None
Low
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.