fedora-infrastructure

#12969 ipa backups not completing
Closed: Fixed with Explanation by zlopez. Opened by kevin.

Closed: Fixed with Explanation
Reopen Issue

ipa01 backups are not completing.

From /var/log/ipabackup.log on ipa01:

2025-12-09T04:48:26Z INFO Backing up FEDORAPROJECT-ORG
2025-12-09T04:48:26Z INFO Waiting for BAK to finish
2025-12-09T04:48:27Z DEBUG   File "/usr/lib/python3.9/site-packages/ipapython/admintool.py", line 219, in execute
    return_value = self.run()
  File "/usr/lib/python3.9/site-packages/ipaserver/install/ipa_backup.py", line 347, in run
    self.db2bak(instance, online=options.online)
  File "/usr/lib/python3.9/site-packages/ipaserver/install/ipa_backup.py", line 602, in db2bak
    raise admintool.ScriptError(
2025-12-09T04:48:27Z DEBUG The ipa-backup command failed, exception: ScriptError: BAK online task failed. Check file systems' free space.
2025-12-09T04:48:27Z ERROR BAK online task failed. Check file systems' free space.
2025-12-09T04:48:27Z ERROR The ipa-backup command failed. See /var/log/ipabackup.log for more information

There seems to be a fine amount of space: 

# df -h /
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/vg_vm-LogVol00   40G   32G  8.2G  80% /

We should fix it so they run.

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: Needs investigation, authentication, high-gain, ops

Looking at zabbix, the disk usage spikes by almost 15% when the backup runs, so I thought I'd make more space. There was 10G in /var/log, so I compressed most of the httpd logs, and did an early rotation on krb5kdc.log - that got us to 12G free.

However, it didn't help, same error. I do note that while the error message is about space, the actual code is making an LDAP connection:

# /usr/lib/python3.9/site-packages/ipaserver/install/ipa_backup.py line ~602
            logger.info("Waiting for BAK to finish")
            if (wait_for_task(conn, dn) != 0):
                raise admintool.ScriptError(
                    'BAK online task failed. Check file systems\' free space.'
                )

so, perhaps the issue is with that connection? Not sure how to troubleshoot it though.

Mental++ note to check if this got fixed today.

It didn't I tested it yesterday and the issue still remains. It was not related to replication issue we fixed yesterday.

Okay, I fixed it ... was one of those super obvious fixes, when you find out what/where to do.

The above part of the script that fails is waiting for something to write to the special backup dir:

paths.SLAPD_INSTANCE_BACKUP_DIR_TEMPLATE

... this expands to:

/var/lib/dirsrv//slapd-FEDORAPROJECT-ORG/bak

...that dir got moved to bak.sav at some point (I guess on the 26-27th of Nov, as that's when backups stopped working). So I just created a new bak dir. and chown'd it to dirsrv, then ran the backup cron job by hand and it worked.
Edited by james

Metadata Update from @james:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Good catch - I see your manual backup at 9pm. However, it looks like the cron failed again overnight. Can you take another look @james? Maybe permissions?

Metadata Update from @gwmngilfen:
- Issue status updated to: Open (was: Closed)

Metadata Update from @gwmngilfen:
- Issue status updated to: Open (was: Closed)

So just to test it out I tried to run the cron job on both ipa02 and ipa03. Both of them have the same issue as ipa01. This means that the backup isn't working on any of them.

@james How did you ran the job? Just running the backup script as root? Or using run-parts /etc/cron.daily/?

So I got some more logs on ipa03:

Backing up FEDORAPROJECT-ORG
Waiting for BAK to finish
Exit_code: -1
Log: Beginning backup of 'ldbm database'
Backing up file 1 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/ipaNTTrustPartner.db)
Backing up file 2 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/nsOsVersion.db)
Backing up file 3 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/memberHost.db)
Backing up file 4 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/owner.db)
Backing up file 5 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/accessRuleType.db)
Backing up file 6 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/automountkey.db)
Backing up file 7 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/ipaCASubjectDN.db)
Backing up file 8 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/memberservice.db)
Backing up file 9 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/sudoorder.db)
Backing up file 10 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/sourcehost.db)
Backing up file 11 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/ipaallowedtarget.db)
Backing up file 12 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/numsubordinates.db)
Backing up file 13 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/ancestorid.db)
Backing up file 14 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/replication_changelog.db)
Backing up file 15 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/uniquemember.db)
Backing up file 16 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/manager.db)
Backing up file 17 (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG/userRoot/id2entry.db)
Backup: error in copying directory (/var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/db/userRoot -> /var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG): err=-1
db2archive failed: removing /var/lib/dirsrv/slapd-FEDORAPROJECT-ORG/bak/FEDORAPROJECT-ORG
Backup failed (error -1)

So I fixed the ipa03. It was really problem with space, I had to add another 20G to get the backups working.

And I understand why it worked for @james. Because after his fix the spam with new users started and the DB started growing. The last backup on ipa02 and ipa03 worked on 16th December, but it failed on 17th December with the start of the spam. The ipa01 hit the problem with space first as most of the authentication is happening through it and it generates the most logs.

I will need to add more space to both ipa02 and ipa01 to fix them as well.

Resized ipa01 and ipa02, the backups are now twice as large unfortunately, but the space should be enough for some time.

So I'm closing this now as all the backups are working again.

Metadata Update from @zlopez:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

Metadata Update from @zlopez:
- Issue assigned to zlopez
Metadata

Needs investigation authentication high-gain ops

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.