I'm running test installations of FreeIPA with Vagrant inside libvirt qemu VMs. Sometimes VMs are not shut down gracefully. Several times a hard crash caused dse.ldif either to be missing completely or to be a zero byte file. The first time the problem occurred it took me a while to realize that the issue was caused by a broken dse.ldif. 389 DS' error message wasn't helpful. The server complained that 00core.ldif references an unknown OID, which turned out to be the OID of UTF-8 string type.
Would it makes sense to auto-detect a missing or empty dse.ldif file and restore the last good copy automatically? In case it doesn't make sense, can you please change the error message and make it obvious that dse.ldif is broken and should be recovered manually? Personally I would prefer auto-recover + warning in journald / auditd to make FreeIPA more robust against hardware faults and filesystem issues.
389-ds-base-1.3.5.15-1.fc25.x86_64
I thought this was fixed by @lkrispen a long time ago. Ludwig, can you confirm?
I ran into the issue just yesterday with 389 on Fedora 25.
The issue was fixed for cases where at least a dse.ldif.bak existed: https://pagure.io/389-ds-base/issue/518
If the environment is more seriously broken I'm afraid we cannot recover
Closing this is there is no safe way to properly recover from this situation.
Metadata Update from @mreynolds: - Custom field type adjusted to defect - Issue close_status updated to: wontfix - Issue status updated to: Closed (was: Open)
the root cause might have been found in ticket #49298 - a missing fsync after rename
We are still seeing this problem every now and then. Most recently on 1.4.1. The one person had a valid dse.ldif.startOK file and was able to recover that way. Should the server be trying the same thing... It's better than losing the entire configuration, right?
Metadata Update from @mreynolds: - Custom field origin adjusted to None - Custom field reviewstatus adjusted to None - Issue status updated to: Open (was: Closed)
Metadata Update from @mreynolds: - Issue priority set to: normal - Issue set to the milestone: 1.4.1
We have to draw a line at some level, where we can make as much effort as possible to prevent this, but at some point, an admin will need to intervene, work out why they are having hard power-offs, and consider restoration from backups or the .startOK file.
I certainly do not understand fully this matter but I agree with William here. The .startOK is the last resort, and AFAIU also a lot could have happened/changed since the last start-up potentially resulting in inconsistencies between .startOK and environment. Unless already so, we should only improve log message in that case, IMHO.
Is there more information about who is having the problem and in what conditions @mreynolds ?
A VM was automatically shutdown due to disk space getting low, and the only thing that survived was dse.ldif.startOK which they used to successfully recover from the situation. This is what was in the errors log:
[29/Mar/2020:03:23:18.465607935 -0600] - INFO - dse_check_file - The config /etc/dirsrv/slapd-INST-COM/dse.ldif has zero length. Attempting restore ... [29/Mar/2020:03:23:18.465784552 -0600] - ERR - dse_check_file - The backup file /etc/dirsrv/slapd-INST-COM/dse.ldif.bak has zero length, refusing to restore it.
I think this code should be extended to the check dse.ldif.startOK as a last resort.
I don't think we should try automatic repair of the .startOK to dse.ldif. It could have been a long time since startOK was written. Telling the person to manually intervene here still seems like the best option ...
I agree with William, we should not silently use startOK, we do not know in which state it is. Using it, and even logging a message which could be ignored, would look like everything is ok, but it is not. The log could have a message loke "look for startOK or any other dse.ldif* and try to use it instead of dse.ldif"
Metadata Update from @mreynolds: - Issue set to the milestone: 1.4.4 (was: 1.4.1)
389-ds-base is moving from Pagure to Github. This means that new issues and pull requests will be accepted only in 389-ds-base's github repository.
This issue has been cloned to Github and is available here: - https://github.com/389ds/389-ds-base/issues/2190
If you want to receive further updates on the issue, please navigate to the github issue and click on subscribe button.
subscribe
Thank you for understanding. We apologize for all inconvenience.
Metadata Update from @spichugi: - Issue close_status updated to: wontfix - Issue status updated to: Closed (was: Open)