#167 Update Dogtag 10 to utilize the shared NSS database model
Closed: fixed Opened by mharmsen.

The existing NSS databases utilize the original pre-Sleepycat Berkeley DB model which is unsafe when being utilized by more than one process (e. g. - Tomcat AND Apache).

As a consequence, Dogtag 10 should strongly consider moving to use the shared NSS SQLite DB model documented at:

- https://wiki.mozilla.org/NSS_Shared_DB

Starting from NSS 3.15 the Berkeley database will be dropped and replaced with SQLite DB.

Linked to Bugzilla bug: https://bugzilla.redhat.com/show_bug.cgi?id=524931 (Dogtag Certificate System)

Linked to Bugzilla bug: https://bugzilla.redhat.com/show_bug.cgi?id=557641 (Dogtag Certificate System)

That "starting from NSS 3.15 the Berkeley database will be dropped and replaced with SQLite DB" is an overstatement that needs clarification. Let me clarify because someone has expressed concern to me at reading that comment. nss-3.13.5 would be the earliest release when that could occur but such decision has not been made. It is not one to be taken without considering its consequences and consulting with all stake-holders. NSS has a fairly good track record of binary compatibility and we want keep it that way.

Currently FUTURE, but since 389 has a similar ticket (https://fedorahosted.org/389/ticket/47681 - RFE: Utilize the shared NSS database model) we should probably sync these efforts (although not directly related, possibly when/if the F21 System Wide BerkleyDB 6 change is addressed?)

vakwetu suggested 10.3 since that puts it on the radar in the 10.3 timeframe.

Per discussions, targeted 10.3

Per CS/DS meeting of 04/28/2014 - 10.4.

Additionally, it was discussed in this meeting to also move PKI TRAC Ticket #974 - Multiple database access via JSS to 10.4.

Metadata Update from @mharmsen:
- Issue assigned to vakwetu
- Issue set to the milestone: UNTRIAGED

cfu, jmagne, mharmsen not sure if you saw this but Kaie is proposing to switch the NSS to default to sqlite format in F-27. https://fedoraproject.org/wiki/Changes/NSSDefaultFileFormatSql
rcrit: thanks -- we were not aware of this, but we do have a long-standing ticket for it -- https://pagure.io/dogtagpki/issue/167
yeah, I figure it'll up the timetable on these
389-ds has a similar ticket, https://pagure.io/389-ds-base/issue/48760
rcrit: also https://pagure.io/389-ds-base/issue/47681
Ok, I'll let Mark know so he can close one as a dup
rcrit: yep
rcrit: just scanned Kaie's doc, but will they have a flag to create the old NSS db format?
you have to specify dbm:/path/to/nss/database

Metadata Update from @mharmsen:
- Custom field feature adjusted to ''
- Custom field proposedmilestone adjusted to ''
- Custom field proposedpriority adjusted to ''
- Custom field reviewer adjusted to ''
- Custom field version adjusted to ''
- Issue close_status updated to: None
- Issue priority set to: blocker (was: major)
- Issue set to the milestone: 0.0 NEEDS_TRIAGE (was: UNTRIAGED)

Metadata Update from @mharmsen:
- Custom field cc adjusted to mreynolds@redhat.com,rcrit@redhat.com

Metadata Update from @mharmsen:
- Issue set to the milestone: 10.5 (was: 0.0 NEEDS_TRIAGE)

Metadata Update from @mharmsen:
- Issue priority set to: critical (was: blocker)

Metadata Update from @mharmsen:
- Issue priority set to: major (was: critical)

[20171025] - Offline Triage ==> 10.6

Metadata Update from @mharmsen:
- Issue set to the milestone: 10.6 (was: 10.5)

[20171114] Per https://fedoraproject.org/wiki/Changes/NSSDefaultFileFormatSql, upgrading this bug to BLOCKER.

Metadata Update from @mharmsen:
- Issue priority set to: blocker (was: major)

Kai's ticket "Document a recommended mechanism to perform explicit NSS database migration from dbm to sql using existing NSS tools" https://bugzilla.mozilla.org/show_bug.cgi?id=1415912

FreeIPA NSSDB migration: https://github.com/freeipa/freeipa/pull/1254

This is needed for ticket #2560 for Fedora 28.

See also ticket #2862.

The default NSS DB type can be defined as build time with -DPKI_NSS_DB_TYPE. https://review.gerrithub.io/c/400731/ is missing to use SQL format on F28 and to convert the NSS DB to SQL format.

Related commits

  • NSS DB related doc updates: https://github.com/dogtagpki/pki/commit/4c93e74de4769ecf05c5bfed0e19d2fc1510bfe2
  • Add methods to convert NSSDB from DBM to SQL: https://github.com/dogtagpki/pki/commit/fa884a1c1d935492837412d914599f520cba5760
  • Unify config loading and NSS default db type: https://github.com/dogtagpki/pki/commit/90d633bcdd1d7f1c46b047184ca5dac4d9014f60
  • Export all conf vars in single location: https://github.com/dogtagpki/pki/commit/edd79d6c774dc3c872cce0de76cc2b7e0a543b1b
  • Execute upgrade and NSSDB conversion with service https://github.com/dogtagpki/pki/commit/25a3e9406124dc89cf649bedfb1f8c74f8fdc521

Metadata Update from @edewata:
- Issue assigned to cheimes (was: vakwetu)

Metadata Update from @cheimes:
- Assignee reset

The last patch has landed. 10.6 should support SQL NSS database and automatically migrate old DBM databases from Dogtag 10.5. The current approach has a bug. The backup method fails because freeIPA has created file that can't be copied as pkiuser:

# ls -la /var/lib/pki/pki-tomcat/ca/conf/
-rw-rw----. 1 root    root    84745 Feb 23 15:02 CS.cfg.ipabkp

and the backup script tries to chown files:

[pid 42588] mkdir("/var/log/pki/server/upgrade/10.5.5/1/oldfiles/var", 0777) = 0
[pid 42588] stat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 42588] utimensat(AT_FDCWD, "/var/log/pki/server/upgrade/10.5.5/1/oldfiles/var", [{tv_sec=1519650213, tv_nsec=956263542} /* 2018-02-26T14:03:33.95626350
[pid 42588] chmod("/var/log/pki/server/upgrade/10.5.5/1/oldfiles/var", 040755) = 0
[pid 42588] chown("/var/log/pki/server/upgrade/10.5.5/1/oldfiles/var", 0, 0) = -1 EPERM (Operation not permitted)
ERROR: [Errno 1] Operation not permitted: '/var/log/pki/server/upgrade/10.5.5/1/oldfiles/var'

The migration issue was fixed in the following changes:

  • https://github.com/dogtagpki/pki/commit/dd8bdc6c9b6e4bd8966237a3606099e74b1a7d9b
  • https://github.com/dogtagpki/pki/commit/d06bed84e50ac3acb578d4e0acc0a538e3826979
  • https://github.com/dogtagpki/pki/commit/e4384d2c5986b98d5c084e7b9d8f4fa93fddb432
  • https://github.com/dogtagpki/pki/commit/c02268cc025ebfb0860d5528e080e208e09b3cbb

Metadata Update from @edewata:
- Issue close_status updated to: fixed
- Issue set to the milestone: 10.6.0 (was: 10.6)
- Issue status updated to: Closed (was: Open)

Metadata Update from @mharmsen:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1596889,https://bugzilla.redhat.com/show_bug.cgi?id=524931, https://bugzilla.redhat.com/show_bug.cgi?id=557641 (was: https://bugzilla.redhat.com/show_bug.cgi?id=524931, https://bugzilla.redhat.com/show_bug.cgi?id=557641)

Metadata Update from @mharmsen:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1596889, https://bugzilla.redhat.com/show_bug.cgi?id=524931, https://bugzilla.redhat.com/show_bug.cgi?id=557641 (was: https://bugzilla.redhat.com/show_bug.cgi?id=1596889,https://bugzilla.redhat.com/show_bug.cgi?id=524931, https://bugzilla.redhat.com/show_bug.cgi?id=557641)

Metadata Update from @mharmsen:
- Issue assigned to cheimes

Dogtag PKI is moving from Pagure issues to GitHub issues. This means that existing or new
issues will be reported and tracked through Dogtag PKI's GitHub Issue tracker.

This issue has been cloned to GitHub and is available here:
https://github.com/dogtagpki/pki/issues/739

If you want to receive further updates on the issue, please navigate to the
GitHub issue and click on Subscribe button.

Thank you for understanding, and we apologize for any inconvenience.

Metadata