Fix for JSS segfault: https://bugzilla.mozilla.org/show_bug.cgi?id=1371147

As for why the HSM fails to do the export, we are still looking into that.

Patch for testing the old 3DES code path in wrapped-key p12 recovery attached.

In kra/CS.cfg set kra.legacyPKCS12=true to use the old behaviour.

Observations from the PKCS #11 debug logging of HSM module:

During the RSA_PKCS unwrapping of session key (by storage key) the following warning is reported:

2017-06-07 22:54:05 [14977] t00070f57f27f0000: pkcs11: 000008D8 Unsafe assumption: CKA_DECRYPT should be in template, not assumed to be based on CKA_ENCRYPT

Then during second phase of unwrapping - the unwrapping of archived private key into token, by session key - the following warning occurs:

2017-06-07 22:54:05 [14977] t00070f57f27f0000: pkcs11-sam: 000008ce Warning: Key type CKK_RSAid "061ab42b72b3261917378dbbee2b87adf7e7a8ba" hash "2c291d6df6c24ac85383ba221dd4b975422ab4c1"
2017-06-07 22:54:05 [14977] t00070f57f27f0000: pkcs11-sam: 000008ce Warning: CKA_DECRYPT set in unwrapping key, the unwrapped key will be insecure; set CKNFAST_OVERRIDE_SECURITY_ASSURANCES=unwrap_kek to avoid

With the non-crashing JSS update, the error message we get from NSS is:

(-8190) security library: received bad data.

This is the message for SEC_ERROR_BAD_DATA.

There are quite a few PKCS #11 error codes that map to this NSS error code:

lib/pk11wrap/pk11err.c
51:    MAPERROR(CKR_SLOT_ID_INVALID, SEC_ERROR_BAD_DATA)
55:    MAPERROR(CKR_ATTRIBUTE_TYPE_INVALID, SEC_ERROR_BAD_DATA)
56:    MAPERROR(CKR_ATTRIBUTE_VALUE_INVALID, SEC_ERROR_BAD_DATA)
58:    MAPERROR(CKR_DATA_INVALID, SEC_ERROR_BAD_DATA)
64:    MAPERROR(CKR_ENCRYPTED_DATA_INVALID, SEC_ERROR_BAD_DATA)
65:    MAPERROR(CKR_ENCRYPTED_DATA_LEN_RANGE, SEC_ERROR_BAD_DATA)
75:    MAPERROR(CKR_MECHANISM_PARAM_INVALID, SEC_ERROR_BAD_DATA)
77:    MAPERROR(CKR_OBJECT_HANDLE_INVALID, SEC_ERROR_BAD_DATA)
87:    MAPERROR(CKR_SESSION_HANDLE_INVALID, SEC_ERROR_BAD_DATA)
92:    MAPERROR(CKR_TEMPLATE_INCOMPLETE, SEC_ERROR_BAD_DATA)
93:    MAPERROR(CKR_TEMPLATE_INCONSISTENT, SEC_ERROR_BAD_DATA)

And there are a few places where NSS raises this on its own; in most cases it
is how NSS handles an invalid PK11 session handle.

Gerrit review for change to restore 3DES codepath, and config to choose:
https://review.gerrithub.io/#/c/364708/

My pbkdf2 program indicates that the Thales nethsm does not support CKM_PKCS5_PBKD2:

~/pbkdf2 $PWD/alias "NHSM-RPATTATH-SOFTCARD" SECret.890
Slot does CKM_PKCS5_PBKD2? 0
Looking for a better slot...
in password callback; arg = SECret.890
 slot: 'NSS FIPS 140-2 User Private Key Services'

The PK11_PBEGenKey then searches for a slot that can do the
resquested operation, and attempts to initialise that slot and do the keygen there. If that succeeds, for the private key encryption it attempts to move
the symmetric key back onto the token where the private key lives.
I am now confident that this is where the problem lies; one of those stages
is failing. I think my next step will be deploying an instrumented NSS build so that I can get more insight into what is happening when it's running inside Dogtag, i.e. to see exactly where it is failing and determine if there is a way
to make it work, or whether we need to implement other hacks or just degrade
to 3DES on HSMs that don't support PBKDF2.

Author: Fraser Tweedale ftweedal@redhat.com
Date: Thu Jun 8 14:25:23 2017 +1000

KRA PKCS #12 export: add config to use 3DES PBE encryption
Restore the 3DES PKCS #12 key recovery code path, alongside the new
AES variant, which is broken on Thales nethsm.  Add the
'kra.legacyPKCS12' config for selecting which version to use, with
the default value of 'true' (i.e., use 3DES).
Part of: https://pagure.io/dogtagpki/issue/2728
Change-Id: Ic02fe8ba3a4c2c049913ff48d3f6dfdc830b4360

Dogtag PKI is moving from Pagure issues to GitHub issues. This means that existing or new
issues will be reported and tracked through Dogtag PKI's GitHub Issue tracker.

This issue has been cloned to GitHub and is available here:
https://github.com/dogtagpki/pki/issues/2848

If you want to receive further updates on the issue, please navigate to the
GitHub issue and click on Subscribe button.

Thank you for understanding, and we apologize for any inconvenience.