Since the last update to CentOS 6.7 389 (as part of IPA) dies on some SSL connections in intel_aes_gcmINIT () (XEN guest)
I debugged the problem as staged here: http://www.port389.org/docs/389ds/FAQ/faq.html#sts=Debugging%C2%A0Crashes
1020 debuginfo-install 389-ds-base 1048 debuginfo-install nss-softokn-freebl gdb -ex 'set confirm off' -ex 'set pagination off' -ex 'thread apply all bt full' -ex 'quit' /usr/sbin/ns-slapd /var/log/dirsrv/slapd-INSTANCE/core.10565 > stacktrace.`date +%s`.txt 2>&1
Here is the stack trace
done. Loaded symbols for /usr/lib64/libnssdbm3.so warning: Can't read pathname for load map: Input/output error. warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffe3e31a000 Core was generated by `/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-INSTANCE -i /var/run/dirsrv/slapd-SS'. Program terminated with signal 4, Illegal instruction. #0 intel_aes_gcmINIT () at intel-gcm.s:71 71 vmovdqu 16*0(KS), T Thread 141 (Thread 0x7f9404f89700 (LWP 10707)): #0 0x00007f9467c743b3 in select () at ../sysdeps/unix/syscall-template.S:82 No locals. #1 0x00007f946a195789 in ?? () No symbol table info available. #2 0x0000000000000000 in ?? () No symbol table info available. Thread 140 (Thread 0x7f940578a700 (LWP 10706)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239 No locals. #1 0x00007f9468584f99 in pt_TimedWait (cv=0x1c2d708, ml=0x1c2d870, timeout=<value optimized out>) at ../../../nspr/pr/src/pthreads/ptsynch.c:264 rv = <value optimized out>
Disabling TLS completely and enabling SSLv3 solved the problem for me, but may cause security issues:
dn: cn=encryption,cn=config objectClass: top objectClass: nsEncryptionConfig cn: encryption nsSSLSessionTimeout: 0 nsSSLClientAuth: allowed nsSSL2: off nsSSL3: on nsTLS1: off creatorsName: cn=server,cn=plugins,cn=config modifiersName: cn=directory manager createTimestamp: 20140725151745Z modifyTimestamp: 20140725151809Z nsSSL3Ciphers: -rsa_null_md5,+rsa_rc4_128_md5,+rsa_rc4_40_md5,+rsa_rc2_40_md5, +rsa_des_sha,+rsa_fips_des_sha,+rsa_3des_sha,+rsa_fips_3des_sha,+fortezza,+fo rtezza_rc4_128_sha,+fortezza_null,+tls_rsa_export1024_with_rc4_56_sha,+tls_rs a_export1024_with_des_cbc_sha,-TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,-TLS_RSA_W ITH_AES_128_GCM_SHA256 numSubordinates: 1
Only disabling GCM ciphers did not help
-TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,-TLS_RSA_W ITH_AES_128_GCM_SHA256
Another workaround may be exporting (not tested)
NSS_DISABLE_HW_AES=1
to workaround the HW_AES detection in libnss, as stated here: https://bugs.chromium.org/p/chromium/issues/detail?id=320524
Package versions
rpm -qa | egrep -e '(389|nss)' openssh-5.3p1-112.el6_7.x86_64 nss-util-3.19.1-2.el6_7.x86_64 389-ds-console-1.2.6-1.el6.noarch nss-sysinit-3.19.1-8.el6_7.x86_64 openssl-1.0.1e-42.el6_7.2.x86_64 389-ds-base-libs-1.2.11.15-69.el6_7.x86_64 389-adminutil-1.1.19-1.el6.x86_64 389-admin-console-doc-1.1.8-1.el6.noarch 389-console-1.1.7-1.el6.noarch 389-admin-console-1.1.8-1.el6.noarch 389-dsgw-1.1.11-1.el6.x86_64 389-ds-base-debuginfo-1.2.11.15-69.el6_7.x86_64 nss-softokn-debuginfo-3.14.3-23.el6_7.x86_64 389-ds-base-1.2.11.15-69.el6_7.x86_64 openssh-server-5.3p1-112.el6_7.x86_64 nss-tools-3.19.1-8.el6_7.x86_64 389-ds-1.2.2-1.el6.noarch openssh-clients-5.3p1-112.el6_7.x86_64 389-ds-console-doc-1.2.6-1.el6.noarch openssl-devel-1.0.1e-42.el6_7.2.x86_64 nss-3.19.1-8.el6_7.x86_64 mod_nss-1.0.10-1.el6.x86_64 389-admin-1.1.35-1.el6.x86_64 nss-softokn-3.14.3-23.el6_7.x86_64 nss-debuginfo-3.19.1-8.el6_7.x86_64 nss-softokn-freebl-3.14.3-23.el6_7.x86_64 python-nss-0.16.0-1.el6.x86_64
Before disabling TLS I got some other crashes that may be related
eb 22 09:59:23 mdc kernel: traps: ns-slapd[2134] trap invalid opcode ip:7fd7fa596d60 sp:7fd7be7bb518 error:0 in libfreeblpriv3.so[7fd7fa544000+72000] Feb 22 16:57:50 mdc kernel: traps: ns-slapd[7772] trap invalid opcode ip:7f06fcc8fd60 sp:7f06e7ffe518 error:0 in libfreeblpriv3.so[7f06fcc3d000+72000] Feb 22 17:11:35 mdc kernel: traps: ns-slapd[9101] trap invalid opcode ip:7f8cdf0d9d60 sp:7f8ccb7fd518 error:0 in libfreeblpriv3.so[7f8cdf087000+72000] Feb 22 17:22:22 mdc kernel: ns-slapd[9576]: segfault at 8 ip 00007f816d2a2191 sp 00007f81762caa40 error 4 in libcos-plugin.so[7f816d29f000+b000] Feb 22 17:24:10 mdc kernel: traps: ns-slapd[10236] trap invalid opcode ip:7f63305edd60 sp:7f6324f77518 error:0 in libfreeblpriv3.so[7f633059b000+72000] Feb 22 17:25:25 mdc kernel: traps: ns-slapd[10579] trap invalid opcode ip:7f945ccf6d60 sp:7f9451d79518 error:0 in libfreeblpriv3.so[7f945cca4000+72000] Feb 22 17:58:13 mdc kernel: ns-slapd[12293]: segfault at 0 ip 00007f5bcf380abd sp 00007f5bd9e65600 error 4 in libback-ldbm.so[7f5bcf35d000+9b000] Feb 22 18:23:47 mdc kernel: traps: ns-slapd[15713] trap invalid opcode ip:7fab72bb5d60 sp:7fab5d578518 error:0 in libfreeblpriv3.so[7fab72b63000+72000]
stacktrace stacktrace.1456161864.txt.1
As you noted, there is a known issue in nss-softokn.
Although you disabled GCM, it die not solve your server's crash. But could you please try setting this environment variable in /etc/sysconfig/dirsrv-YOUR_SERVER_INSTANCE? NSS_DISABLE_HW_GCM=1
If it does not solve, please add this to the file, as well.
Another workaround may be exporting (not tested) NSS_DISABLE_HW_AES=1
See also this ticket: https://fedorahosted.org/389/ticket/48282
Hello, hstrack.
Are there any update on your crash problem?
If either of the environment variables solve your crash, is it okay to close this ticket?
Thanks.
Since we don't hear the response for 4 weeks, we are closing this ticket for now.
Please feel free to reopen it if necessary.
Metadata Update from @hstrack: - Issue set to the milestone: 0.0 NEEDS_TRIAGE
Metadata Update from @vashirov: - Issue set to the milestone: None (was: 0.0 NEEDS_TRIAGE)
389-ds-base is moving from Pagure to Github. This means that new issues and pull requests will be accepted only in 389-ds-base's github repository.
This issue has been cloned to Github and is available here: - https://github.com/389ds/389-ds-base/issues/1801
If you want to receive further updates on the issue, please navigate to the github issue and click on subscribe button.
subscribe
Thank you for understanding. We apologize for all inconvenience.
Metadata Update from @spichugi: - Issue close_status updated to: wontfix (was: Invalid)