#48713 kernel: traps: ns-slapd trap invalid opcode error:0 in libfreeblpriv3.so
Closed: wontfix Opened by hstrack.

Problem

Since the last update to CentOS 6.7 389 (as part of IPA) dies on some SSL connections in intel_aes_gcmINIT () (XEN guest)

I debugged the problem as staged here: http://www.port389.org/docs/389ds/FAQ/faq.html#sts=Debugging%C2%A0Crashes

 1020  debuginfo-install 389-ds-base
 1048  debuginfo-install nss-softokn-freebl
  gdb -ex 'set confirm off' -ex 'set pagination off' -ex 'thread apply all bt full' -ex 'quit' /usr/sbin/ns-slapd /var/log/dirsrv/slapd-INSTANCE/core.10565 > stacktrace.`date +%s`.txt 2>&1

Here is the stack trace

done.
Loaded symbols for /usr/lib64/libnssdbm3.so
warning: Can't read pathname for load map: Input/output error.
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffe3e31a000
Core was generated by `/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-INSTANCE -i /var/run/dirsrv/slapd-SS'.
Program terminated with signal 4, Illegal instruction.
#0  intel_aes_gcmINIT () at intel-gcm.s:71
71          vmovdqu      16*0(KS), T
Thread 141 (Thread 0x7f9404f89700 (LWP 10707)):
#0  0x00007f9467c743b3 in select () at ../sysdeps/unix/syscall-template.S:82
No locals.
#1  0x00007f946a195789 in ?? ()
No symbol table info available.
#2  0x0000000000000000 in ?? ()
No symbol table info available.
Thread 140 (Thread 0x7f940578a700 (LWP 10706)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
No locals.
#1  0x00007f9468584f99 in pt_TimedWait (cv=0x1c2d708, ml=0x1c2d870, timeout=<value optimized out>) at ../../../nspr/pr/src/pthreads/ptsynch.c:264
        rv = <value optimized out>

Workaround

Disabling TLS completely and enabling SSLv3 solved the problem for me, but may cause security issues:

dn: cn=encryption,cn=config
objectClass: top
objectClass: nsEncryptionConfig
cn: encryption
nsSSLSessionTimeout: 0
nsSSLClientAuth: allowed
nsSSL2: off
nsSSL3: on
nsTLS1: off
creatorsName: cn=server,cn=plugins,cn=config
modifiersName: cn=directory manager
createTimestamp: 20140725151745Z
modifyTimestamp: 20140725151809Z
nsSSL3Ciphers: -rsa_null_md5,+rsa_rc4_128_md5,+rsa_rc4_40_md5,+rsa_rc2_40_md5,
 +rsa_des_sha,+rsa_fips_des_sha,+rsa_3des_sha,+rsa_fips_3des_sha,+fortezza,+fo
 rtezza_rc4_128_sha,+fortezza_null,+tls_rsa_export1024_with_rc4_56_sha,+tls_rs
 a_export1024_with_des_cbc_sha,-TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,-TLS_RSA_W
 ITH_AES_128_GCM_SHA256
numSubordinates: 1

Only disabling GCM ciphers did not help

-TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,-TLS_RSA_W
 ITH_AES_128_GCM_SHA256

Another workaround may be exporting (not tested)

NSS_DISABLE_HW_AES=1

to workaround the HW_AES detection in libnss, as stated here: https://bugs.chromium.org/p/chromium/issues/detail?id=320524

additional info

Package versions

 rpm -qa | egrep -e '(389|nss)'
openssh-5.3p1-112.el6_7.x86_64
nss-util-3.19.1-2.el6_7.x86_64
389-ds-console-1.2.6-1.el6.noarch
nss-sysinit-3.19.1-8.el6_7.x86_64
openssl-1.0.1e-42.el6_7.2.x86_64
389-ds-base-libs-1.2.11.15-69.el6_7.x86_64
389-adminutil-1.1.19-1.el6.x86_64
389-admin-console-doc-1.1.8-1.el6.noarch
389-console-1.1.7-1.el6.noarch
389-admin-console-1.1.8-1.el6.noarch
389-dsgw-1.1.11-1.el6.x86_64
389-ds-base-debuginfo-1.2.11.15-69.el6_7.x86_64
nss-softokn-debuginfo-3.14.3-23.el6_7.x86_64
389-ds-base-1.2.11.15-69.el6_7.x86_64
openssh-server-5.3p1-112.el6_7.x86_64
nss-tools-3.19.1-8.el6_7.x86_64
389-ds-1.2.2-1.el6.noarch
openssh-clients-5.3p1-112.el6_7.x86_64
389-ds-console-doc-1.2.6-1.el6.noarch
openssl-devel-1.0.1e-42.el6_7.2.x86_64
nss-3.19.1-8.el6_7.x86_64
mod_nss-1.0.10-1.el6.x86_64
389-admin-1.1.35-1.el6.x86_64
nss-softokn-3.14.3-23.el6_7.x86_64
nss-debuginfo-3.19.1-8.el6_7.x86_64
nss-softokn-freebl-3.14.3-23.el6_7.x86_64
python-nss-0.16.0-1.el6.x86_64

possibly related problems

Before disabling TLS I got some other crashes that may be related

eb 22 09:59:23 mdc kernel: traps: ns-slapd[2134] trap invalid opcode ip:7fd7fa596d60 sp:7fd7be7bb518 error:0 in libfreeblpriv3.so[7fd7fa544000+72000]
Feb 22 16:57:50 mdc kernel: traps: ns-slapd[7772] trap invalid opcode ip:7f06fcc8fd60 sp:7f06e7ffe518 error:0 in libfreeblpriv3.so[7f06fcc3d000+72000]
Feb 22 17:11:35 mdc kernel: traps: ns-slapd[9101] trap invalid opcode ip:7f8cdf0d9d60 sp:7f8ccb7fd518 error:0 in libfreeblpriv3.so[7f8cdf087000+72000]
Feb 22 17:22:22 mdc kernel: ns-slapd[9576]: segfault at 8 ip 00007f816d2a2191 sp 00007f81762caa40 error 4 in libcos-plugin.so[7f816d29f000+b000]
Feb 22 17:24:10 mdc kernel: traps: ns-slapd[10236] trap invalid opcode ip:7f63305edd60 sp:7f6324f77518 error:0 in libfreeblpriv3.so[7f633059b000+72000]
Feb 22 17:25:25 mdc kernel: traps: ns-slapd[10579] trap invalid opcode ip:7f945ccf6d60 sp:7f9451d79518 error:0 in libfreeblpriv3.so[7f945cca4000+72000]
Feb 22 17:58:13 mdc kernel: ns-slapd[12293]: segfault at 0 ip 00007f5bcf380abd sp 00007f5bd9e65600 error 4 in libback-ldbm.so[7f5bcf35d000+9b000]
Feb 22 18:23:47 mdc kernel: traps: ns-slapd[15713] trap invalid opcode ip:7fab72bb5d60 sp:7fab5d578518 error:0 in libfreeblpriv3.so[7fab72b63000+72000]

As you noted, there is a known issue in nss-softokn.

Although you disabled GCM, it die not solve your server's crash. But could you please try setting this environment variable in /etc/sysconfig/dirsrv-YOUR_SERVER_INSTANCE?
NSS_DISABLE_HW_GCM=1

If it does not solve, please add this to the file, as well.

Another workaround may be exporting (not tested)
NSS_DISABLE_HW_AES=1

See also this ticket:
https://fedorahosted.org/389/ticket/48282

Hello, hstrack.

Are there any update on your crash problem?

If either of the environment variables solve your crash, is it okay to close this ticket?

Thanks.

Since we don't hear the response for 4 weeks, we are closing this ticket for now.

Please feel free to reopen it if necessary.

Thanks.

Metadata Update from @hstrack:
- Issue set to the milestone: 0.0 NEEDS_TRIAGE

Metadata Update from @vashirov:
- Issue set to the milestone: None (was: 0.0 NEEDS_TRIAGE)

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/1801

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: Invalid)

Metadata