Please provide the output of the following commands:

rpm -q 389-ds-base openldap nss nss-softokn nss-util nspr db4 openldap-debuginfo nss-debuginfo nss-softokn-debuginfo nss-util-debuginfo nspr-debuginfo db4-debuginfo

cat /etc/redhat-release

rpm -q 389-ds-base openldap nss nss-softokn nss-util nspr db4 openldap-debuginfo nss-debuginfo nss-softokn-debuginfo nss-util-debuginfo nspr-debuginfo db4-debuginfo

389-ds-base-1.2.11.15-48.el6_6.x86_64
openldap-2.4.39-8.el6.x86_64
nss-3.18.0-5.3.el6_6.x86_64
nss-softokn-3.14.3-22.el6_6.x86_64
nss-util-3.18.0-1.el6_6.x86_64
nspr-4.10.8-1.el6_6.x86_64
db4-4.7.25-18.el6_4.x86_64
package openldap-debuginfo is not installed
package nss-debuginfo is not installed
package nss-softokn-debuginfo is not installed
package nss-util-debuginfo is not installed
package nspr-debuginfo is not installed
package db4-debuginfo is not installed

cat /etc/redhat-release
CentOS release 6.6 (Final)

Thanks!

You're missing a lot of debuginfo packages, which explains why the stack traces are incompletely. If you do

debuginfo-install 389-ds-base

it should install the 389-ds-base-debuginfo and all of the others. If it doesn't, try doing

yum erase 389-ds-base-debuginfo

Then try the

debuginfo-install 389-ds-base

again.

Is this system fully up-to-date with the latest packages? It looks like you may be running into some problems involving interactions between 389, openldap, and nss, which should have been fixed by now.

Ok, I installed missing debuginfo packages, rebuilt stacktraces and re-uploaded overwrintng old ones.

The systems are up to date.

The stack traces still don't look quite right. Can you please confirm the versions of the packages:

rpm -q 389-ds-base 389-ds-base-debuginfo openldap nss nss-softokn nss-util nspr db4 openldap-debuginfo nss-debuginfo nss-softokn-debuginfo nss-util-debuginfo nspr-debuginfo db4-debuginfo

I notice this file in the stack trace:

11 0x00007f8d73b9e238 in krb5_bind () from /usr/lib64/dirsrv/plugins/libkrb5-plugin.so

What is /usr/lib64/dirsrv/plugins/libkrb5-plugin.so? That is not provided by either 389 or freeipa.

Ok, done a yum install nss-softokn-debuginfo nss-util-debuginfo

libkrb5-plugin is a custom plugin to check password against kerberos looking for principal in a custom attribute.

Reattaching stacktraces.

attachment
stacktrace.11211.2.txt

attachment
stacktrace.31720.txt

attachment
stacktrace.11211.txt

Replying to [comment:6 dael]:

Ok, done a yum install nss-softokn-debuginfo nss-util-debuginfo

libkrb5-plugin is a custom plugin to check password against kerberos looking for principal in a custom attribute.

Is it possible that this plugin is the source of the problems? Have you tried reproducing the problem without this plugin?

Reattaching stacktraces.

Unfortunately the problem happens rarely but in burst, only in our production servers where I can't stop the plugin. So I never be able to reproduce myself the crash.

I hoped that the problem could be identified by the stacktrace. I can add the tail of the access log from the core dump:
[18/Jun/2015:14:43:22 +0200] conn=87 op=1273 EXT oid=\"2.16.840.1.113730.3.5.6\" name=\"Netscape Replication Total Update Entry\"
[18/Jun/2015:14:43:22 +0200] conn=87 op=1273 RESULT err=0 tag=120 nentries=0 etime=0
[18/Jun/2015:14:43:22 +0200] conn=87 op=1274 EXT oid=\"2.16.840.1.113730.3.5.6\" name=\"Netscape Replication Total Update Entry\"
[18/Jun/2015:14:43:22 +0200] conn=87 op=1274 RESULT err=0 tag=120 nentries=0 etime=0
[18/Jun/2015:14:43:22 +0200] conn=87 op=1275 EXT oid=\"2.16.840.1.113730.3.5.6\" name=\"Netscape Replication Total Update Entry\"
[18/Jun/2015:14:43:22 +0200] conn=87 op=1275 RESULT err=0 tag=120 nentries=0 etime=0
=0 etime=0
e=0

"

Thank you

Replying to [comment:8 dael]:

Unfortunately the problem happens rarely but in burst, only in our production servers where I can't stop the plugin. So I never be able to reproduce myself the crash.

I hoped that the problem could be identified by the stacktrace.

The stacktrace.31720.txt​ is not very useful - it seems corrupted, or we don't have the right combination of debuginfo packages.
The stacktrace.11211.txt​ points to the krb5-plugin.

I can add the tail of the access log from the core dump:
[18/Jun/2015:14:43:22 +0200] conn=87 op=1273 EXT oid=\"2.16.840.1.113730.3.5.6\" name=\"Netscape Replication Total Update Entry\"
[18/Jun/2015:14:43:22 +0200] conn=87 op=1273 RESULT err=0 tag=120 nentries=0 etime=0
[18/Jun/2015:14:43:22 +0200] conn=87 op=1274 EXT oid=\"2.16.840.1.113730.3.5.6\" name=\"Netscape Replication Total Update Entry\"
[18/Jun/2015:14:43:22 +0200] conn=87 op=1274 RESULT err=0 tag=120 nentries=0 etime=0
[18/Jun/2015:14:43:22 +0200] conn=87 op=1275 EXT oid=\"2.16.840.1.113730.3.5.6\" name=\"Netscape Replication Total Update Entry\"
[18/Jun/2015:14:43:22 +0200] conn=87 op=1275 RESULT err=0 tag=120 nentries=0 etime=0
=0 etime=0
e=0

The problem is that the access log is buffered, so the actual operation that caused the crash is still in the servers memory, not in the access log file. See http://www.port389.org/docs/389ds/FAQ/faq.html#printing-access-log-buffer for information about how to print out the access log buffer from a core file.

Is the krb5-plugin open source? Could you share the source somehow, so that we can review it? Where did it come from? Who wrote it?

"

Thank you

I know the log is buffered, the log I pasted was produced by that procedure.

The plugin is home made and I can share it, should it be uploaded here or it's better i.e. on a page of our wiki?

Another thing I can say about the crash: we have 3 master servers and 2 slave servers, each with 30 databases, when the problem occurs all these server crashes, master and slaves, not exactly at the same time but in minutes. Anyway we also have other 15 server that receive update from the 3 masters but only for 2 databases, these 15 servers don't crash, also these have krb5-plugin active.

Replying to [comment:10 dael]:

I know the log is buffered, the log I pasted was produced by that procedure.

Ok. Interesting - it is a total update operation. Are all of the servers running 1.2.11.15-48?

The plugin is home made and I can share it, should it be uploaded here or it's better i.e. on a page of our wiki?

If you can share a page of your wiki, that would be fine.

Another thing I can say about the crash: we have 3 master servers and 2 slave servers, each with 30 databases, when the problem occurs all these server crashes, master and slaves, not exactly at the same time but in minutes. Anyway we also have other 15 server that receive update from the 3 masters but only for 2 databases, these 15 servers don't crash, also these have krb5-plugin active.

Ok. We need to figure out how to get a better stack trace. Can you please confirm the versions of the packages:

rpm -q 389-ds-base 389-ds-base-debuginfo openldap nss nss-softokn nss-util nspr db4 openldap-debuginfo nss-debuginfo nss-softokn-debuginfo nss-util-debuginfo nspr-debuginfo db4-debuginfo

The version of the debuginfo package must match exactly the version of the corresponding binary package, or stack traces will be hard to read.

Hello dael,

Could there be any progress in your investigation/debugging?

Any updates? If not, we are going to close this bug.

Closing this ticket for now.
Please feel free to reopen it when it occurs and more info is available.

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/1531

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.