#3130 ipa-server-install fails in pkispawn step with a java coredump
Closed: invalid by cipherboy. Opened by frenaud.

In freeipa nightly tests on rawhide, ipa-server-install fails in the pkispawn step with a java coredump.
See our nightly tests PR 156, for instance the logs in the test dns_locations:

Mar 15 14:07:47 master.ipa.test systemd[1]: Started PKI Tomcat Server pki-tomcat.
Mar 15 14:07:47 master.ipa.test audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 15 14:07:47 master.ipa.test server[26160]: Java virtual machine used: /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
Mar 15 14:07:47 master.ipa.test server[26160]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/lib/java/commons-daemon.jar
Mar 15 14:07:47 master.ipa.test server[26160]: main class used: org.apache.catalina.startup.Bootstrap
Mar 15 14:07:47 master.ipa.test server[26160]: flags used:
Mar 15 14:07:47 master.ipa.test server[26160]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager     -Djava.security.manager     -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
Mar 15 14:07:47 master.ipa.test server[26160]: arguments used: start
Mar 15 14:07:50 master.ipa.test java[26160]: usr/lib/api/apiutil.c Could not open /run/lock/opencryptoki/LCK..APIlock
Mar 15 14:08:00 master.ipa.test server[26160]: #
Mar 15 14:08:00 master.ipa.test server[26160]: # A fatal error has been detected by the Java Runtime Environment:
Mar 15 14:08:00 master.ipa.test server[26160]: #
Mar 15 14:08:00 master.ipa.test server[26160]: #  SIGSEGV (0xb) at pc=0x00007f2cf1971fbe, pid=26160, tid=0x00007f2cdc433700
Mar 15 14:08:00 master.ipa.test server[26160]: #
Mar 15 14:08:00 master.ipa.test server[26160]: # JRE version: OpenJDK Runtime Environment (8.0_242-b08) (build 1.8.0_242-b08)
Mar 15 14:08:00 master.ipa.test server[26160]: # Java VM: OpenJDK 64-Bit Server VM (25.242-b08 mixed mode linux-amd64 compressed oops)
Mar 15 14:08:00 master.ipa.test server[26160]: # Problematic frame:
Mar 15 14:08:00 master.ipa.test server[26160]: # V  [libjvm.so+0x701fbe]  JNIHandleBlock::oops_do(OopClosure*)+0xae
Mar 15 14:08:00 master.ipa.test server[26160]: #
Mar 15 14:08:00 master.ipa.test server[26160]: # Core dump written. Default location: /usr/share/tomcat/core or core.26160
Mar 15 14:08:00 master.ipa.test server[26160]: #
Mar 15 14:08:00 master.ipa.test server[26160]: # An error report file with more information is saved as:
Mar 15 14:08:00 master.ipa.test server[26160]: # /tmp/hs_err_pid26160.log
Mar 15 14:08:00 master.ipa.test server[26160]: #
Mar 15 14:08:00 master.ipa.test server[26160]: # If you would like to submit a bug report, please visit:
Mar 15 14:08:00 master.ipa.test server[26160]: #   http://bugreport.java.com/bugreport/crash.jsp
Mar 15 14:08:00 master.ipa.test server[26160]: #
Mar 15 14:08:00 master.ipa.test audit[26160]: ANOM_ABEND auid=4294967295 uid=17 gid=17 ses=4294967295 subj=system_u:system_r:tomcat_t:s0 pid=26160 comm=564D20546872656164 exe="/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.fc33.x86_64/jre/bin/java" sig=6 res=1
Mar 15 14:08:00 master.ipa.test systemd[1]: Created slice system-systemd\x2dcoredump.slice.
Mar 15 14:08:00 master.ipa.test systemd[1]: Started Process Core Dump (PID 26207/UID 0).
Mar 15 14:08:00 master.ipa.test audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@0-26207-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 15 14:08:01 master.ipa.test ns-slapd[25787]: [15/Mar/2020:14:08:01.351096716 +0000] - ERR - ipa-topology-plugin - ipa_topo_util_get_replica_conf: server configuration missing
Mar 15 14:08:01 master.ipa.test ns-slapd[25787]: [15/Mar/2020:14:08:01.357399091 +0000] - ERR - ipa-topology-plugin - ipa_topo_util_get_replica_conf: cannot create replica
Mar 15 14:08:03 master.ipa.test systemd[1]: pki-tomcatd@pki-tomcat.service: Main process exited, code=killed, status=6/ABRT
Mar 15 14:08:03 master.ipa.test systemd[1]: pki-tomcatd@pki-tomcat.service: Failed with result 'signal'.
Mar 15 14:08:03 master.ipa.test audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Mar 15 14:08:03 master.ipa.test systemd[1]: pki-tomcatd@pki-tomcat.service: Consumed 13.619s CPU time.
Mar 15 14:08:04 master.ipa.test systemd-coredump[26208]: Process 26160 (java) of user 17 dumped core.
                                                         Stack trace of thread 26172:
                                                         #0  0x00007f2cf22779f5 raise (libc.so.6 + 0x3c9f5)
                                                         #1  0x00007f2cf2260892 abort (libc.so.6 + 0x25892)
                                                         #2  0x00007f2cf148b631 _ZN2os5abortEb.cold (libjvm.so + 0x21b631)
                                                         #3  0x00007f2cf1db4a6b _ZN7VMError14report_and_dieEv (libjvm.so + 0xb44a6b)
                                                         #4  0x00007f2cf1bacbcd JVM_handle_linux_signal (libjvm.so + 0x93cbcd)
                                                         #5  0x00007f2cf1ba846c _Z13signalHandleriP9siginfo_tPv (libjvm.so + 0x93846c)
                                                         #6  0x00007f2cf2277a80 __restore_rt (libc.so.6 + 0x3ca80)
                                                         #7  0x00007f2cf1971fbe _ZN14JNIHandleBlock7oops_doEP10OopClosure (libjvm.so + 0x701fbe)
                                                         #8  0x00007f2cf1d58b6c _ZN10JavaThread7oops_doEP10OopClosureP10CLDClosureP15CodeBlobClosure (libjvm.so + 0xae8b6c)
                                                         #9  0x00007f2cf1d5c6e1 _ZN7Threads25possibly_parallel_oops_doEP10OopClosureP10CLDClosureP15CodeBlobClosure (libjvm.so + 0xaec6e1)
                                                         #10 0x00007f2cf1855831 _ZN16GenCollectedHeap13process_rootsEbNS_14ScanningOptionEP10OopClosureS2_P10CLDClosureS4_P20CodeBlobToOopClosure (libjvm.so + 0x5e5831)
                                                         #11 0x00007f2cf1855aa2 _ZN16GenCollectedHeap17gen_process_rootsEibbNS_14ScanningOptionEbP16OopsInGenClosureS2_P10CLDClosure (libjvm.so + 0x5e5aa2)
                                                         #12 0x00007f2cf1778b54 _ZN16DefNewGeneration7collectEbbmb (libjvm.so + 0x508b54)
                                                         #13 0x00007f2cf185500d _ZN16GenCollectedHeap13do_collectionEbbmbi (libjvm.so + 0x5e500d)
                                                         #14 0x00007f2cf16f4164 _ZN18GenCollectorPolicy25satisfy_failed_allocationEmb (libjvm.so + 0x484164)
                                                         #15 0x00007f2cf1db58dd _ZN26VM_GenCollectForAllocation4doitEv (libjvm.so + 0xb458dd)
                                                         #16 0x00007f2cf1dbd1af _ZN12VM_Operation8evaluateEv (libjvm.so + 0xb4d1af)
                                                         #17 0x00007f2cf1dbbed9 _ZN8VMThread18evaluate_operationEP12VM_Operation (libjvm.so + 0xb4bed9)
                                                         #18 0x00007f2cf1dbc6a3 _ZN8VMThread4loopEv (libjvm.so + 0xb4c6a3)
                                                         #19 0x00007f2cf1dbbb51 _ZN8VMThread3runEv (libjvm.so + 0xb4bb51)
                                                         #20 0x00007f2cf1b9fd78 _ZL10java_startP6Thread (libjvm.so + 0x92fd78)
                                                         #21 0x00007f2cf2201462 start_thread (libpthread.so.0 + 0x9462)
                                                         #22 0x00007f2cf233cb53 __clone (libc.so.6 + 0x101b53)
                                                         Stack trace of thread 26160:
                                                         #0  0x00007f2cf2202aa7 __pthread_clockjoin_ex (libpthread.so.0 + 0xaaa7)
                                                         #1  0x00007f2cf24191fd ContinueInNewThread0 (libjli.so + 0xa1fd)
                                                         #2  0x00007f2cf2417004 ContinueInNewThread (libjli.so + 0x8004)
                                                         #3  0x00007f2cf24133bd JLI_Launch (libjli.so + 0x43bd)
                                                         #4  0x000056113b4b60d2 main (java + 0x10d2)
                                                         #5  0x00007f2cf2262052 __libc_start_main (libc.so.6 + 0x27052)
                                                         #6  0x000056113b4b610e _start (java + 0x110e)
                                                         Stack trace of thread 26171:
                                                         #0  0x00007f2cf2207b42 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xfb42)
                                                         #1  0x00007f2cf1bab3fb _ZN2os13PlatformEvent4parkEv (libjvm.so + 0x93b3fb)
                                                         #2  0x00007f2cf1b5e805 _ZN7Monitor5ILockEP6Thread (libjvm.so + 0x8ee805)
                                                         #3  0x00007f2cf1b5effa _ZN7Monitor28lock_without_safepoint_checkEv (libjvm.so + 0x8eeffa)
                                                         #4  0x00007f2cf1c3ff71 _ZN20SafepointSynchronize5blockEP10JavaThread.part.0 (libjvm.so + 0x9cff71)
                                                         #5  0x00007f2cf1d58180 _ZN10JavaThread44check_safepoint_and_suspend_for_native_transEPS_ (libjvm.so + 0xae8180)
                                                         #6  0x00007f2cf193ebf5 jni_SetIntField (libjvm.so + 0x6cebf5)
                                                         #7  0x00007f2cdc0cfa21 Java_java_net_PlainSocketImpl_socketConnect (libnet.so + 0xea21)
                                                         #8  0x00007f2cdd0183c7 n/a (n/a + 0x0)
                                                         #9  0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #10 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #11 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #12 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #13 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #14 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #15 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #16 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #17 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #18 0x00007f2cdd007e54 n/a (n/a + 0x0)
                                                         #19 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #20 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #21 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #22 0x00007f2cdd007d80 n/a (n/a + 0x0)
                                                         #23 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #24 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #25 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #26 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #27 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #28 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #29 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #30 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #31 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #32 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #33 0x00007f2cdd008042 n/a (n/a + 0x0)
                                                         #34 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #35 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #36 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #37 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #38 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #39 0x00007f2cdd0004e7 n/a (n/a + 0x0)
                                                         #40 0x00007f2cf1915eaa _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread (libjvm.so + 0x6a5eaa)
                                                         #41 0x00007f2cf1c23a26 _ZN10Reflection6invokeE19instanceKlassHandle12methodHandle6Handleb14objArrayHandle9BasicTypeS3_bP6Thread (libjvm.so + 0x9b3a26)
                                                         #42 0x00007f2cf1c24b3b _ZN10Reflection13invoke_methodEP7oopDesc6Handle14objArrayHandleP6Thread (libjvm.so + 0x9b4b3b)
                                                         #43 0x00007f2cf198c0e9 JVM_InvokeMethod (libjvm.so + 0x71c0e9)
                                                         #44 0x00007f2cdd43c5ad n/a (n/a + 0x0)
                                                         #45 0x00007f2cdd43b7fe n/a (n/a + 0x0)
                                                         #46 0x00007f2cdd007d80 n/a (n/a + 0x0)
                                                         #47 0x00007f2cdd0004e7 n/a (n/a + 0x0)
                                                         #48 0x00007f2cf1915eaa _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread (libjvm.so + 0x6a5eaa)
                                                         #49 0x00007f2cf1979951 JVM_DoPrivileged (libjvm.so + 0x709951)
                                                         #50 0x00007f2cdd208255 n/a (n/a + 0x0)
                                                         #51 0x00007f2cdd007d80 n/a (n/a + 0x0)
                                                         #52 0x00007f2cdd007d80 n/a (n/a + 0x0)
                                                         #53 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #54 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #55 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #56 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #57 0x00007f2cdd007d80 n/a (n/a + 0x0)
                                                         #58 0x00007f2cdd008042 n/a (n/a + 0x0)
                                                         #59 0x00007f2cdd0077d0 n/a (n/a + 0x0)
                                                         #60 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #61 0x00007f2cdd008042 n/a (n/a + 0x0)
                                                         #62 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         #63 0x00007f2cdd007ffd n/a (n/a + 0x0)
                                                         Stack trace of thread 26173:
                                                         #0  0x00007f2cf2207b42 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xfb42)
                                                         #1  0x00007f2cf1bab3fb _ZN2os13PlatformEvent4parkEv (libjvm.so + 0x93b3fb)
                                                         #2  0x00007f2cf1b8db6d _ZN13ObjectMonitor4waitElbP6Thread (libjvm.so + 0x91db6d)
                                                         #3  0x00007f2cf1d1fa5b _ZN18ObjectSynchronizer4waitE6HandlelP6Thread (libjvm.so + 0xaafa5b)
                                                         #4  0x00007f2cf197484b JVM_MonitorWait (libjvm.so + 0x70484b)
                                                         #5  0x00007f2cdd0183c7 n/a (n/a + 0x0)
                                                         #6  0x00007f2cdd77b7b8 n/a (n/a + 0x0)
Mar 15 14:08:04 master.ipa.test systemd[1]: systemd-coredump@0-26207-0.service: Succeeded.
Mar 15 14:08:04 master.ipa.test audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@0-26207-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 15 14:08:04 master.ipa.test systemd[1]: systemd-coredump@0-26207-0.service: Consumed 3.219s CPU time.

Not surprising -- and not our fault I don't think. See:

ML: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/2UZJAFWGC7LGEKHR5ORYLQQHL6FQH4NT/

BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1813550

Metadata Update from @cipherboy:
- Custom field component adjusted to None
- Custom field feature adjusted to None
- Custom field origin adjusted to None
- Custom field proposedmilestone adjusted to None
- Custom field proposedpriority adjusted to None
- Custom field reviewer adjusted to None
- Custom field type adjusted to None
- Custom field version adjusted to None

Hey @cipherboy, do you have some way to check if this crash is also present with this pending f32 build of openjdk? I was able to reproduce the crash with this update on fedora 32 too:

https://bodhi.fedoraproject.org/updates/FEDORA-2020-af190951f6

I'd rather not have a crashing JVM get pushed to stable right before f32 gets final freeze'd.

@decathorpe I tried reproducing this just on F32 Java and JSS, but I wasn't able to. I think you got a response from Andrew though, so should I go ahead and close this and let it get sorted out on BZ / devel-list?

@cipherboy yeah that's fine.

Please see OpenJDK 8 / gcc 10.x bug for more details: https://bugzilla.redhat.com/show_bug.cgi?id=1813550

@frenaud -- please open a new bug if this persists after GCC 10.x / OpenJDK 8 is fixed. Thanks!!

Metadata Update from @cipherboy:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1813550
- Issue close_status updated to: invalid
- Issue status updated to: Closed (was: Open)

Dogtag PKI is moving from Pagure issues to GitHub issues. This means that existing or new
issues will be reported and tracked through Dogtag PKI's GitHub Issue tracker.

This issue has been cloned to GitHub and is available here:
https://github.com/dogtagpki/pki/issues/3247

If you want to receive further updates on the issue, please navigate to the
GitHub issue and click on Subscribe button.

Thank you for understanding, and we apologize for any inconvenience.

Metadata