Issue #1305: CRL publishing fails after Java heap out of memory error - dogtagpki

dogtagpki

#1305 CRL publishing fails after Java heap out of memory error

Closed: Fixed Opened Mar 17, 2015 by mharmsen.

CRL Publishing throws exception in the catalina.out and debug log after creation of CRL, fails to publish properly, and prevents any subsequent CRL generation from occurring until CA subsystem is restarted.

How reproducible:

Always

Steps to Reproduce:

1. Create CA with 2G of RAM
2. Stock CS.cfg config with the following changes:
   ca.crl.MasterCRL.dailyUpdates=0:00,6:00,12:00,18:00;0:00,6:00,12:00,18:00
   ca.crl.MasterCRL.enableCRLCache=false
   ca.crl.MasterCRL.nextUpdateGracePeriod=6840
   ca.crl.MasterCRL.signingAlgorithm=SHA1withRSA
   ca.crl.MasterCRL.extension.AuthorityKeyIdentifier.enable=true
   ca.crl.MasterCRL.extension.AuthorityKeyIdentifier.id=MasterCRL
3. Enroll 1.5M certificates
4. Revoke 1.5M certificates
5. Generate CRL

Actual results:

- Get trace in log and CRL fails to generate:
- catalina.out trace:
~~~
Exception in thread "CertStatusUpdateThread" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2894)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:117)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:407)
        at java.lang.StringBuffer.append(StringBuffer.java:241)
        at java.text.MessageFormat.subformat(MessageFormat.java:1318)
        at java.text.MessageFormat.format(MessageFormat.java:853)
        at java.text.Format.format(Format.java:157)
        at java.text.MessageFormat.format(MessageFormat.java:829)
        at org.apache.naming.StringManager.getString(StringManager.java:121)
        at org.apache.naming.StringManager.getString(StringManager.java:144)
        at org.apache.naming.resources.FileDirContext.lookup(FileDirContext.java:215)
        at org.apache.naming.resources.ProxyDirContext.lookup(ProxyDirContext.java:294)
        at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:1934)
        at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:1805)
        at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:888)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1341)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1220)
        at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2402)
        at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1424)
        at java.util.ResourceBundle.findBundle(ResourceBundle.java:1383)
        at java.util.ResourceBundle.findBundle(ResourceBundle.java:1310)
        at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1252)
        at java.util.ResourceBundle.getBundle(ResourceBundle.java:730)
        at com.netscape.cmscore.apps.CMSEngine.getLogMessage(CMSEngine.java:1310)
        at com.netscape.cmscore.apps.CMSEngine.getLogMessage(CMSEngine.java:1377)
        at com.netscape.certsrv.apps.CMS.getLogMessage(CMS.java:616)
        at com.netscape.cmscore.dbs.CertificateRepository.updateCertStatus(CertificateRepository.java:267)
        at com.netscape.cmscore.dbs.CertStatusUpdateThread.run(CertificateRepository.java:1928)
elegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:622)
        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAsPrivileged(Subject.java:537)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:218)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:210)
        at org.apache.catalina.core.ApplicationFilterChain.access$0(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:542)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
        at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
        at java.lang.Thread.run(Thread.java:701)
Version-Release number of selected component (if applicable):
~~~
- debug log trace:
~~~
[26/Aug/2014:14:28:12][CRLIssuingPoint-MasterCRL]: update CRL error Failed constructing CRL : java.lang.OutOfMemoryError: Java heap space
Failed constructing CRL : java.lang.OutOfMemoryError: Java heap space
    at com.netscape.ca.CRLIssuingPoint.updateCRLNow(CRLIssuingPoint.java:2694)
    at com.netscape.ca.CRLIssuingPoint.updateCRLNow(CRLIssuingPoint.java:2270)
    at com.netscape.ca.CRLIssuingPoint.updateCRL(CRLIssuingPoint.java:1752)
    at com.netscape.ca.CRLIssuingPoint.run(CRLIssuingPoint.java:1703)
    at java.lang.Thread.run(Thread.java:701)
~~~

Expected results:

- CRL generates and publishes correctly without trace and next scheduled CRL is generated.
OR
- CA handles exception, cleans thread, and continues with next scheduled CRL generation.

Metadata Update from @mharmsen:
- Issue set to the milestone: 10.2.2

Feb 27, 2017

dmoluguw commented Oct 4, 2020

Dogtag PKI is moving from Pagure issues to GitHub issues. This means that existing or new
issues will be reported and tracked through Dogtag PKI's GitHub Issue tracker.

This issue has been cloned to GitHub and is available here:
https://github.com/dogtagpki/pki/issues/1867

If you want to receive further updates on the issue, please navigate to the
GitHub issue and click on Subscribe button.

Thank you for understanding, and we apologize for any inconvenience.

Metadata

Assignee

None

Tags

None

Blocking

None

Depending on

None

Priority

Normal

type

defect

component

CRL

rhbz

https://bugzilla.redhat.com/show_bug.cgi?id=1134405
https://bugzilla.redhat.com/show_bug.cgi?id=1202522

origin

RHCust

dogtagpki

#1305 CRL publishing fails after Java heap out of memory error Closed: Fixed Opened Mar 17, 2015 by mharmsen.

Metadata

#1305 CRL publishing fails after Java heap out of memory error

Closed: Fixed Opened Mar 17, 2015 by mharmsen.