Skip to content

Instantly share code, notes, and snippets.

@cfalzone
Created May 8, 2014 17:54
Show Gist options
  • Save cfalzone/f64cad070afc2eae4d22 to your computer and use it in GitHub Desktop.
Save cfalzone/f64cad070afc2eae4d22 to your computer and use it in GitHub Desktop.
Java Deadlock on Startup with OSGI Plugin deployed.
Found one Java-level deadlock:
=============================
"http-8080-4":
waiting to lock monitor 0x00007f931c772178 (object 0x00000006c00274e0, a org.apache.catalina.loader.WebappClassLoader),
which is held by "http-8080-2"
"http-8080-2":
waiting to lock monitor 0x00007f931c7720c8 (object 0x00000006a64b2b50, a com.dotmarketing.osgi.UrlOsgiClassLoader),
which is held by "fileinstall-/data/wwwroot/dotcms/current/tomcat/webapps/../../dotCMS/WEB-INF/felix/load"
"fileinstall-/data/wwwroot/dotcms/current/tomcat/webapps/../../dotCMS/WEB-INF/felix/load":
waiting to lock monitor 0x00007f931c772178 (object 0x00000006c00274e0, a org.apache.catalina.loader.WebappClassLoader),
which is held by "http-8080-2"
Java stack information for the threads listed above:
===================================================
"http-8080-4":
at org.apache.velocity.tools.struts.MessageResourcesTool.init(MessageResourcesTool.java:62)
at org.apache.velocity.tools.struts.ActionMessagesTool.init(ActionMessagesTool.java:80)
at org.apache.velocity.tools.struts.ErrorsTool.init(ErrorsTool.java:62)
at org.apache.velocity.tools.view.ViewToolInfo.getInstance(ViewToolInfo.java:192)
at org.apache.velocity.tools.view.servlet.ServletToolboxManager.getToolbox(ServletToolboxManager.java:401)
at org.apache.velocity.tools.view.XMLToolboxManager.getToolboxContext(XMLToolboxManager.java:181)
at com.dotmarketing.util.VelocityUtil.getWebContext(VelocityUtil.java:208)
at com.dotmarketing.util.VelocityUtil.getWebContext(VelocityUtil.java:178)
at com.dotmarketing.velocity.VelocityServlet.doLiveMode(VelocityServlet.java:495)
at com.dotmarketing.velocity.VelocityServlet.service(VelocityServlet.java:232)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
at com.dotmarketing.filters.CMSFilter.doFilter(CMSFilter.java:510)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.dotmarketing.filters.AutoLoginFilter.doFilter(AutoLoginFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.dotmarketing.cms.urlmap.filters.URLMapFilter.doFilter(URLMapFilter.java:334)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.dotmarketing.filters.TimeMachineFilter.doFilter(TimeMachineFilter.java:162)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.dotmarketing.filters.CharsetEncodingFilter.doFilter(CharsetEncodingFilter.java:146)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:724)
"http-8080-2":
at com.dotmarketing.osgi.UrlOsgiClassLoader.loadClass(UrlOsgiClassLoader.java:71)
- waiting to lock <0x00000006a64b2b50> (a com.dotmarketing.osgi.UrlOsgiClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
- locked <0x00000006a8326648> (a java.lang.Object)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
- locked <0x00000006a8326588> (a java.lang.Object)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1431)
- locked <0x00000006a8326468> (a java.lang.String)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1361)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:792)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2527)
- locked <0x00000006a7fc28a8> (a java.lang.String)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1010)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1483)
- locked <0x00000006a7fc28a8> (a java.lang.String)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1361)
at com.thoughtworks.xstream.XStream.<init>(XStream.java:381)
at com.thoughtworks.xstream.XStream.<init>(XStream.java:356)
at com.dotcms.publisher.business.PublishAuditHistory.getObjectFromString(PublishAuditHistory.java:126)
at com.dotcms.publisher.mapper.PublishAuditStatusMapper.mapObject(PublishAuditStatusMapper.java:16)
at com.dotcms.publisher.business.PublishAuditAPIImpl.getPublishAuditStatus(PublishAuditAPIImpl.java:209)
at com.dotcms.rest.AuditPublishingResource.get(AuditPublishingResource.java:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1483)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1414)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1363)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1353)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:414)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)
at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.dotmarketing.filters.CMSFilter.doFilter(CMSFilter.java:121)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.dotmarketing.filters.AutoLoginFilter.doFilter(AutoLoginFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.dotmarketing.cms.urlmap.filters.URLMapFilter.doFilter(URLMapFilter.java:85)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.dotmarketing.filters.TimeMachineFilter.doFilter(TimeMachineFilter.java:162)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.dotmarketing.filters.CharsetEncodingFilter.doFilter(CharsetEncodingFilter.java:146)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:724)
"fileinstall-/data/wwwroot/dotcms/current/tomcat/webapps/../../dotCMS/WEB-INF/felix/load":
at org.apache.felix.framework.FakeURLStreamHandler.openConnection(FakeURLStreamHandler.java:39)
at java.net.URL.openConnection(URL.java:971)
at java.net.URLClassLoader.getPermissions(URLClassLoader.java:643)
at java.security.SecureClassLoader.getProtectionDomain(SecureClassLoader.java:206)
- locked <0x00000006a64b2dc8> (a java.util.HashMap)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at com.dotmarketing.osgi.UrlOsgiClassLoader.findClass(UrlOsgiClassLoader.java:66)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
- locked <0x00000006a64b2b50> (a com.dotmarketing.osgi.UrlOsgiClassLoader)
at com.dotmarketing.osgi.UrlOsgiClassLoader.loadClass(UrlOsgiClassLoader.java:71)
- locked <0x00000006a64b2b50> (a com.dotmarketing.osgi.UrlOsgiClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
- locked <0x00000006a652a668> (a java.lang.Object)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
- locked <0x00000006a652a5d0> (a java.lang.Object)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1431)
- locked <0x00000006a64a0118> (a java.lang.String)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1361)
at org.quartz.simpl.LoadingLoaderClassLoadHelper.loadClass(LoadingLoaderClassLoadHelper.java:59)
at org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:101)
at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:894)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1404)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$7.execute(JobStoreSupport.java:1396)
at org.quartz.impl.jdbcjobstore.JobStoreCMT.executeInLock(JobStoreCMT.java:242)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeWithoutLock(JobStoreSupport.java:3693)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1393)
at org.quartz.core.QuartzScheduler.getJobDetail(QuartzScheduler.java:1431)
at org.quartz.impl.StdScheduler.getJobDetail(StdScheduler.java:539)
at com.dotmarketing.quartz.QuartzUtils.scheduleTask(QuartzUtils.java:452)
at com.dotmarketing.osgi.GenericBundleActivator.scheduleQuartzJob(GenericBundleActivator.java:439)
at com.aquent.osgi.AquentActivator.scheduleJobs(AquentActivator.java:142)
at com.aquent.osgi.AquentActivator.start(AquentActivator.java:57)
at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:645)
at org.apache.felix.framework.Felix.activateBundle(Felix.java:2146)
at org.apache.felix.framework.Felix.startBundle(Felix.java:2064)
at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:955)
at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:1175)
at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:1153)
at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:458)
at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:263)
Found 1 deadlock.
Heap
par new generation total 471872K, used 187723K [0x00000006a0000000, 0x00000006c0000000, 0x00000006c0000000)
eden space 419456K, 32% used [0x00000006a0000000, 0x00000006a8422d48, 0x00000006b99a0000)
from space 52416K, 99% used [0x00000006bccd0000, 0x00000006bffffff8, 0x00000006c0000000)
to space 52416K, 0% used [0x00000006b99a0000, 0x00000006b99a0000, 0x00000006bccd0000)
concurrent mark-sweep generation total 62964K, used 37777K [0x00000006c0000000, 0x00000006c3d7d000, 0x00000007e0000000)
concurrent-mark-sweep perm gen total 105660K, used 63393K [0x00000007e0000000, 0x00000007e672f000, 0x0000000800000000)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment