« Apache httpd 2.3.11-beta Released | Main | Apache Tomcat 7.0.10 Released »

03/08/2011

Tomcat Crashes Soon After Successful Startup - Common Causes*

( * plus the one that burned us...)

Perhaps you are seeing this, too.  You start tomcat up, tailing the logs for that Tomcat Started message.  All looks good.  Monitors are happy.  Smoke tests are working fine.  Log off the server and the next thing you know, your NOC is calling you because tomcat is down.  You log back onto the server, run a "ps -ef | grep tomcat" and sure enough, no tomcat process.

The tomcat logs don't seem to show anything.  There are no heap dumps getting written.  No stack traces, no exceptions.  Hoping its not going to be one of those unpleasant mysteries that take hours to fix, you start tomcat successfully again to restore service.  Everything is working fine again.  Hoping that shrugging your shoulders and commenting to the management on the complexities of tomcat classpath loading and bootstrapping, you end the call while silently muttering to yourself about these java "crapplications".  Seconds to minutes later, tomcat is down again...seemingly crashing as soon as you ended your ssh session!

Below are the three most common causes of tomcat failures after successful startup with no exceptions being logged:

  1. You Suspect:  There is something wrong with the logging configuration.  Maybe you've got logging disabled or are writing to a directory you don't have permissions to write to.  Maybe the log level is wrong.
  2. You Hope:  Someone on the development team has written, somewhere in the application, code that calls System.exit(0).  This would cause tomcat to terminate when called.  This has a few benefits to the systems administrator.  First, there is the timeless "wtf" that you can heave against the developers.  "WTF! Who would intentionally call System.exit(0) in production code!?!?"  Secondly, unless you like decompiling java .class files, you can get development to spend a lot of time pouring through their source to find the offending piece of code as this problem isn't necessarily yours to fix.
  3. You Fear:  The JVM is crashing.  The JVM should be writing crash logs but if you think the jvm is crashing and you're not seeing crash logs getting generated, add the -XX:HeapDumpPath option to your JVM startup parameters, including a path to a directory tomcat can write to.  My Unofficial First Law of JEE System Administration dictates however that this will never be the cause unless you had dinner plans.

On to the reason that caused our failure, since, in our case, it wasn't any of the above.

More careful perusal of all the app server logs unearthed the following error:

X connection to localhost:10.0 broken (explicit kill or server shutdown).

Interestingly, in our case, one of the sysadmins starting tomcat up was connecting to the Linux host as the user that tomcat was running as with an ssh cllient that had X11 Port Forwarding enabled.  Once connected, the local DISPLAY environment variable was changed to point to his host.  Upon ending his session, the DISPLAY session that one of the third party graphing libraries apparently used if set, cleared, causing tomcat to die.  

A simple workaround was to add "unset DISPLAY" to tomcat's setenv.sh file, since, again in our case, the third party library worked just fine if loaded without DISPLAY being set.

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a01156fbc6fe6970c0147e306988f970b

Listed below are links to weblogs that reference Tomcat Crashes Soon After Successful Startup - Common Causes*:

Comments