Using Java core files for debugging purposes

Diagnosing The Problem

Limitations of this technote:

The information in this technote is based on:
- IBM® 32-bit SDK for Linux® on Intel® architecture Java™ 2 Technology Edition, Version 5.0
- IBM WebSphere Application Server 6.0 and later

What is a Javacore file?

A Javacore file is a formatted and pre-analyzed text file that is created by the JVM during an event or created by manual intervention.

A Javacore file contains vital information about the running JVM process, such as:

• The JVM command line
• Environment information
• Loaded libraries
• Snapshot information about all running threads, their stack traces and the monitors (locks) held by the threads
• GC (Garbage Collection) history and storage management (memory) information (since JDK 5)
• When deadlocks are detected this event will be already noted in the Java core file

Important:
A Javacore file is not the same as a core file or a system dump file. The difference is that a core file consists of the complete dump of the virtual memory for the process and can be large. Special tools are needed to parse these Javacore files.

The Javacore file follows these naming conventions:

• On LINUX platforms: javacore.YYYYMMDD.HHMMSS.PID.txt
• On AIX platforms: javacorePID.TIME.txt

Where is a Javacore file located?

For environments that use WebSphere Application Server, the Javacore file is found in the first viable location of:

1. The directory specified on the IBM_JAVACOREDIR environment variable
2. The <WAS_install-root>/profiles/<profile> directory
3. The directory specified on the TMPDIR or TEMP environment variable
4. The directory where the Java command was executed.

Obtaining a Javacore file

The Java dump mechanism is enabled by default and a Javacore file is created when one of the following occurs:

• A fatal native exception in the JVM (not a Java Exception)
• The JVM has insufficient memory to continue an operation
• A user-created signal to the JVM (such as, SIGABRT, SIGQUIT)
• Calling the JavaDump() method within Java code

However, this feature might have been disabled.

• On IBM-JDK Versions before V5 the creation of Javacore files can be disabled by exporting the environment variable with the command: DISABLE_JAVADUMP=TRUE
  
• In IBM-JDK Version 5 or later, the Xdump:java option is available which allows you to specify in a more granular way when a Java dump has to be created.

The command: java -Xdump:what lists the settings specific for any type of dump, like HEAPDUMP, SYSDUMP or JAVADUMP.

Following are the parameters for the Java dump program:

dumpFn=doJavaDump
events=gpf+user+abort
filter=
label=/opt/wpc/envs/jonas14/javacore.%Y%m%d.%H%M%S.%pid.txt
range=1..0
priority=10
request=exclusive
opts=

To have a Java dump generated on a user signal, SIGQUIT (kill -3), the events parameter needs to be set at least to events=user.

The command to enable Java dump creation upon manual user events is:
java -Xdump:java:events=user

The command to disable the Java dump is:
java -Xdump:java:none

After you have specified the prerequisites for creating the java dump, it can be created by calling the user signal SIGQUIT manually. This is done on POSIX-compliant UNIX Operating Systems by using the "kill -3 <Process ID>" which is mapped to the QUIT Signal in the header file signal.h.

Important:
Depending on the current state of the application, for example, if it is blocked, no Java dump is produced. Also user limits, file system limits, or write permissions to the respective dump directory can prevent a successful Java dump creation. More details about reasons why a Javacore file is not written to the file system can be found in the following technotes:

Crash on AIX produces no core or a truncated core
http://www-1.ibm.com/support/docview.wss?rs=180&context=SSEQTP&uid=swg21052642

MustGather: Crash on LINUX produces no core or truncated core
http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21115658

More information about using the "new" -Xdump option can be found in the following technote:

MustGather: Using the -Xdump Option http://www-01.ibm.com/support/docview.wss?uid=swg21242497

Typical reason for collecting Javacore files

For a typical hang, it is advisable to collect three dumps at a few minutes interval.
This allows you to see if anything is moving within the process (but slowly).

To get a Javacore file for one of our JVMs, such as the scheduler service or the workflow engine, it is advisable to write the -Xdump option into the respective start script.

The following example shows how you can do this for the scheduler.

In the directory $TOP/bin/go/start, the shell script start_scheduler.sh needs to be updated with the needed -Xdump option.

The preceding sample shows that first the general Java heap dump creation is turned off and then all events are captured and a filter is applied that only triggers the dump creation when the Java classes from the com.ibm.ccd.db package are used. So on every start of the workflow engine this settings are in place until they are reset.

------------snippet from start_scheduler.sh script--------------------------------------------------------------------
startWithLogs ${JAVA_RT_HOTSPOT} ${SCHEDULER_MEMORY_FLAG} ${CCD_SCHEDULER_JVM_DEBUG_OPTS} ${CCD_SCHEDULER_JVM_PROFILING_AGENT} -Dsvc_name=${SVC_NAME} -Dprofiler_opts="${CCD_SCHEDULER_PROFILER_NAME}__${CCD_SCHEDULER_PROFILER_PORT}" -Xdump:java:none -Xdump:java:events=any,filter=com/ibm/ccd/db/* com.ibm.ccd.scheduler.common.Scheduler force

------------end---------------------------------------------------------------------------------------------------------------

There is a difference for the appsvr JVM, as it is deployed as a WAR file on WebSphere Application Server.

Applying the -Xdump options to the appsvr JVM by updating the start_appsvr.sh script is not possible, as we only start the deployed application server. No additional arguments can be applied to the startServer.sh script.

------------snippet from start_appsvr.sh script----------------------------------------------------------------------

startWithLogs ${WAS_PROFILE_HOME}/bin/startServer.sh ${WAS_APPSERVERNAME}

-----------end--------------------------------------------------------------------------------------------------------------

To apply the JVM -Xdump options the WebSphere Application Server Admin Console can be used.

Go to Servers-> Application servers -> <Application Server Name> -> Java and Process Management -> Process Definition -> Java Virtual Machine -> Debug Arguments.

In the form field labeled Debug Arguments, the respective -Xdump options can be applied.



Useful references:

IBM JVM Diagnosis documentation
ibm.com/developerworks/java/jdk/diagnosis

Memory leak detection and analysis in WebSphere Application Server
ibm.com/developerworks/websphere/library/techarticles/0606_poddar/0606_poddar.html

IBM Guided Activity Assistant
The IBM Guided Activity Assistant contains much content for JVM problem determination

Redbooks "WebSphere Application Server V6 Problem Determination for Distributed Platforms", SG24-6798-00