Troubleshooting: Liberty on Docker

Resolving The Problem

Table of contents:

• Gather a thread dump of IBM Java or IBM Semeru Runtimes
• Gather a thread dump of HotSpot Java
• Gather an operating system core dump of IBM Java or IBM Semeru Runtimes

Gather a thread dump of IBM Java or IBM Semeru Runtimes

1. Find the relevant containers:

   docker ps

2. For each relevant container, replace $CONTAINER based on the output above to remote into it:

   docker exec -it $CONTAINER -- /bin/sh

3. Find the process ID of the Liberty server by listing all running processes and find the one that is Java and discover the PID after /proc/:

   ls -l /proc/[0-9]*/exe

   1. In the following example, the Java process is PID 1:

      $ ls -l /proc/[0-9]*/exe
      lrwxrwxrwx. 1 1000850000 root 0 May 15 15:48 /proc/1/exe -> /opt/ibm/java/jre/bin/java
      lrwxrwxrwx. 1 1000850000 root 0 May 15 18:40 /proc/417/exe -> /usr/bin/bash

4. Request a thread dump, replacing $PID based on the process ID found in the previous step:

   kill -3 $PID

5. Find the current working directory of the Java process, replacing $PID with the process ID:

   ls -l /proc/$PID/cwd

   1. In the following example, the current working directory of the Java process is /opt/ibm/wlp/output/defaultServer:

      $ ls -l /proc/1/cwd
      lrwxrwxrwx. 1 1000840000 root 0 May 21 18:22 /proc/1/cwd -> /opt/ibm/wlp/output/defaultServer

6. List javacore*.txt files, replacing $JAVACWD with the directory found in the previous step:

   ls -l $JAVACWD/javacore*txt

   1. If no javacore*.txt files are found, then the default javacore directory has been overridden with -Xdump JVM options or the IBM_JAVACOREDIR environment variable is used.
7. Exit the remote shell:

   exit

8. Download the file, replacing $CONTAINER with the container name and $PATH with the full path to the javacore*.txt file:

   docker cp $CONTAINER:$PATH javacore.txt

Gather a thread dump of HotSpot Java

1. Find the relevant containers:

   docker ps

2. For each relevant container, replace $CONTAINER based on the output above to remote into it:

   docker exec -it $CONTAINER -- /bin/sh

3. Find the process ID of the Liberty server by listing all running processes and find the one that is Java and discover the PID after /proc/:

   ls -l /proc/[0-9]*/exe

   1. In the following example, the Java process is PID 1:

      $ ls -l /proc/[0-9]*/exe
      lrwxrwxrwx. 1 1000850000 root 0 May 15 15:48 /proc/1/exe -> /opt/ibm/java/jre/bin/java
      lrwxrwxrwx. 1 1000850000 root 0 May 15 18:40 /proc/417/exe -> /usr/bin/bash

4. Request a thread dump, replacing $PID based on the process ID found in the previous step:

   kill -3 $PID

5. Exit the remote shell:

   exit

6. The thread dump will go to stderr, so replace $CONTAINER with the container name:

   docker logs $CONTAINER