Troubleshooting: Liberty on OpenShift with the Liberty Operator

Resolving The Problem

Table of contents:

• Gather a thread dump of IBM Java or IBM Semeru Runtimes
• Gather a thread dump of HotSpot Java
• Gather an operating system core dump of IBM Java or IBM Semeru Runtimes

Gather a thread dump of IBM Java or IBM Semeru Runtimes

1. Ensure you are in the right namespace of the target application pods by replacing $NAME with your project name:

   oc project $NAME

2. Find the relevant pods:

   oc get pods

3. For each relevant pod, replace $POD based on the output above to remote into it (if there are multiple containers, use -c $CONTAINER after $POD to specify the Liberty container):

   oc exec -it $POD -- /bin/sh

4. Find the process ID of the Liberty server by listing all running processes and find the one that is Java and discover the PID after /proc/:

   ls -l /proc/[0-9]*/exe

   1. In the following example, the Java process is PID 1:

      $ ls -l /proc/[0-9]*/exe
      lrwxrwxrwx. 1 1000850000 root 0 May 15 15:48 /proc/1/exe -> /opt/ibm/java/jre/bin/java
      lrwxrwxrwx. 1 1000850000 root 0 May 15 18:40 /proc/417/exe -> /usr/bin/bash

5. Request a thread dump, replacing $PID based on the process ID found in the previous step:

   kill -3 $PID

6. Find the current working directory of the Java process, replacing $PID with the process ID:

   ls -l /proc/$PID/cwd

   1. In the following example, the current working directory of the Java process is /opt/ibm/wlp/output/defaultServer:

      $ ls -l /proc/1/cwd
      lrwxrwxrwx. 1 1000840000 root 0 May 21 18:22 /proc/1/cwd -> /opt/ibm/wlp/output/defaultServer

7. List javacore*.txt files, replacing $JAVACWD with the directory found in the previous step:

   ls -l $JAVACWD/javacore*txt

   1. If no javacore*.txt files are found, then the default javacore directory has been overridden with -Xdump JVM options or the IBM_JAVACOREDIR environment variable is used.
8. Exit the remote shell:

   exit

9. Download the file, replacing $POD with the pod name and $PATH with the full path to the javacore*.txt file:

   oc cp $POD:$PATH javacore.txt --retries=999

Gather a thread dump of HotSpot Java

1. Ensure you are in the right namespace of the target application pods by replacing $NAME with your project name:

   oc project $NAME

2. Find the relevant pods:

   oc get pods

3. For each relevant pod, replace $POD based on the output above to remote into it (if there are multiple containers, use -c $CONTAINER after $POD to specify the Liberty container):

   oc exec -it $POD -- /bin/sh

4. Find the process ID of the Liberty server by listing all running processes and find the one that is Java and discover the PID after /proc/:

   ls -l /proc/[0-9]*/exe

   1. In the following example, the Java process is PID 1:

      $ ls -l /proc/[0-9]*/exe
      lrwxrwxrwx. 1 1000850000 root 0 May 15 15:48 /proc/1/exe -> /opt/ibm/java/jre/bin/java
      lrwxrwxrwx. 1 1000850000 root 0 May 15 18:40 /proc/417/exe -> /usr/bin/bash

5. Request a thread dump, replacing $PID based on the process ID found in the previous step:

   kill -3 $PID

6. Exit the remote shell:

   exit

7. The thread dump will go to stderr, so replace $POD with the pod name:

   oc logs --all-containers=true $POD