MustGather: Liberty on Docker

Diagnosing The Problem

Table of contents:

• Gather logs and configuration
• Enable diagnostic trace at runtime
• Enable diagnostic trace at startup
• Change Java options at startup
• Execute a server dump

Gather logs and configuration

1. Find the relevant containers:

   docker ps

2. For each relevant container, send the standard logs to a local file, replacing $CONTAINER twice based on the CONTAINER ID in the previous step:

   docker logs $CONTAINER > $CONTAINER.txt

3. For each relevant container, replace $CONTAINER based on the output above to remote into it:

   docker exec -it $CONTAINER -- /bin/sh

4. Optional: Copy some useful process information into /tmp

   cp -R --no-preserve=all --parents /proc/cpuinfo /proc/stat /proc/schedstat /proc/vmstat /proc/meminfo /proc/version /proc/pressure /proc/loadavg /proc/[0-9]*/cgroup /proc/[0-9]*/environ /proc/[0-9]*/cmdline /proc/[0-9]*/smaps /proc/[0-9]*/limits /proc/[0-9]*/stat /proc/[0-9]*/status /proc/[0-9]*/sched /proc/[0-9]*/schedstat /proc/[0-9]*/wchan /proc/[0-9]*/task/*/stat /proc/[0-9]*/task/*/wchan /proc/[0-9]*/task/*/sched /proc/[0-9]*/task/*/status /sys/fs/cgroup/cpu* /sys/fs/cgroup/memory* /sys/fs/cgroup/*/*/cpu* /sys/fs/cgroup/*/*/memory* /tmp/ 2>/dev/null

5. Create a compressed file with all logs and configuration:

   tar czhf /tmp/liberty_${HOSTNAME}_$(date +%Y%m%d_%H%M%S).tar.gz /logs /config /serviceability/*/${HOSTNAME} /opt/*/wlp/usr/servers/*/logs /opt/*/wlp/usr/servers/*/configDropins /opt/*/wlp/usr/servers/*/*xml /opt/*/wlp/usr/servers/*/*options /opt/*/wlp/usr/servers/*/*env /opt/*/wlp/usr/servers/*/*properties /opt/*/wlp/usr/servers/*/javacore* /opt/*/wlp/usr/servers/*/verbosegc* /opt/*/wlp/usr/servers/*/heapdump* /opt/*/wlp/usr/servers/*/core* ${LOG_DIR} ${X_LOG_DIR} ${WLP_OUTPUT_DIR} ${SERVER_WORKING_DIR} ${VARIABLE_SOURCE_DIRS} ${JAVA_HOME}/jre/lib/security/java.security /tmp/proc /tmp/sys 2>/dev/null

6. Optional: If you performed step 5

   Delete only the temporary files produced in step 5:

   rm -rf /tmp/proc /tmp/sys

7. List the compressed file and then exit the container:

   cd /tmp; ls *.tar.gz; exit &>/dev/null

8. Download the compressed file, replacing $CONTAINER with the container name and $FILE twice from the output of the previous step:

   docker cp $CONTAINER:/tmp/$FILE $FILE

9. Gather various resource state:
   1. For each relevant container, describe the container, replacing $CONTAINER twice with the container name:

      docker inspect $CONTAINER > $CONTAINER_inspect.json

10. Upload:
    1. Container standard logs (step 3)
    2. Liberty logs and configuration (step 9)
    3. Resource state text files (step 10)

Enable diagnostic trace at runtime

1. If runtime configuration updates are not enabled, then you must enable diagnostic trace at startup.
2. If runtime configuration updates are enabled (as they are by default):
   1. Find the relevant containers:

      docker ps

   2. For each of the relevant containers, replace $CONTAINER based on the output above and replace $TRACE with an IBM support-requested trace specification or your desired trace specification:

      docker exec -it $CONTAINER -- /bin/sh -c "echo '<?xml version=\"1.0\" encoding=\"UTF-8\"?><server><logging traceSpecification=\"*=info:$TRACE\" maxFileSize=\"100\" maxFiles=\"10\" /></server>' > /config/configDropins/overrides/trace.xml"

   3. Reproduce the problem
   4. Disable the diagnostic trace:

      docker exec -it $CONTAINER -- /bin/sh -c "rm /config/configDropins/overrides/trace.xml"

   5. Gather and upload all logs

Enable diagnostic trace at startup

1. Create a local file named tracefromstartup.xml with the following contents and replace $TRACE with an IBM support-requested trace specification or your desired trace specification:

   <?xml version="1.0" encoding="UTF-8"?>
   <server>
     <logging traceSpecification="*=info:$TRACE" maxFileSize="100" maxFiles="10" />
   </server>

2. Rebuild the image with this file in the configDropins overrides directory such as /config/configDropins/overrides/
3. Start the new container
4. Reproduce the problem
5. Gather and upload all logs

Execute a server dump

1. Find the relevant containers:

   docker ps

2. For each of the relevant containers, replace $CONTAINER based on the output above:

   docker exec -it $CONTAINER -- /bin/sh -c "/opt/*/wlp/bin/server dump --include=thread"

3. The output of the command should state where the server dump is written; for example:

   Dumping server defaultServer.
   Server defaultServer dump complete in /opt/ibm/wlp/output/defaultServer/defaultServer.dump-24.05.15_17.11.35.zip.

4. Download the file, replacing $CONTAINER with the container name and $PATH with the full path from the output of the previous step (without the period at the end):

   docker cp $CONTAINER:$PATH serverdump.zip