Troubleshooting
Problem
This is a problem determination document to help collect data for Liberty on podman.
Tab navigation
Environment
Tab navigation
Diagnosing The Problem
Table of contents:
- Gather logs and configuration
- Enable diagnostic trace at runtime
- Enable diagnostic trace at startup
- Change Java options at startup
- Execute a server dump
Gather logs and configuration
Perform the following steps to gather logs and configuration:
- Find the relevant containers:
podman ps
- For each relevant container, send the standard logs to a local file, replacing $CONTAINER twice based on the CONTAINER ID in the previous step:
podman logs $CONTAINER > $CONTAINER.txt
- For each relevant container, replace $CONTAINER based on the output above to remote into it:
podman exec -it $CONTAINER -- /bin/sh
-
Optional: Copy some useful process information into /tmp
cp -R --no-preserve=all --parents /proc/cpuinfo /proc/stat /proc/schedstat /proc/vmstat /proc/meminfo /proc/version /proc/pressure /proc/loadavg /proc/[0-9]*/cgroup /proc/[0-9]*/environ /proc/[0-9]*/cmdline /proc/[0-9]*/smaps /proc/[0-9]*/limits /proc/[0-9]*/stat /proc/[0-9]*/status /proc/[0-9]*/sched /proc/[0-9]*/schedstat /proc/[0-9]*/wchan /proc/[0-9]*/task/*/stat /proc/[0-9]*/task/*/wchan /proc/[0-9]*/task/*/sched /proc/[0-9]*/task/*/status /sys/fs/cgroup/cpu* /sys/fs/cgroup/memory* /sys/fs/cgroup/*/*/cpu* /sys/fs/cgroup/*/*/memory* /tmp/ 2>/dev/null
- Create a compressed file with all logs and configuration:
tar czhf /tmp/liberty_${HOSTNAME}_$(date +%Y%m%d_%H%M%S).tar.gz /logs /config /serviceability/*/${HOSTNAME} /opt/*/wlp/usr/servers/*/logs /opt/*/wlp/usr/servers/*/configDropins /opt/*/wlp/usr/servers/*/*xml /opt/*/wlp/usr/servers/*/*options /opt/*/wlp/usr/servers/*/*env /opt/*/wlp/usr/servers/*/*properties /opt/*/wlp/usr/servers/*/javacore* /opt/*/wlp/usr/servers/*/verbosegc* /opt/*/wlp/usr/servers/*/heapdump* /opt/*/wlp/usr/servers/*/core* ${LOG_DIR} ${X_LOG_DIR} ${WLP_OUTPUT_DIR} ${SERVER_WORKING_DIR} ${VARIABLE_SOURCE_DIRS} ${JAVA_HOME}/jre/lib/security/java.security /tmp/proc /tmp/sys 2>/dev/null
-
Optional: If you performed step 5
Delete only the temporary files produced in step 5:rm -rf /tmp/proc /tmp/sys
- List the compressed file and then exit the container:
cd /tmp; ls *.tar.gz; exit &>/dev/null
- Download the compressed file, replacing $CONTAINER with the container name and $FILE twice from the output of the previous step:
podman cp $CONTAINER:/tmp/$FILE $FILE
- Gather various resource state:
- For each relevant container, describe the container, replacing $CONTAINER twice with the container name:
podman inspect $CONTAINER > $CONTAINER_inspect.json
- For each relevant container, describe the container, replacing $CONTAINER twice with the container name:
- Upload:
-
Container standard logs (step 3)
-
Liberty logs and configuration (step 9)
-
Resource state text files (step 10)
-
Enable diagnostic trace at runtime
Perform the following steps to enable diagnostic trace at runtime:
- If runtime configuration updates are not enabled, then you must enable diagnostic trace at startup.
- If runtime configuration updates are enabled (as they are by default):
- Find the relevant containers:
podman ps
- For each of the relevant containers, replace $CONTAINER based on the output above and replace $TRACE with an IBM support-requested trace specification or your desired trace specification:
podman exec -it $CONTAINER -- /bin/sh -c "echo '<?xml version=\"1.0\" encoding=\"UTF-8\"?><server><logging traceSpecification=\"*=info:$TRACE\" maxFileSize=\"100\" maxFiles=\"10\" /></server>' > /config/configDropins/overrides/trace.xml"
- Reproduce the problem
- Disable the diagnostic trace:
podman exec -it $CONTAINER -- /bin/sh -c "rm /config/configDropins/overrides/trace.xml"
- Gather and upload all logs
- Find the relevant containers:
Enable diagnostic trace at startup
Perform the following steps to enable diagnostic trace at startup:
- Create a local file named tracefromstartup.xml with the following contents and replace $TRACE with an IBM support-requested trace specification or your desired trace specification:
<?xml version="1.0" encoding="UTF-8"?> <server> <logging traceSpecification="*=info:$TRACE" maxFileSize="100" maxFiles="10" /> </server>
- Rebuild the image with this file in the configDropins overrides directory such as /config/configDropins/overrides/
- Start the new container
- Reproduce the problem
- Gather and upload all logs
Change Java options at startup
The JVM_ARGS Liberty environment variable may be configured at container startup to change JVM options. Perform the following steps to add Java options at startup:
- First, check if this environment variable is already set:
- Find the relevant containers:
podman ps
- For one of the relevant containers, replace $CONTAINER based on the output above to search its environment variables:
podman exec -it $CONTAINER -- /bin/sh -c "cat /proc/[0-9]*/environ | tr '\0' '\n' | grep JVM_ARGS="
- Find the relevant containers:
- Append your desired arguments to the value found in the above steps (if any).
- Restart the container with the updated JVM_ARGS environment variable specified (steps will differ depending on how you are starting the container).
- Reproduce the problem
- Gather and upload all logs
Execute a server dump
Perform the following steps to execute and gather a Liberty server dump.
Warning: These commands will start a new process which will consume some memory (likely in the range of dozens of MB). If your container has a memory limit and it is near its limit, this may cause the container to crash. Alternatively, you may gather logs and configuration manually.
- Find the relevant containers:
podman ps
- For each of the relevant containers, replace $CONTAINER based on the output above:
podman exec -it $CONTAINER -- /bin/sh -c "/opt/*/wlp/bin/server dump --include=thread"
- The output of the command should state where the server dump is written; for example:
Dumping server defaultServer. Server defaultServer dump complete in /opt/ibm/wlp/output/defaultServer/defaultServer.dump-24.05.15_17.11.35.zip.
- Download the file, replacing $CONTAINER with the container name and $PATH with the full path from the output of the previous step (without the period at the end):
podman cp $CONTAINER:$PATH serverdump.zip
Notes & Tips
- If you are on macOS or Linux (or Cygwin on Windows), then you may use variables to simplify the above commands. For example, various commands use $CONTAINER for the target pod, so you may first execute a statement to set the CONTAINER variable and then future references of $CONTAINER in this terminal window will be replaced with what you specified. In the following example, CONTAINER is set to 60578bd402be and therefore the final command of podman logs $CONTAINER will use the specified value and thus you can just copy/paste commands from the instructions above without needing to modify them.
$ podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 60578bd402be websphere-liberty:latest /bin/sh 10 minutes ago Up 10 minutes test $ CONTAINER=60578bd402be $ podman logs $CONTAINER
Copied!
Copie
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB67","label":"IT Automation \u0026 App Modernization"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"ARM Category":[{"code":"a8mKe000000GmbMIAS","label":"IBM WebSphere Liberty-All Platforms-\u003EContainers"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
22 May 2024
UID
ibm17152485