IBM Support

MustGather: Performance, hang, or high CPU issues with Liberty on AIX

Troubleshooting


Problem

If you are experiencing performance degradation, hang, no response, hung threads, CPU starvation, high CPU utilization, or deadlocks with Liberty on AIX this MustGather assists you in collecting the critical data that is needed to troubleshoot your issue.

Diagnosing The Problem

  1. If verboseGC is not enabled, enable it per "WebSphere Liberty verbosegc from Startup Recipe"
     
  2. Download the attached script and put it in the liberty_home/bin folder.
    aixperf.sh
     
  3. At the time of the problem, run the aixperf.sh script so it generates javacores and important OS level data:
    • From the liberty_home/bin, type: ./aixperf.sh PID  (where PID is the Process ID of one or more problematic JVMs). To specify multiple PIDs, ensure they are separated by a space. The script needs to be executed as the root user. As with any script, you might need to add execute permissions prior to executing the script (chmod).
    • Skip this step if the issue you are experiencing does not involve high CPU usage. Run this command with -c option and specify the CPU threshold value. For example, to monitor PID(s) with a CPU usage of 80%. Replace the CPU_THRESHOLD_VALUE with 80 or little lower (without percentage). It will monitor the CPU usage of provided PID(s) every 30 seconds. If the CPU usage exceeds the threshold, it will automatically collect data and exit.
      ./aixperf.sh -c CPU_THRESHOLD_VALUE 1234 5678
    • The script completes in four minutes and creates a file named aixperf_RESULTS.<DATE>.<TIMESTAMP>.tar.gz. It also generates a series of javacore.*.txt files.
  4. Once the script finishes its work, collect the following files:
    • Logs(messages.log, console.log, etc)
    • aixperf_RESULTS.<DATE>.<TIMESTAMP>.tar.gz (including javacores)
    • server.env, server.xml, and jvm.options
Send the data to IBM Support: "Exchanging information with IBM Support"
Previous aixperf.sh version:

Document Location

Worldwide

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSD28V","label":"WebSphere Application Server Liberty Core"},"Component":"Hangs\/Performance Degradation","Platform":[{"code":"PF002","label":"AIX"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB67","label":"IT Automation \u0026 App Modernization"}},{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS4JBE","label":"Liberty for Java for IBM Cloud"},"Component":"Hangs\/Performance Degradation","Platform":[{"code":"PF002","label":"AIX"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB21","label":"Public Cloud Platform"}}]

Document Information

Modified date:
14 March 2024

UID

ibm11073534