IBM Support

MustGather: Crash on AIX (WebSphere Application Server)

Troubleshooting


Problem

Your application server terminated without warning or a system core is generated but you are not sure what data to collect. The following MustGather will assist you in collecting the critical data to troubleshoot issues with an IBM® WebSphere® Application Server crash on the AIX® platform.

Symptom

System core generated, JVM process disappears or terminates.

Resolving The Problem

Your application server terminated without warning or a system core is generated but you are not sure what data to collect. The following MustGather will assist you in collecting the critical data to troubleshoot issues with an IBM® WebSphere® Application Server crash on the AIX® platform.

Collecting data manually


DATA TO COLLECT

WebSphere Application Server
Location of dumps are usually in the profile root (/opt/IBM/WebSphere/AppServer/profiles/PROFILE_NAME/)
  • javacore*.txt, heapdump*.phd, snap*.trc files (if produced).
  • Output from aix_osinfo.sh (see below)
    • This will include the errpt -a output
  • jextract output (from processing the IBM SDK dump file)
    • Provide the core dump file if possible (especially if the processing fails)
  • AppServer logs, including...
    • SystemOut.log
    • SystemErr.log
    • native_stdout.log
    • native_stderr.log (verbose GC is logged here by default)
  • Configuration data: server.xml
    • NOTE: You may also run the collector tool to capture all configuration (and logs) for a particular profile into a single JAR file. The collector.sh script is located in the bin directory in the profile’s root.

WebSphere Liberty
Location of dumps are usually in the server root (/opt/IBM/wlp/user/servers/SERVER_NAME)
 
  • javacore*.txt, heapdump*.phd, Snap*.trc files (if produced).
  • Output from aix_osinfo.sh (see below)
    • This will include the errpt -a output
  • jextract output (from processing the IBM SDK dump file)
    • Provide the core dump file if possible (especially if the processing fails)
  • AppServer logs, including...
    • messages.log
    • console.log
    • SystemOut.log (possible to have depending on configuration)
       
  • Configuration data, including:
    • server.xml
    • jvm.options
    • server.env

Send the results to IBM support.

PROCESSING THE CORE DUMP
How to process an IBM SDK core dump with jextract
http://www.IBM.com/support/docview.wss?uid=swg21577379

Make sure to attempt processing the core dump with the command jextract. You will need to locate which IBM SDK was used to run the application server (the javacore contains this information, or simply restart your application server and locate the process with the ps command), and run the associated jextract executable for that SDK install.

Example with a 8192 MB heap, and a JVM that was running with compressed references from the "/java_home/jre/bin" directory

./jextract -J-Xcompressedrefs -J-Xmx8192m /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/core.DATE.TIME.PID.COUNT.dmp

The result will be a zip file that contains the core dump, some XMLs, and any required libraries for processing.


NO CORE DUMP GENERATED
Crash on AIX produces no core or a truncated core
http://www.IBM.com/support/docview.wss?rs=180&uid=swg21052642

If you find no core dump, and none of the logs (such as native_stderr.log) indicate one was generated, check first if the nodeagent had terminated the JVM and restarted it, as a dump may not generate if that occurs.

Another reason for this, especially if a crash was recorded in the log, is that the ulimits for the core and file are not set to unlimited.

If there were no dumps and no signs of a crash, but the process did terminate, check this document for further debugging.

WebSphere Application Server process exits without leaving a footprint in log files or no core dumps
http://www.IBM.com/support/docview.wss?uid=swg21144595


AIX OS INFO SCRIPT
Run aix_osinfo.sh (without arguments, recommended as root) and collect the resulting aix_osinfo_RESULTS.tar.gz and aix_osinfo.log files. This script collects general information about the AIX operating system.

./aix_osinfo.sh

         

Commands executed in this script above include:
  • /usr/sbin/emgr -lv3 (if the root user)
  • oslevel -s
  • lslpp -la
  • instfix -ia
  • prtconf
  • lparstat -i
  • lsattr -El sys0
  • df -k
  • errpt -a
  • ps -ef


ADDITIONAL SCRIPTS (optional)
  • The rest of the scripts below are designed for WebSphere Application Server 6.0.2 and below, and are only needed if IBM support specifically asks for this data. These scripts require DBX installed on the system.

    ./dbxtrace_aix.sh [JVM_PATH] [CORE_PATH] > dbxtrace.out
    ./getlib.sh [JVM_PATH] [CORE_PATH]

    Notes:
    • Set the [JVM_PATH] to the following directory:
      install_root/java/jre/bin/java
    The getlib.sh script generates getlib_archive.tar.Z in the script's current directory.






    •  

 


For a listing of all technotes, downloads, and educational materials specific to the Crash component, search the WebSphere Application Server support site.

RELATED INFORMATION
How to process an IBM SDK core dump with jextract
Crash on AIX produces no core or truncated 
Checking if the dbx utility is installed on AIX
Submitting information to IBM support
Steps to getting support for WebSphere Application Server
MustGather: Readme first for WebSphere Application Server
Troubleshooting guide for WebSphere Application Server


Exchanging data with IBM Support

To diagnose or identify a problem, it is sometimes necessary to provide Technical Support with data and information from your system. In addition, Technical Support might also need to provide you with tools or utilities to be used in problem determination. You can submit files using one of following methods to help speed problem diagnosis:

  • IBM Support Assistant (ISA)
  • Service Request (SR)
  • E-mail
  • FTP to the Enhanced Customer Data Repository (ECuRep)

Instructions for exchanging information with IBM Support

Read first and related MustGathers

MustGather: Read first for WebSphere Application Server

Hang / High CPU / Performance Problem Out of Memory Problem

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Crash","Platform":[{"code":"PF002","label":"AIX"}],"Version":"9.0.0.0;8.5.5;8.5;8.0;7.0","Edition":"Base;Express;Liberty;Network Deployment","Line of Business":{"code":"LOB36","label":"IBM Automation"}},{"Product":{"code":"SSNVBF","label":"Runtimes for Java Technology"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Java SDK","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}},{"Product":{"code":"SS7JFU","label":"WebSphere Application Server - Express"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":" ","Platform":[{"code":"PF002","label":"AIX"}],"Version":"8.5.5;8.0;7.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
04 May 2020

UID

swg21055387

Page Feedback