Troubleshooting
Problem
MustGather for native memory problems with the WebSphere Application Server on AIX.
Cause
A native OutOfMemoryError condition occurs when the Java™ Virtual Machine is unable to allocate native memory.
Resolving The Problem
If you have already contacted IBM WebSphere technical support, continue to the component-specific MustGather information recommended by that analyst. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.
Out of Memory (native) specific MustGather information
Monitoring Memory Usage For All Processes
If requested by IBM support, run the ibm-psscript.sh script while recreating the issue. The ibm-psscript.sh script is located at the bottom of this page.
Run as follows:
ibm-psscript.sh [-f filename] [-i interval]
Where:
<filename> is the output filename file. The default value is ibm-psscript.log
<interval> is the time in seconds between each iteration. the default value is 1800 seconds (30 minutes).
The default values for these options are usually sufficient. IBM Support will direct you if parameters are required.
Monitoring Memory Usage For a Single Process
If requested by IBM support, please run the AIX_memory_leak.sh script while recreating the issue. The AIX_memory_leak.sh script is located at the bottom of this page.
Run as follows:
AIX_memory_leak.sh -p pid -f filename -i interval
Where the following parameters are required:
<pid> is the process id to be profiled.
<filename> is the 'base' output filename file.
<interval> is the time in seconds between each iteration.
IBM Support will direct you if specific parameters are required.
Out of Memory (native) specific MustGather information
- Enable verbosegc. For more details, see Enabling verbosegc in WebSphere Application Server.
- Make sure AIX is set up to produce full system core dumps per "Crash on AIX produces no core or a truncated core"
https://www.ibm.com/support/pages/crash-aix-produces-no-core-or-truncated-core
** Items 1 - 3 ONLY - Stop the WebSphere Application Server.
- Clear all Application Server log files.
- Restart the Application Server.
- Recreate the Out of Memory condition, or wait for the condition to reoccur. If you want to manually capture data while native memory use is high before an OOM occurs, create one Java Core and one System Dump via the Admin Console per "Collecting Java dumps and core files using the administrative console"
- Collect the following logs:
- All files located in the following directory:
profile_root/logs/server_name - javacore*.txt and core.*.dmp for the affected server
- The server.xml for the affected server
- All files located in the following directory:
- Process the core.*.dmp with jextract per
"How to process an IBM SDK core dump with jextract"
https://www.ibm.com/support/pages/how-process-ibm-sdk-core-dump-jextract-aix-linux-windows - Package and compress these logs into a single file. Follow the instructions for Submitting data to IBM Support to send in the preceding diagnostic information.
---
Monitoring Memory Usage For All Processes
If requested by IBM support, run the ibm-psscript.sh script while recreating the issue. The ibm-psscript.sh script is located at the bottom of this page.
Run as follows:
ibm-psscript.sh [-f filename] [-i interval]
Where:
<filename> is the output filename file. The default value is ibm-psscript.log
<interval> is the time in seconds between each iteration. the default value is 1800 seconds (30 minutes).
The default values for these options are usually sufficient. IBM Support will direct you if parameters are required.
Monitoring Memory Usage For a Single Process
If requested by IBM support, please run the AIX_memory_leak.sh script while recreating the issue. The AIX_memory_leak.sh script is located at the bottom of this page.
Run as follows:
AIX_memory_leak.sh -p pid -f filename -i interval
Where the following parameters are required:
<pid> is the process id to be profiled.
<filename> is the 'base' output filename file.
<interval> is the time in seconds between each iteration.
IBM Support will direct you if specific parameters are required.
Related Information
[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Out of Memory","Platform":[{"code":"PF002","label":"AIX"}],"Version":"9.0;8.5;8.0;7.0;6.1;6.0.2","Edition":"Advanced;Base;Express;Network Deployment;Single Server","Line of Business":{"code":"LOB36","label":"IBM Automation"}},{"Product":{"code":"SS7JFU","label":"WebSphere Application Server - Express"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"General","Platform":[{"code":"PF002","label":"AIX"}],"Version":"8.5;7.0;6.1;6.0.2","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Was this topic helpful?
Document Information
Modified date:
18 February 2021
UID
swg21405353