Troubleshooting
Problem
This technote details how to set up and use libumem to track native memory usage for IBM WebSphere Application Server processes on Solaris.
Symptom
Suspicious or excessive native memory usage.
Environment
Solaris 9 and later
Resolving The Problem
- Set the libumem environment variables as WebSphere custom properties
"Setting environment entries in WebSphere Application Server"
http://www.ibm.com/support/docview.wss?rs=180&uid=swg21254153
Set libumem library to the LD_PRELOAD environment variable:
Name: LD_PRELOAD
Value: libumem.so.1
Set the libumem debugging environment variable:
Name: UMEM_DEBUG
Value: default
- Collect multiple system cores using gcore
The following command will generate a system core dump...
gcore [JVM PID]
How often system cores are collected depends on how long it takes for the process to critically fail due to native memory exhaustion. An appropriate interval should be used with the understanding that better results come from longer running processes. Ideally the last (or only) system core is generated just before the process fails, or from the failure itself.
IBM support can advise on how often a core dump should be generated.
- For Each Generated System Core Dump
- Run the attached script as follows...
./probenativeheap.sh [ core ]
This will create five files for the core file
probenativeheap.[core file].findleaks.out
probenativeheap.[core file].umausers.out
probenativeheap.[core file].umastat.out
probenativeheap.[core file].umalog.out
probenativeheap.[core file].umausers_oversize.out
- Extract the Java heap dump from each system core using jmap. The output of this command will create a memory dump named "heap.bin".
32bit*
<JavaHome>/bin/jmap -heap:format=b <JavaHome>/bin/java [ core ]
64bit*
<JavaHome>/bin/jmap -J-d64 -heap:format=b <JavaHome>/bin/java [ core ]
<JavaHome> = JVM running under WebSphere Application Server
*32bit or 64bit should be selected based on the installed JVM version
- Combine the heap.bin and *.out files into a single compressed archive, named appropriately Example.
[PMR_Number].dump1.<timestamp>.tar.gz
-Each system core dump should be processed as they are created since there is no control of the name "heap.bin". This would make overwriting heap dumps possible.
"Exchanging information with IBM Support"
http://www.ibm.com/support/docview.wss?uid=swg21153852
[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Out of Memory","Platform":[{"code":"PF027","label":"Solaris"}],"Version":"8.5.5;8.5;8.0;7.0;6.1","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Was this topic helpful?
Document Information
Modified date:
15 June 2018
UID
swg21504345