Troubleshooting
Problem
This is the data gathering procedure for native Out Of Memory (OOM) errors for IBM WebSphere Application Server on Windows. If you know, or strongly suspect, that you have a native memory problem, or have been informed by an IBM support representative that you have a native memory problem, these are the steps to collecting the data needed for proper diagnosis of a root cause.
Symptom
Some indications that you have a native memory problem are as follows. If you have any uncertainty about these points you should contact an IBM support representative first.
- Failure to create/fork new operating system threads (messages in SystemOut.log)
- JVMDBG001 (malloc) errors in native_stderr.log
Resolving The Problem
Here are some data collections that can be used from a Windows environment (without engineered debug .jar files or executable files). They are as follows and should be collected in this order or as directed by an IBM Support representative:
Ideally all of this data can be collected at once, however it is understood that circumstances will affect how much of this data can be gathered and when. In any event, it is important to have an understanding of the problem so that data can be collected at the right time(s).
When applicable, please compress then upload any/all of the following:
Using UMDH to track a native leak
http://support.microsoft.com/kb/268343
Exchanging Information with IBM Technical Support for Problem Determination
- General OutOfMemory MustGather data
- processed user.dmp
- Sometimes after several malloc errors occur, the process can crash. The user.dmp is a file that is often generated from this crash and can provide a glimpse at what is filling up native space. It might be necessary to create this file manually under some circumstances. As this is a crash, it needs to be processed with jextract similar to any other crash on Windows.
See MustGather: Crash on Windows
- Sometimes after several malloc errors occur, the process can crash. The user.dmp is a file that is often generated from this crash and can provide a glimpse at what is filling up native space. It might be necessary to create this file manually under some circumstances. As this is a crash, it needs to be processed with jextract similar to any other crash on Windows.
- UMDH (when requested)
- This is a Microsoft application that will allow us to see growth over time of native libraries in use. This is more useful if a native leak has already been discovered or at least highly suspected. It involves taking 'snapshots' of the heap at given intervals and doing a comparative analysis of these snapshots.
DETAILS
- This is a Microsoft application that will allow us to see growth over time of native libraries in use. This is more useful if a native leak has already been discovered or at least highly suspected. It involves taking 'snapshots' of the heap at given intervals and doing a comparative analysis of these snapshots.
Ideally all of this data can be collected at once, however it is understood that circumstances will affect how much of this data can be gathered and when. In any event, it is important to have an understanding of the problem so that data can be collected at the right time(s).
When applicable, please compress then upload any/all of the following:
- Heapdumps/Javacores
- verboseGC (native_stderr.log)
- Server Logs (SystemOut.log, SystemErr.log, etc.)
- server.xml
- processed user.dmp
- snapshots
- comparisons
Using UMDH to track a native leak
- Ensure the verbose garbage collection trace is enabled:
Enabling verbose garbage collection (verbosegc) in WebSphere Application Server
- Download and install the Microsoft debugging tools:
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools
Look for the “Standalone Debugging Tools for Windows (WinDbg)” paragraph. Select “Get the Debugging Tools”. During the install, only select “Debugging Tools for Windows”. Deselect all other components. - From a command line, or Run box, run the following against the installed IBM WebSphere SDK:
gflags /i <path to AppServer JDK>/java.exe +ust
- Set the environment variable _NT_SYMBOL_PATH to the following:
srv*<local_dir>*http://msdl.microsoft.com/download/symbols
Where local_dir is a directory that can be set aside for storing the symbol table.
- Take a series of snapshots over the period of the process's life. If there is a clear spike in memory usage, then collect snapshots before and after the spike. Otherwise be sure to chose an appropriate interval for the environment. Use the following command:
umdh -p:<pid> -f:<your dump filename>
Ensure the filename is appropriate to identify the sequence the in which snapshots were taken.
- Once the snapshots are collected, they need to be compared with the following command:
umdh -v <dump1> <dump2> -f:<compare file name>
Ensure the filename is appropriate to identify which snapshots were compared.
http://support.microsoft.com/kb/268343
Exchanging Information with IBM Technical Support for Problem Determination
Related Information
[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Component":"Out of Memory","Platform":[{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5.5;8.5;8.0;7.0;6.1;6.0;5.1.1","Edition":"","Line of Business":{"code":"LOB67","label":"IT Automation \u0026 App Modernization"}}]
Was this topic helpful?
Document Information
Modified date:
09 July 2024
UID
swg21313578