IBM Support

Diagnosing Native OutOfMemory in Windows Using Debug Diagnostic Tool (DebugDiag)

Troubleshooting


Problem

Native OutOfMemory problems become easier to diagnose using the Microsoft Debug Diagnostic Tool, (DebugDiag).

Symptom

Some indications that you have a native memory problem are as follows. If you have any uncertainty about these points, you should contact IBM Support:

  • failure to create/fork new operating system threads (messages in SystemOut.log)
  • JVMDBG001 (malloc) errors in native_stderr.log

Environment

Windows

Diagnosing The Problem

    Download Debug Diagnostic Tool

    Download:
    https://www.microsoft.com/en-us/download/details.aspx?id=26798
    DebugDiagx64.msi (for 64bit OS)
    DebugDiagx86.msi (for 32bit OS)

    Install:
    • Install and take the defaults

    Set-up DebugDiag "Analysis" :
    1. Start DebugDiag Analysis: Start->All Programs->Debug Diagnostics Tool ->DebugDiag Analysis
    2. Click Settings -> Preferences
      select the option: Save reports in relative subfolders when possible.
    3. To configure the symbol path that DebugDiag uses:
      Symbol Search Path for Analysis:

      srv*c:\symcache*http://msdl.microsoft.com/download/symbols
      and
      <java-home>\jre\bin

      For more information about symbol tables see:
      Use the Microsoft Symbol Server to obtain debug symbol files
    http://support.microsoft.com/kb/311503
      Note: Replace <java-home> with the actual path where the JRE is installed.This is required to give the path for all the JRE PDB files so that DebugDiag can pick up the necessary JRE symbols.

    Set-up DebugDiag "Collection":
    1. Start DebugDiag Collection: Start->All Programs->Debug Diagnostics Tool->DebugDiag Collection
    2. Click "Cancel" on the "Select Rule Type" Dialog Box.
    3. To configure the symbol path that Debug Diagnostic Tool uses for Debugging:
      Go to Tools->Options and Settings..., which brings up a dialog box.
      Enter the following text under Folders and Search Paths->Symbol Search Path for Debugging:

      <java-home>\jre\bin

      For more information about symbol tables see:
      Use the Microsoft Symbol Server to obtain debug symbol files
      http://support.microsoft.com/kb/311503

      Note: Replace <java-home> with the actual path where the JRE is installed.This is required to give the path for all the JRE PDB files so that DebugDiag can pick up the necessary JRE symbols.
    5. Enter the path where you want to save the userdumps under - Manual Userdump Save Folder.
    6. Go to the Preferences Tab
      • select the option: Include source and line information in analysis report.
      • select the option: Use fixed initial folder for data files.
      • select the option: Record call stacks immediately when monitoring for leaks.
      • Click OK to save and close the Dialog box.

    Use DebugDiag for tracking memory leaks:
    1. Start DebugDiag Collection: Start->All Programs->Debug Diagnostics Tool ->DebugDiag Collection
    2. In the main window, activate the Processes tab. Locate the process (process id) that is suspected to be leaking memory, Right click the process and choose "Monitor For Leaks".
    3. After choosing this option, let the process run and leak memory. It is suggested the process runs for a minimum of 15 minutes with the tracking DLL in place in order to get valid results.
    4. Let the process be monitored for the entire growth period (say some X minutes). After every X/4 minutes right click the process and choose "Create Full Userdump". This will generate a userdump.

    Note - Dumps should be collected as follows:
    1. At least 15 min after the start of the application (1 dump).
    2. During initial run of the application (1 dump).
    3. Gradual increase in the memory (2 dumps at certain intervals).
    4. On hitting OOM or when memory has grown very large (1 dump).
    *The idea is to capture dumps so that when comparing dumps side by side, there is a very noticeable difference in memory size.

    Review dumps(optional for self analysis):
    1. Start DebugDiag Analysis: Start->All Programs->Debug Diagnostics Tool ->DebugDiag Analysis
    2. Click the "Add Data Files" button.
    3. Locate dumps (default: <Debug_Diag_Install_Dir>\Logs\Misc or where you set: "Manual Userdump Save Folder" in preferences).
    4. Select the dumps from the same PID that you are tracking memory (all PIDs should match).
    5. Check the box for Memory Analysis under Memory Pressure Analyzers
    6. Click "Start Analysis".
    7. This will produce a html page (*.mht) in the <USER>\Documents\DebugDiag\Reports folder with leak "suspects" at the top among other memory statistics.


Resolving The Problem

Check the report at <USER>\Documents\DebugDiag\Reports for the reported leak suspect.

For further diagnosis, submit the following to IBM:

  • DebugDiag report (*.mht)
  • The matching system core dumps (jextracted)
    IBM How to process an IBM SDK core dump with Jextract
    http://www-01.ibm.com/support/docview.wss?uid=swg21577379
  • SystemOut.log, systemErr.log, native_stdout.log, native_stderr.log, matching javacores and matching heapdumps (if created).

"Exchanging information with IBM Technical Support for problem determination"
http://www.ibm.com/support/docview.wss?uid=swg21153852

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Out of Memory","Platform":[{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5;8.0;7.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
15 June 2018

UID

swg21514752

Page Feedback