IBM Support

MustGather: WAS JVM Out of Memory/Heap Leak Issues on IBM i OS

Troubleshooting


Problem

Collecting data for Out of Memory/Heap Leak errors with IBM WebSphere Application Server. Gathering this MustGather information before calling IBM Support will help familiarize you with the troubleshooting process and save you time.

Cause

An Out of Memory error is generally indicated when a java.lang.OutofMemoryError exception is thrown. Two conditions can cause a java.lang.OutOfMemoryError exception to occur:

  1. The Java Virtual Machine (JVM) might run out of contiguous Java heap space to allocate a Java object.
  2. The JVM might not be able to allocate native memory.

Resolving The Problem

If you have already contacted support, continue through this document as requested by your support representative. Otherwise, refer to MustGather: Read first for WebSphere Application Server.


Out of Memory/Heap Leak specific MustGather information:
If you suspect a memory leak in the Java™ heap, the first step is to eliminate any misconfiguration or tuning as a potential cause:

  1. Increase the Maximum Java Heap Value (-Xmx) and test again.

    For information on setting these properties, see the Generic JVM Arguments section of the following document:
  2. Follow instructions for Enabling verbosegc in WebSphere Application Server.

For all Java Virtual Machine (JVM) memory issues, collect the following information. There is a list of links provided at the bottom of this document that routes you to the how-to documents for steps that require additional instructions.

Collect the following information when the application server process (job) is experiencing the problem.

Note: If you are running the Network Deployment (ND) edition of WebSphere Application Server, complete the following steps for the Deployment Manager AND the Federated Node experiencing the problem.
  1. Run the following commands to collect system information to be sent to IBM:
    WRKPTFGRP (use F6 to print or send a screen shot showing the installed status of PTFs)

    DSPPTF OUTPUT(*PRINT)

    WRKSYSSTS OUTPUT(*PRINT)

    WRKSBS OUTPUT(*PRINT)

    WRKSHRPOOL OUTPUT(*PRINT)

    WRKACTJOB OUTPUT(*PRINT)
  2. Gather 3 or more Java and Heap Dumps about 5 - 10 minutes apart and then 1 GcCycles Dump for the JVM.

    Note: Gather the Java and Heap Dumps in pairs and gather the GcCycles Dump after the Thread and Heap Dumps. Appendix A: How to collect a JVM thread and heap dump for a server experiencing an Out of Memory/Heap Leak issue.

  3. Run the collector tool for your V9.0, V8.5.5, V8.5, V8.0, V7.0, or V6.1 installation (search on collector.)

  4. Follow the instructions to send diagnostic information to IBM support. Appendix B: How to send diagnostic information to IBM support.

For a listing of all technotes, downloads, and educational materials specific to a component, search the WebSphere Application Server support site.


Appendix A: How to collect a JVM thread and heap dump for a server experiencing an Out of Memory/Heap Leak issue

Multiple JVMs are available on IBM i 5.4 and higher. The original JVM is known as the "Classic JVM." The new JVM is called "IBM Technology for Java." The approach to gather data is different for each JVM, so you will need to figure out which JVM is in use before you proceed. The easiest way to find this is by displaying the job log for the job experiencing the problem.

For example, if the server is called 'server1'; you would run the following from the CL command line:

WRKJOB SERVER1 <enter>
Choose option 1 to select the active job.
Choose option 10 to display the job log.
Press F10 to see all messages
The joblog will show:

Java Virtual Machine is IBM Technology for Java.  PID(xxx)
OR

Java Virtual Machine is Classic.

NOTE: The "Classic" JDK is no longer supported by IBM and thus is not covered in this document.

Java Virtual Machine is IBM Technology for Java
Gather 1 JVM summary, then 3 or more Java and Heap Dumps about 5 - 10 minutes apart and finally 1 GcCycles Dump for the JVM when it is experiencing the Out of Memory/Heap Leak issue.

Java Dump: WRKJVMJOB, Enter option 5 next to the job and press ENTER. Then, proceed to enter option 32 and press ENTER to request a Java dump.

Heap Dump: WRKJVMJOB, Enter option 5 next to the job and press ENTER. Then, proceed to enter option 30 and press ENTER to request a Heap dump.

GcCycles Dump: WRKJVMJOB, Enter option 9 next to the job and press ENTER. Then, proceed to press F7 to print the GC tables.

Notes:
  • As an alternative to the Java Dump, you can get similar thread information by collecting a PROCESSINFO. For detailed instructions on gathering PROCESSINFO, refer to the following Rochester Support Center knowledge base document:
    Processinfo: The Alternative to Viewing JVM Thread Data

Appendix B: How to send diagnostic information to IBM support

Java Virtual Machine is IBM Technology for Java
  1. Collect the following pieces of data output:
    • WRKPTFGRP
    • DSPPTF
    • WRKSYSSTS
    • WRKSBS
    • WRKSHRPOOL
    • WRKACTJOB
    • JVM Summary Dump
    • Java, Heap, and GcCycle Dumps

    To retrieve the file:
    1. Open the System i Navigator.

    2. Expand your system and log in as the user profile who created the spool files.

    3. Expand Basic Operations, and then left-click on Printer Output to display the output files.

    4. Locate the following spool files:
      • QSYSPRT (WRKPTFGRP)
      • QSYSPRT (DSPPTF)
      • QPDSPSTS (WRKSYSSTS)
      • QPDSPSBS (WRKSBS)
      • QSYSPRT (WRKSHRPOOL)
      • QPDSPAJB (WRKACTJOB)
      • QPJVAWRKVM (JVM Summary & GcCycles)

    5. Finally, drag-and-drop or right-click and select "Export" to move the files from the System i Navigator to your Desktop. Plain text files will be created.

  2. Collect the javacore*.txt and heapdump*.phd files from the location specified in the server's joblog.

  3. Collect the Collector host_name-cell_name-node_name-profile_name.JAR output file.

    Note: The collector program writes its output JAR file to the current directory.

  4. Send the diagnostic information to IBM support using Uploading files directly to ECuRep using FTP.

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Out of Memory","Platform":[{"code":"PF012","label":"IBM i"}],"Version":"9.0;8.5.5;8.5;8.0;7.0;6.1","Edition":"Base;Express;Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
15 June 2018

UID

swg21382896

Page Feedback