IBM Support

MustGather: Diagnosing out-of-memory problems in WebSphere InterChange Server V4.3 on Windows and AIX platforms

Troubleshooting


Problem

Collect the MustGather information for out-of-memory problems WebSphere InterChange Server. Gathering this information before calling IBM support helps support staff understand the problem and saves time when analyzing the data.Garbage-collection output (verbose:gc), javacore files, and heap dumps can be helpful in diagnosing out-of-memory problems in WebSphere InterChange Server.

Resolving The Problem

Collect the MustGather information for out-of-memory problems WebSphere InterChange Server. Gathering this information before calling IBM support helps support staff understand the problem and saves time when analyzing the data.Garbage-collection output (verbose:gc), javacore files, and heap dumps can be helpful in diagnosing out-of-memory problems in WebSphere InterChange Server.

 


If you have already contacted IBM Support, continue reading to begin collecting WebSphere InterChange Server MustGather data. Otherwise, refer to the following document: WebSphere InterChange Server MustGather: read first.

To improve the accuracy of complete data collection, IBM recommends you use the automated data collectors within IBM Support Assistant. Not only will the automated collector gather the equivalent of the manual process, it will also provide a secure file transfer of the collection to IBM.

IBM Support Assistant Lite IBM Support Assistant Lite (ISA Lite)
A special offering that contains just the data collector component of the IBM Support Assistant Workbench
Download | Demo | Screenshot | Learn more
IBM Support Assistant Workbench IBM Support Assistant Workbench (ISA Workbench)
For a full range of problem solving features, the workbench contains all the data collection capability of ISA Lite plus much more!
Download | Demo | Screenshot | Learn more

Collecting data using IBM Support Assistant

ISA Lite

  1. Download ISA Lite and extract it to your WICS_HOME directory.
     
  2. Start ISA Lite by running runISALite.[sh|bat].
     
  3. Select the following collector:
    WebSphere InterChange Server > Runtime > Out of Memory Problem
     
  4. Follow the prompts to automatically submit the collected data to IBM Support.

ISA Workbench

  1. Download and install the ISA Workbench.
     
  2. Install the product add-on for your release of WebSphere InterChange Server.
     
  3. Open the Workbench and navigate to the Collect and Send Data page.
     
  4. Select the following collector:
    WebSphere InterChange Server > Runtime > Out of Memory Problem
     
  5. Follow the prompts to automatically submit the collected data to IBM Support.

Collecting data manually

The sections that follow describe the steps you should take to gather and submit information to IBM Support.



Enabling and collecting garbage-collection output (verbose:gc)
Verbose:gc logs provide statistics on the state of memory during garbage collection. To enable verbose:gc, add the verbose:gc flag to the Java executable file in the startup script. Then restart InterChange Server to activate verbose:gc logging. To enable verbose:gc logging, refer to the following instructions for your platform:
  • AIX
    Modify the ics_manager file so that the Java executable command line reads as follows:

    exec $JAVA_ICS -verbose:gc -classpath "${CWCLASSES}"...

    Verify that verbose:gc messages are logged to the ics_manager.log file.
  • Windows
    Modify the start_server.bat file so that the Java executable command line reads as follows:

    NaV -verbose:gc -Djava.....%2 %3 %4 %5 2>verbosegc.log

    In this example, garbage-collection output is redirected to the verbosegc.log file that is written to in the NaV\jre\ directory.

Collecting heap dumps and javacore files
Heap dumps contain a list of objects in the Java heap. They are usually created when an out-of-memory problem occurs in WebSphere InterChange Server. You can also force a heap dump. Because the heap dump file can be large, make sure that adequate disk space is available. To enable heap dumps, refer to the following instructions for your platform:
  • AIX
    Enable heap dumps by adding the following environment variables:




  •  
  •  
  • Set 1:

    export IBM_HEAP_DUMP=true
    export IBM_HEAPDUMP=true
    export IBM_HEAPDUMP_OUTOFMEMORY=true
    export IBM_JAVADUMP_OUTOFMEMORY=true
    export IBM_JAVA_HEAPDUMP_TEXT=true


    You can also set the JAVA_DUMP_OPTS variable to explicitly trigger dumps. You can use this variable for any signal or for a specific condition, such as out of memory. Set up the JAVA_DUMP_OPTS variable as follows:

    export JAVA_DUMP_OPTS="ONANYSIGNAL(JAVADUMP[n],HEAPDUMP[n])"

    where n is the maximum number of heap dumps or Java dumps that can be generated. For example, if n=5 then a maximum of 5 javacores and heap dumps are generated. After that, for any signal, no more dumps are generated. This variable is particularly useful for generating dumps explicitly when an out-of-memory condition is not encountered.

    Refer to the Java Diagnostics Guide for more options for using this variable for out-of-memory conditions.

    You can enable dumps using variables in Set 1 or using the JAVA_DUMP_OPTS variable. When you use multiple variables, depending on the service refresh versions, the order of processing varies. Refer to the Java Diagnostics Guide for detailed information.

    (Optional) To specify the location of the dumps, you can set the variables as follows:

    export IBM_HEAPDUMPDIR=alternate_directory
    export IBM_JAVACOREDIR=alternate_directory

    You can add the variables to the ics_manager script or to the profile. Restart InterChange Server
    to activate the heap dumps. Check that heap dumps have been activated by running the command kill -3 Interchange_Server_process_ID while InterChange Server is running. The WebSphere InterChange Server process is momentarily interrupted to write the heap dump, but the process is not stopped. A message is logged to the ics_manager.log file that indicates the location of the heap dump.




  •  
  •  
  • Windows
    Enable heap dumps by adding the following environment variables:

    Set 1:

    set IBM_HEAP_DUMP=true
    set IBM_HEAPDUMP=true
    set IBM_HEAPDUMP_OUTOFMEMORY=true
    set IBM_JAVADUMP_OUTOFMEMORY=true
    set IBM_JAVA_HEAPDUMP_TEXT=true


    You can also set the JAVA_DUMP_OPTS variable to explicitly trigger dumps. You can use this variable for any signal or for a specific condition, such as out of memory. Set up the JAVA_DUMP_OPTS variable as follows:

    export JAVA_DUMP_OPTS="ONANYSIGNAL(JAVADUMP[n],HEAPDUMP[n])"

    where n is the maximum number of heap dumps or Java dumps that can be generated. For example, if n=5 then a maximum of 5 javacores and heap dumps are generated. After that, for any signal, no more dumps are generated. This variable is particularly useful for generating dumps explicitly when an out-of-memory condition is not encountered.

    Refer the Java Diagnostics Guide for more options for using this variable for out-of-memory conditions.

    You can enable dumps using variables in Set 1 or using the JAVA_DUMP_OPTS variable. JAVA_DUMP_OPTS works for all service refreshes of JDK 1.4.2. When you use multiple variables, depending on the service refresh versions, the order of processing varies. Refer to the Java Diagnostics Guide for detailed information.

    As an option, you can specify the location of the dumps, as follows:

    set IBM_HEAPDUMPDIR=another_directory
    set IBM_JAVACOREDIR=another_directory

    You can add the variables to the start_server.bat script or to the Windows environment. Restart InterChange Server to activate the heap dumps. Check that heap dumps have been activated by sending a Ctrl-Break signal in the InterChange Server console window. The WebSphere InterChange Server process is momentarily interrupted to write the heap dump, but the process is not stopped. A message is logged indicating the exact location of the heap dump file.

Sending diagnostic information to IBM Software Support
To send diagnostic information to IBM Technical Support, follow the instructions in Exchanging information with IBM Technical Support for problem determination for problem determination.

For a list of all technotes, downloads, and educational materials that are specific to WebSphere InterChange Server, search the WebSphere InterChange Server support site.

Exchanging data with IBM Support

To diagnose or identify a problem, it is sometimes necessary to provide Technical Support with data and information from your system. In addition, Technical Support might also need to provide you with tools or utilities to be used in problem determination. You can submit files using one of following methods to help speed problem diagnosis:

 



[{"Product":{"code":"SSBTEG","label":"WebSphere InterChange Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"ICS","Platform":[{"code":"PF002","label":"AIX"}],"Version":"4.3","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

More support for:
WebSphere InterChange Server

Software version:
4.3

Operating system(s):
AIX

Document number:
87425

Modified date:
02 July 2018

UID

swg21250293