IBM Support

MustGather: Collecting a PEX Heap Trace - QMGTOOLS/STRPSC

Troubleshooting


Problem

Use this document to collect PEX trace data for debugging a heap leak issue.

Resolving The Problem

PEX heap traces are used to isolate which programs and procedures are allocating the most heap storage during an active trace.  The PEX trace must be active while the heap consumption is occurring.  If heap errors are already being signaled because the heap is full, then the heap must first be reset and the workload restarted before starting the trace.   The scope of PEX heap trace should be limited to the specific heap type, specific jobs, and a specific heap control segment.    Since heaps can have high volumes of activity and heap leaks can be very slow, scoping the trace correctly reduces the size of the trace and allows for a larger window of time to capture the issue.
vvvvvvvvvvvvv BEFORE YOU BEGIN vvvvvvvvvvvvvvvvvvvvvvvv
        These items must be known before using this document:
  • a) How can the heap storage be reset after it fills?  
    This depends on how the code uses the heap.  Resetting the heap may be as simple as restarting a given job, may require an IPL, or it may require a series of steps.
  • b) What jobs should be traced and make use of the heap?    A specific job, a group of generic jobs, all jobs?
  • c) What type of heap is being consumed?   Activation Group (*ACTGRPHEAP), Resident (*RESHEAP), System (*SYSHEAP), Handle (*HDLHEAP), Local (*LCLHEAP), User (*USRHEAP)?
  • d) What heap control segment (HCS) address should be traced?  This will be a 10 character hexadecimal number. 
  • e) How long should the trace be run?  Does the heap consumption happen over a period of hours or days?
  • f) How will the heap usage be identified?  
^^^^^^^^  BEFORE YOU BEGIN ^^^^^^^^^^^^^

- Starting the trace if the heap is already full is not useful.
- What type of heap is involved?  Activation Group (*ACTGRPHEAP), Resident (*RESHEAP), System (*SYSHEAP), Handle (*HDLHEAP), Local (*LCLHEAP), User (*USRHEAP)?
- Which of the following macros can be used to determine the current state of the heap and monitor for growth?

USERHEAPS (user heap):  https://www.ibm.com/support/pages/node/6557466
ACTIVATIONHEAPS (activation group heap):  https://www.ibm.com/support/pages/node/6556810
SYSTEMHEAPS (resident and system heap): https://www.ibm.com/support/pages/node/6557332
HANDLEHEAPS (handle heap):  https://www.ibm.com/support/pages/node/6557454  
 
Step 1: Verify that required Performance Explorer (PEX) PTFs are applied:
Step 2: Verify that QMGTOOLS library is installed and current:
  1. ADDLIBLE QMGTOOLS
  2. GO MG
  3. Option 12:  Display Build Date
  • If the build date is more than one month old and the release is correct, update the tool using option 13:  Check IBM for updated QMGTOOLS.
  • If the QMGTOOLS library  does not exist or the release is incorrect,  install the tool using these instructions.
Step 3: Add the PEX definition for the specific heap type and specific jobs:
  • For job_name specify one of the following:
    *ALL for all jobs, a generic job name such as QPADEV*, a specific job name such as 999999/user_id/job_name
  • For heap_type, specify one of the following: 
    *ACTGRPHEAP, *RESHEAP, *SYSHEAP, *HDLHEAP, *LCLHEAP, *USRHEAP
ADDPEXDFN DFN(HEAP)         
          TYPE(*TRACE)      
          JOB((job_name *ALL))  
          TASK(*ALL)        
          MAXSTG(4000000)   
          TRCTYPE(*SLTEVT)  
          SLTEVT(*YES)      
          MCHINST(*NONE)    
     STGEVT((heap_type)) 
Example for *ACTGRPHEAP and *ALL jobs:
ADDPEXDFN DFN(HEAP)         
          TYPE(*TRACE)      
          JOB((*ALL *ALL))  
          TASK(*ALL)        
          MAXSTG(4000000)   
          TRCTYPE(*SLTEVT)  
          SLTEVT(*YES)      
          MCHINST(*NONE)    
     STGEVT((*ACTGRPHEAP)) 
Step 4 (Optional): Add the PEX filter for the heap type and Heap Control Segment (HCS):
  • *ACTGRPHEAP:  ADDPEXFTR FTR(FILTER) USRDFNFTR((*EQ 4 11 64 *HEX (aabbccddee) *NONE))
  • *RESHEAP:  ADDPEXFTR FTR(FILTER) USRDFNFTR((*EQ 4 8 64 *HEX (aabbccddee) *NONE))
  • *SYSHEAP:  ADDPEXFTR FTR(FILTER) USRDFNFTR((*EQ 4 7 64 *HEX (aabbccddee) *NONE))
  • *HDLHEAP:  ADDPEXFTR FTR(FILTER) USRDFNFTR((*EQ 4 12 64 *HEX (aabbccddee) *NONE))
  • *LCLHEAP:  ADDPEXFTR FTR(FILTER) USRDFNFTR((*EQ 4 9 64 *HEX (aabbccddee) *NONE))
  • *USRHEAP:  ADDPEXFTR FTR(FILTER) USRDFNFTR((*EQ 4 10 64 *HEX (aabbccddee) *NONE))
Example for *ACTGRPHEAP and HCS:
ADDPEXFTR FTR(FILTER) USRDFNFTR((*EQ 4 11 64 *HEX (F1CD6A7044) *NONE))
Step 5: Start the PEX trace.  If a filter was created in step 4, then specify PEXFILER(FILTER) instead of PEXFILTER(*NONE):
QMGTOOLS/STRPSC FUNCTION(*START)   
                LIBRARY(IBMPEX)    
                CLRLIB(N)          
                STRPEX(Y)          
                PEXDFN(HEAP)       
                PEXFILTER(*NONE)   
                SIZE(6)            
                PEX2KEEP(6)        
                JOBWATCHER(N)      
                CSI(N)             
Example of starting with a PEX filter:
QMGTOOLS/STRPSC FUNCTION(*START)   
                LIBRARY(IBMPEX)    
                CLRLIB(N)          
                STRPEX(Y)          
                PEXDFN(HEAP)       
                PEXFILTER(FILTER)
                SIZE(6)            
                PEX2KEEP(6)        
                JOBWATCHER(N)      
                CSI(N)         
Step 6:  Recreate the problem.  Monitor the heap usage for a period consistent with the time of the problem.  
Here are common macros used for monitoring the different types of heaps:
USERHEAPS (user heap):  https://www.ibm.com/support/pages/node/6557466
ACTIVATIONHEAPS (activation group heap):  https://www.ibm.com/support/pages/node/6556810
SYSTEMHEAPS (resident and system heap): https://www.ibm.com/support/pages/node/6557332
HANDLEHEAPS (handle heap):  https://www.ibm.com/support/pages/node/6557454  
 
Step 7 : Stop the trace.  Wait for the submitted job to complete:
QMGTOOLS/STRPSC FUNCTION(*DUMP)
Step 8. Send the data to IBM:
Save the IBMPEX data library to a save file and upload it to the case.


 

[{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG60","label":"IBM i"},"ARM Category":[{"code":"a8m0z0000000CGHAA2","label":"MustGather"}],"ARM Case Number":"","Platform":[{"code":"PF012","label":"IBM i"}],"Version":"All Versions"}]

Historical Number

587269612

Document Information

Modified date:
05 March 2024

UID

ibm16427019

Page Feedback