Mustgather: Read first for problems encountered with Automatic Binary Optimizer for z/OS

Answer

Gathering General Information (Required for ALL Problem Types)

Gathering Problem-Specific Information

. Optimizer Problem - What to Gather?

. Run-time Problem - What to Gather?

. Performance Problem - What to Gather?

General Hints and Tips

. Find z/OS Version & PTF Level

. Find ABO Version & PTF Level

. Find ABO Runtime APAR Level

. How to Create a SYSMDUMP

. How to Search for Known Problems

Submitting Information to IBM Support

What is a Mustgather document and how does it help me?

---

Gathering General Information (Required for ALL Problem Types)

General information is information that is needed for any reported problem:

• What error/problem occurred?
• What action were you attempting at the time of the failure?
• Was a temporary fix applied?
• Operating system (z/OS) version, release and PTF level. This information can be obtained through SMP/E.
• Compiler language/version of the optimized module(s) involved.
• Subsystem (CICS, DB2, IMS, WAS) version, release, modification, and PTF levels, if applicable.
• A contact phone number and email address.

Gathering Problem-Specific Information

From the list below, select the type of problem that best describes the problem you are experiencing:

Optimizer Problem

Run-time Problem

Performance Problem

Optimizer Problem - What to Gather?

Gather the following documentation for ABO optimization problems before contacting IBM support:

• When opening a PMR, please the provide the general information requested here . Then return to this section to continue gathering additional documentation to provide to IBM.
• The Automatic Binary Optimizer version, release, and modification level. Click here for information about how to obtain this information.
• The original COBOL compile listing(s) for the optimized module(s) that appear to be involved in the issue (tersed). The program(s) must be compiled with the LIST, MAP, NOOFFSET compiler options to be usable by IBM. Tersed file names must end in .TRS when uploaded, in order to trigger IBM's automation.
• The original executable module(s), i.e. the input into ABO. These executable(s) must be put into sequential (XMIT) format prior to tersing. Click here for the JCL to XMIT and TERSE one or more executable modules. Tersed file names in XMIT format must end in .XMIT.TRS when uploaded, in order to trigger IBM's automation.
• The ABO optimization job output (tersed). The job output should contain all of the following:
1. The JCL and JES messages
2. The OPTLOG messages from DD=OPTLOG
3. The ABO listing transform from DD=SYSPRINT
If either (or both) the OPTLOG or SYSPRINT is written to a dataset, terse and upload the individual dataset(s), along with the job output.
• The full list of input parameters used in the SYSIN DD in the ABO optimization job, such as ARCH, ALLOW, REPLACE, etc...
• The CEEDUMP (tersed) if available.

Run-time Problem - What to Gather?

Gather the following documentation for run-time problems with modules optimized by Automatic Binary Optimizer before contacting IBM support:

• When opening a PMR, please the provide the general information requested here . Then return to this section to continue gathering additional documentation to provide to IBM.
• The Automatic Binary Optimizer version, release, and modification level. Click here for information about how to obtain this information.
• The Automatic Binary Optimizer runtime APAR level. Click here for information about how to obtain this information.
• The z/OS level.
• The original COBOL compile listing(s) for the optimized module(s) that appear to be involved in the issue (tersed). The program(s) must be compiled with the LIST, MAP, NOOFFSET compiler options to be usable by IBM. Tersed file names must end in .TRS when uploaded, in order to trigger IBM's automation.
• The original executable module(s), i.e. the input into ABO. These executable(s) must be put into sequential (XMIT) format prior to tersing. Click here for the JCL to XMIT and TERSE one or more executable modules. Tersed file names in XMIT format must end in .XMIT.TRS when uploaded, in order to trigger IBM's automation.
• The ABO optimization job output (tersed). The job output should contain all of the following:
1. The JCL and JES messages
2. The OPTLOG messages from DD=OPTLOG
3. The ABO listing transform from DD=SYSPRINT
If either (or both) the OPTLOG or SYSPRINT is written to a dataset, terse and upload the individual dataset(s), along with the job output.
• The full list of input parameters used in the SYSIN DD in the ABO optimization job, such as ARCH, ALLOW, REPLACE, etc...
• A small, reproducible test case (if possible).
• The job output from the execution of the optimized module(s) (tersed). The job output should include the execution JCL and JES messages.

• If your application abended, include the following:
• SYSMDUMP: For additional information on how to request a dump, refer to http://pic.dhe.ibm.com/infocenter/zos/v2r1/topic/com.ibm.zos.v2r1.ceea100/gsd.htm
• For program checks (S0Cx), you can add the following DD to the execution JCL to generate a SYSMDUMP:

//CEEOPTS DD *
 TERMTHDACT(UAIMM),TRAP(ON,NOSPIE),DYNDUMP(yourhlq,FORCE,TDUMP)
/*

- If running under IMS, provide the CEEOPTS override to the IMS region.
- DYNDUMP is ignored by CICS. (Instructions for generating a SYSMDUMP in
CICS can be found below.)
- Do not include the SYSMDUMP DD.
- The DYNDUMP assures that any setting of the customer's for this option is overridden.
- The FORCE will dump even in the presence of other dump DDs like SYSUDUMP.
- “yourhlq” will be used to form the initial part (HLQ) of the dump dataset name and can
be up to 26 characters in length. The HLQ should be one that the job has ALTER
authority to create.
- The name of the dump dataset can be found in the job's JESMSGLG and looks like:
+CEE3798I ATTEMPTING TO TAKE A DUMP FOR {abend} TO DATA SET:
          {dsn-prefix}.Dddd.Thhmmsss.{jobname}
If there are multiple instances, usually the first one is desired by IBM.

• For non-program checks (i.e. abends) in batch or for a CICS U4038, you should use the following to generate the SYSMDUMP:

//SYSMDUMP DD DSN=yourhlq.SYSMDUMP,DISP=(NEW,CATLG),
//            UNIT=SYSDA,SPACE=(CYL,(250,250),RLSE),
//            DCB=(LRECL=4160,RECFM=FBS,BLKSIZE=4160)
//CEEOPTS DD *
 TERMTHDACT(UADUMP) ABTERMENC(ABEND) TRAP(ON,SPIE)
/*
If either the SYSABEND or SYSUDUMP DD names are in the failing step, comment them out as they compete against the SYSMDUMP DD.

• To turn off competing software, you may need to add one or more of the following DDs:

//IDIOFF   DD  DUMMY   IBM FAULT ANALYZER OFF
//ABNLIGNR DD  DUMMY   ABEND-AID OFF              
//ESPYIBM  DD  DUMMY   EYE-SPY OFF                
//CAOESTOP DD  DUMMY   CA-OPT II & CA-SYMDUMP OFF        
//DMBENAN  DD  DUMMY   DUMPMASTER OFF        
//PSPOFF   DD  DUMMY   SOFTWORKS PERFORMANCE ESSENTIAL OFF
• If your environment includes a CICS region, include a SYSMDUMP that includes CICS region information. To create a SYSMDUMP that includes full CICS region/trace details, you must specify the following settings:
  • For CETR, all domains defined as ‘1’ except for ‘AP’ which is defined ‘1-2’.
  • Trace Table Size must be set at a minimum of 12 megabytes.
• If requested, create a CICS auxiliary trace. For information on using CICS traces, click here.
• If possible, determine if the abend is in IBM code or in your application code.



Back to top

Performance Problem - What to Gather?

In situations where performance gains are not seen from a module optimized by Automatic Binary Optimizer, gather the following documentation before contacting IBM support:
Note: Please open a separate problem record (PMR) for each performance issue to avoid confusion and decrease the mean time to resolution.

• When opening a PMR, please the provide the general information requested here . Then return to this section to continue gathering additional documentation to provide to IBM.
• The Automatic Binary Optimizer version, release, and modification level. Click here for information about how to obtain this information.
• The Automatic Binary Optimizer runtime APAR level. Click here for information about how to obtain this information.
• The z/OS level.
• The original COBOL compile listing(s) for the optimized module(s) that appear to be involved in the issue (tersed). The program(s) must be compiled with the LIST, MAP, NOOFFSET compiler options to be usable by IBM. Tersed file names must end in .TRS when uploaded, in order to trigger IBM's automation.
• The ABO optimization job output (tersed). The job output should contain all of the following:
1. The JCL and JES messages
2. The OPTLOG messages from DD=OPTLOG
3. The ABO listing transform from DD=SYSPRINT
If either (or both) the OPTLOG or SYSPRINT is written to a dataset, terse and upload the individual dataset(s), along with the job output.
• The full list of input parameters used in the SYSIN DD in the ABO optimization job, such as ARCH, ALLOW, REPLACE, etc...
• The original executable module(s), i.e. the input into ABO. The executable(s) must be put into sequential (XMIT) format prior to tersing. Click here for the JCL to XMIT and TERSE one or more executable modules. Tersed file names in XMIT format must end in .XMIT.TRS when uploaded, in order to trigger IBM's automation.
• The job output from the execution of the optimized module(s) (tersed). The job output should include the execution JCL and JES messages.
• Provide two profile reports from a tool such as IBM’s APA or Compuware’s Strobe, etc..., generated

(1) during the execution of the application with the original module(s) and
(2) during the execution of the application with the optimized module(s).
If such a tool is not available, use the RTI Profiler to generate the reports.
• The RTI Profiler is shipped with ABO 1.3 and produces a text report that shows the “hot” areas in the original module(s) and helps determine if time is actually spent in the CSECT(s) being optimized.
• The RTI Profiler can also be downloaded directly from http://www-01.ibm.com/support/docview.wss?uid=swg24043885

General Hints and Tips

• z/OS Operating System Version, Release and PTF Level: This information can be obtained through SMP/E.

• ABO Version, Release & Modification Level: The ABO OPTLOG listing page header provides information about the version, release, and modification level of the optimizer that was used to optimize your program. For example:

5697-AB1 IBM Automatic Binary Optimizer for z/OS 1.2.0
======== Tuesday Nov 01 2016 ========
10:53:37 Optimizer build level : tr_r16_binopt_20161006_125630 (Oct  6 2016 14:08:29)

• ABO Runtime APAR Level: The ABO run-time library release and APAR/PTF level information can be determined through SMP/E or by doing the following:

Browse *.SCEERUN(IGZOLPA)
Do a FIND on “HLQ=”.
Scroll to the right and locate the text similar to
“Prod build 2016-01-13 15:02:4020160113150252020100..PI55281”
to get the latest build date and APAR number for the ABO runtime.

• XMIT and TERSE JCL: The following JCL can be used to XMIT and TERSE one or more executable modules. (Please note that the file name being uploaded to IBM must end in .XMIT.TRS to trigger IBM's automation.)


//*********************************************************************
//* TO TAILOR, DO THE FOLLOWING:                                      
//* 1) CHANGE ALL AA1 HLQ                                              
//*          - WHERE 'HLQ' IS YOUR DATASET HIGH LEVEL QUALIFIER OR    
//*            USERID.                                                
//*          - TYPE IN ALL CAPITAL LETTERS.                            
//* 2) CHANGE ALL BB1 INDSN                                            
//*          - WHERE 'INDSN' IS THE PDS/PDSE CONTAINING THE            
//*            EXECUTABLE(S).                                          
//*          - TYPE IN ALL CAPITAL LETTERS.                            
//* 3) CHANGE ALL CC1 'MEMBERS'                                        
//*          - WHERE 'MEMBERS' IS THE SINGLE EXECUTABLE MEMBER OR      
//*            MULTIPLE MEMBERS SUCH AS 'ABERD100' OR                  
//*            'ABERD100 ABERD200 ABERD300'.                          
//*          - IF THE ENTIRE PDS/PDSE IS TO BE TRANSMITTED, THEN      
//*            SIMPLY DELETE THE MEMBERS KEYWORD IN THE JCL BELOW.    
//*********************************************************************
//* FOR CUSTOMERS:  DO NOT COPY PDSE MEMBERS INTO A PDS UNDER ANY      
//* CIRCUMSTANCES AND THEN ATTEMPT TO RUN THIS JOB ON THE PDS FILE.    
//* THE PROGRAM OBJECT WILL BE RENDERED UNUSABLE IF THIS IS DONE.
//*********************************************************************
//* DELETE THE XMIT FILES CREATED IN PREVIOUS RUN.                    
//*********************************************************************
//BR14S01  EXEC PGM=IEFBR14                                            
//DEL1     DD DSN=AA1.XMIT,DISP=(MOD,DELETE),                          
//            SPACE=(CYL,(20,20),RLSE)                                
//DEL2     DD DSN=AA1.XMIT.TRS,DISP=(MOD,DELETE),                      
//            SPACE=(CYL,(20,20),RLSE)                                
//*********************************************************************
//* XMIT PDS/PDSE TO A SEQUENTIAL FILE.                                
//* THE SEQUENTIAL FILE WILL ALWAYS BE LRECL=80,RECFM=FB,BLKSIZE=3120.
//* 'X.Y' IS A REQUIRED PLACEHOLDER FOR SYSTEM.USERID (BUT IS IGNORED)
//* AND MAY BE CODED AS-IS (THE LETTER X THEN A DOT THEN THE LETTER Y).
//* THE MEMBER NAME IS OPTIONAL. IF OMITTED, THE ENTIRE PDS/PDSE      
//* WILL BE USED AS INPUT.  THIS WILL CONVERT THE PDS/PDSE INTO A      
//* FORMAT THAT CAN BE TRANSMITTED.                                    
//*********************************************************************
//XMITS02  EXEC PGM=IKJEFT01,REGION=6144K,DYNAMNBR=999                
//SYSPRINT DD SYSOUT=*                                                
//SYSTSPRT DD SYSOUT=*                                                
//SYSTSIN  DD *                                                        
 XMIT X.Y -                                                            
 DSN('BB1') -                                                          
 MEMBERS(CC1) -                                                        
 OUTDSN('AA1.XMIT')                                                    
/*                                                                    
//*********************************************************************
//* TERSE (COMPRESS) THE FILE.                                        
//*********************************************************************
//TRSS03   EXEC PGM=TRSMAIN,PARM='PACK',COND=(0,NE)                    
//SYSPRINT DD SYSOUT=*                                                
//INFILE   DD DSN=AA1.XMIT,                                            
//            DISP=SHR                                                
//OUTFILE  DD DSN=AA1.XMIT.TRS,                                        
//            DISP=(NEW,CATLG),UNIT=SYSDA,                            
//            SPACE=(CYL,(50,50),RLSE),                                
//            DCB=(LRECL=1024,BLKSIZE=6144)    
//********************************************************************* 

• Search For Known Problems: The following locations can be searched for problem information:
• Search for known problems on Google, using search text "texttosearch SITE:IBM.COM".
• Search for known problems on the Automatic Binary Optimizer Fixlist webpage.
• Search the Automatic Binary Optimizer documentation in the IBM Automatic Binary Optimizer for z/OS documentation library at http://www-01.ibm.com/support/docview.wss?uid=swg27046990 .
• Review Preventive Service Planning buckets if you are installing IBM Automatic Binary Optimizer for z/OS.

Submitting Information to IBM Support

After a Problem Management Record (PMR) is open, you can submit diagnostic MustGather data to IBM using one of the following methods:

• FTP diagnostic data to IBM. For instructions, click here.
• If FTP is not possible, email diagnostic data to techsupport@ecurep.ibm.com. You must add PMR xxxxx bbb ccc in the subject line of your email. xxxxx is your PMR number, bbb is your branch office, and ccc is your IBM country code. Click here for more details.

Update your Problem Management Record (PMR) to indicate that data has been sent. You can update your PMR, or open a new PMR, in one of two ways:

• Online: Go to the Submit and track problems page on the IBM Software Support site (www.ibm.com/software/support/probsub.html) or IBMLink to open an Electronic Technical Response (ETR). Enter your information into the appropriate problem submission tool.
• By phone: Call 1-800-IBMSERV (1-800-426-7378) in the United States or, from other countries, go to the contacts page of the IBM Software Support Handbook on the Web (techsupport.services.ibm.com/guides/contacts.html) and click the name of your geographic region.

What is a Mustgather document and how does it help me?

MustGather documents aid in problem determination and save time resolving Problem Management Records (PMRs). These documents are located on the product support sites and contain specific instructions about what documentation to gather for specific problems. You can find available MustGather documents by searching on the word MustGather on the Support Portal Web pages:

http://www.ibm.com/support/entry/portal/Overview

Collecting MustGather data early, even before opening the PMR, helps IBM® Support quickly determine if:

1. Symptoms match known problems (rediscovery).
2. There is a non-defect problem that can be identified and resolved.
3. There is a defect that identifies a workaround to reduce severity.
4. Locating root cause can speed development of a code fix.