db2cos (callout script) output files

A db2cos script is invoked by default when the database manager cannot continue processing due to a panic, trap, segmentation violation or exception. Each default db2cos script will invoke db2pd commands to collect information in an unlatched manner.

The names for the db2cos scripts are in the form db2cos_hang, db2cos_trap, and so on. Each script behaves in a similar way except db2cos_hang which is called from the db2fodc tool.

The default db2cos scripts are found under the bin directory. On UNIX operating systems, this directory is read-only. You can copy the db2cos script file and all the relevant scripts to the adm directory if customization on all files is required. If a db2cos script is found in the adm directory, it is run; otherwise, the default script in the bin directory is run. For example, for data corruption issues, Db2® looks for a db2cos_datacorruption script. If there is a customized script in the adm directory, Db2 uses this customized script. Otherwise, the default db2cos_datacorruption script in the bin directory is used. For information on what scripts are called automatically and for which situation see, Collecting diagnosis information based on common outage problems.

In a multiple partition configuration, the script will only be invoked for the trapping agent on the partition encountering the trap. If it is required to collect information from other partitions, you can update the db2cos script to use the db2_all command or, if all of the partitions are on the same machine, specify the -alldbpartitionnums option on the db2pd command.

The types of signals that trigger the invocation of db2cos are also configurable via the db2pdcfg -cos command. The default configuration is for the db2cos script to run when either a panic or trap occurs. However, generated signals will not launch the db2cos script by default.

The order of events when a panic, trap, segmentation violation or exception occurs is as follows:

Trap file is created
Signal handler is called
db2cos script is called (depending on the db2cos settings enabled)
An entry is logged in the administration notification log
An entry is logged in the db2diag log file

The default information collected by the db2pd command in the db2cos script includes details about the operating system, the Version and Service Level of the installed Db2 product, the database manager and database configuration, as well as information about the state of the agents, memory pools, memory sets, memory blocks, applications, utilities, transactions, buffer pools, locks, transaction logs, table spaces and containers. In addition, it provides information about the state of the dynamic, static, and catalog caches, table and index statistics, the recovery status, as well as the reoptimized SQL statements and an active statement list. If you have to collect further information, update the db2cos script with the additional commands.

When the default db2cos script is called, it produces output files in the directory specified by the DIAGPATH database manager configuration parameter. The files are named XXX.YYY.ZZZ.cos.txt, where XXX is the process ID (PID), YYY is the thread ID (TID) and ZZZ is the database partition number (or 000 for single partition databases). If multiple threads trap, there will be a separate invocation of the db2cos script for each thread. In the event that a PID and TID combination occurs more than once, the data will be appended to the file. There will be a timestamp present so you can distinguish the iterations of output.

The db2cos output files will contain different information depending on the commands specified in the db2cos script. If the default script is not altered, entries similar to the following will be displayed (followed by detailed db2pd output):

2005-10-14-10.56.21.523659                                                      
PID     : 782348               TID  : 1           PROC : db2cos
INSTANCE: db2inst1             NODE : 0           DB   : SAMPLE
APPHDL  :                      APPID: *LOCAL.db2inst1.051014155507              
FUNCTION: oper system services, sqloEDUCodeTrapHandler, probe:999
EVENT   : Invoking /home/db2inst1/sqllib/bin/db2cos from 
oper system services sqloEDUCodeTrapHandler
Trap Caught

Instance db2inst1 uses 64 bits and Db2 code release SQL09010
...
Operating System Information:

OSName:   AIX        
NodeName: n1     
Version:  5                    
Release:  2          
Machine:  000966594C00 

...

The db2diag log files will contain entries related to the occurrence as well. For example:

2005-10-14-10.42.17.149512-300 I19441A349         LEVEL: Event
PID     : 782348               TID  : 1           PROC : db2sysc
INSTANCE: db2inst1             NODE : 000
FUNCTION: Db2, trace services, pdInvokeCalloutScript, probe:10
START   : Invoking /home/db2inst1/sqllib/bin/db2cos from oper system
services sqloEDUCodeTrapHandler

2005-10-14-10.42.23.173872-300 I19791A310         LEVEL: Event
PID     : 782348               TID  : 1           PROC : db2sysc
INSTANCE: db2inst1             NODE : 000
FUNCTION: Db2, trace services, pdInvokeCalloutScript, probe:20
STOP    : Completed invoking /home/db2inst1/sqllib/bin/db2cos

2005-10-14-10.42.23.519227-300 E20102A509         LEVEL: Severe
PID     : 782348               TID  : 1           PROC : db2sysc
INSTANCE: db2inst1             NODE : 000
FUNCTION: Db2, oper system services, sqloEDUCodeTrapHandler, probe:10
MESSAGE : ADM0503C  An unexpected internal processing error has occurred.  ALL
          DB2 PROCESSES ASSOCIATED WITH THIS INSTANCE HAVE BEEN SHUTDOWN.
          Diagnostic information has been recorded.  Contact IBM Support for
          further assistance.

2005-10-14-10.42.23.520111-300 E20612A642         LEVEL: Severe
PID     : 782348               TID  : 1           PROC : db2sysc
INSTANCE: db2inst1             NODE : 000
FUNCTION: Db2, oper system services, sqloEDUCodeTrapHandler, probe:20
DATA #1 : Signal Number Recieved, 4 bytes
11
DATA #2 : Siginfo, 64 bytes
0x0FFFFFFFFFFFD5C0 : 0000 000B 0000 0000 0000 0009 0000 0000    ................
0x0FFFFFFFFFFFD5D0 : 0000 0000 0000 0000 0000 0000 0000 0000    ................
0x0FFFFFFFFFFFD5E0 : 0000 0000 0000 0000 0000 0000 0000 0000    ................
0x0FFFFFFFFFFFD5F0 : 0000 0000 0000 0000 0000 0000 0000 0000    ................