Must Gather - OMNIbus Probes Core Dumping

Answer

Identify the version and Fix Pack in use

$OMNIHOME/probes/nco_p_<probe> -version

ls -ltr $NCHOME/_uninst
/home/<username>/.acsi_<username>/bin/listIU.sh
(where <username> is the non-root user id used to install OMNIbus)

If the installation was performed by the root user, the output of: /usr/ibm/common/acsi/bin/listIU.sh

/home/<username>/IBM/InstallationManager/eclipse/tools/imcl listInstalledPackages

Operational Information (gathered using typescript):

script /tmp/L2_ProbeCore.txt
* This records all commands until you exit from the script using Ctrl + D
uname -a
* For Linux, cat /etc/redhat-release or cat /etc/suse-release
* For AIX, oslevel -r
ulimit –a
env > /tmp/env.txt
ps -ef | grep nco_p*
* To check which probe processes are currently running.

Start the probe as follows:

$OMNIHOME/probe/nco_p_<probe> -server <server> -messagelevel debug &

find $OMNIHOME -name "*core*" -print

Technote Id 1241093 Troubleshooting coring applications
http://www-01.ibm.com/support/docview.wss?uid=swg21241093


* For AIX:
dbx <path to binary> <path to core>
hit ENTER
type WHERE
hit ENTER
get the output

The most useful tool for troubleshooting coring application is a debugging tool, such as GDB:

Download GDB from the internet and install on the server where the program cored (or a copy of the server).

See below for the links.

Place gdb in your user path (/usr/local/bin)

Then run
gdb fullpath_to_binary_that_cored fullpath_to_core_file
(gdb) bt
(gdb) info threads

For example:
gdb $OMNIHOME/probes/solaris2/nco_p_mttrad /tmp/core
(gdb) bt
(gdb) info threads

The results of back trace provides details of where the program exited and possibly why.

Alongside the raw information from GDB the full debug log from the cored process, and the long listing of the core file (to provide the core file creation timestamp).

Linux : example using GBD
file /tmp/core
ls -l /tmp/core

gdb $OMNIHOME/probes/solaris2/nco_p_mttrad /tmp/core
(gdb) bt
(gdb) info threads

Solaris : example using 'pstack'
cd /tmp
file core
ls -l core

pstack core
pflags core

Control C script

Corba Probes:
Enable nonnative logging:
.
NDE_DEFAULT_LOG_LEVEL=debug
NDE_FORCE_LOG_MODULE=$OMNIHOME/log/nonnative_forced.log
NCO_P_NONNATIVE_TRANSCRIPT=$OMNIHOME/log/nonnative_debug.log
export NDE_DEFAULT_LOG_LEVEL
export NDE_FORCE_LOG_MODULE
export NCO_P_NONNATIVE_TRANSCRIPT

Enable Visobroker logging:
Edit $OMNIHOME/probes/java/nco_p_<probe>.env file and add the following
line to allow Visibroker logging.
NCO_JPROBE_JAVA_FLAGS="-Dvbroker.orb.debug=true
-Dvbroker.orb.logLevel=debug
-Dvbroker.orb.logger.output=$OMNIHOME/log/orb.debug
$NCO_JPROBE_JAVA_FLAGS"

How to turn on IBM Orb trace logging
Edit $OMNIHOME/probes/java/nco_p_<probe>.env file and add the following line to allow IBM Orb trace logging.
NCO_JPROBE_JAVA_FLAGS="-Dcom.ibm.CORBA.Debug=true -Dcom.ibm.CORBA.CommTrace=true -Dcom.ibm.CORBA.Debug.Output=$OMNIHOME/log/ibm_orb_trace.log $NCO_JPROBE_JAVA_FLAGS"

Some of the latest probes have two properties, to see if the version of the probe has the properties run -dumpprops against the wrapper script. 

ORBDebug : 'TRUE'
ORBDebugFile : '$OMNIHOME/orb_debug.log'