SPU core file analysis on IBM Netezza 1000 systems

Answer

SPU core files contain debugging information that can be useful if a SPU aborts on the host. On IBM Netezza 1000 (formerly TwinFin) systems, SPU core files are stored as gzip files and located in the /nz/kit/log/spucores directory.  To generate a back trace on an IBM Netezza 1000 SPU core you must first gunzip the file.

The nzgdb utility can be used on IBM Netezza 1000 systems to process SPU core files (or an active process).  The utility is located in the /nz/kit/bin/adm directory.  

The following example displays output generated by running the utility:

[nz@TEST /nz/kit/log/spucores]$ nzgdb -core core.1.3.spu10.t1254160070.p6256.s6 | more
Using debug-enabled command: /nz/kit.5.0.2/sbin/spu10.elf
Cannot determine plan object file: skipping.
warning: core file may not match specified executable file.
warning: .dynamic section for "/nz/kit.5.0.2/sbin/crosscc/spu10/sysroot/lib/libdl.so.2" is not at the expected address
warning: difference appears to be caused by prelink, adjusting expectations
warning: .dynamic section for "/nz/kit.5.0.2/sbin/crosscc/spu10/sysroot/lib/libpthread.so.0" is not at the expected address
warning: difference appears to be caused by prelink, adjusting expectations
warning: .dynamic section for "/nz/kit.5.0.2/sbin/crosscc/spu10/sysroot/usr/lib/libsysfs.so.2" is not at the expected address
warning: difference appears to be caused by prelink, adjusting expectations
warning: .dynamic section for "/nz/kit.5.0.2/sbin/crosscc/spu10/sysroot/usr/lib/libz.so.1" is not at the expected address
warning: difference appears to be caused by prelink, adjusting expectations
warning: .dynamic section for "/nz/kit.5.0.2/sbin/crosscc/spu10/sysroot/lib/libm.so.6" is not at the expected address
warning: difference appears to be caused by prelink, adjusting expectations
warning: .dynamic section for "/nz/kit.5.0.2/sbin/crosscc/spu10/sysroot/lib/libc.so.6" is not at the expected address
warning: difference appears to be caused by prelink, adjusting expectations
warning: .dynamic section for "/nz/kit.5.0.2/sbin/crosscc/spu10/sysroot/lib/ld-linux.so.2" is not at the expected address
warning: difference appears to be caused by prelink, adjusting expectations
Error while mapping shared library sections:
/dev/shm/spuplans/4672_1_13.o: No such file or directory.
Core was generated by `spu ev'.
Program terminated with signal 6, Aborted.
[New process 6256]
#0  0x4c01589b in raise () from /nz/kit.5.0.2/sbin/crosscc/spu10/sysroot/lib/libpthread.so.0
#0  0x4c01589b in raise () from /nz/kit.5.0.2/sbin/crosscc/spu10/sysroot/lib/libpthread.so.0
#1  0x0816453e in _raise_dfl (sig=6) at /production/builds/rel-5.0.2/090911-10217rel-5.0.2/main/src/nde/spu/spujobtask.cpp:1329
#2  0x081645f9 in _crash_handler (sig=6, si=0x855e76c, ignore=0x855e7ec) at /production/builds/rel-5.0.2/090911-10217rel-5.0.2/main/s

rc/nde/spu/spujobtask.cpp:1433
#3  <signal handler called>
#4  0x4beec5a6 in raise () from /nz/kit.5.0.2/sbin/crosscc/spu10/sysroot/lib/libc.so.6
#5  0x4beefd18 in abort () from /nz/kit.5.0.2/sbin/crosscc/spu10/sysroot/lib/libc.so.6
#6  0x081fa0df in CError_abort (err=@0xffe32454) at /production/builds/rel-5.0.2/090911-10217rel-5.0.2/main/src/nde/sys/error.cpp:450
#7  0x081fa145 in CError_AssertFailed (line=275, src=0x82ea3b4 "/production/builds/rel-5.0.2/090911-10217rel-5.0.2/main/src/nde/spu/s
tmcatalog.cpp") at /production/builds/rel-5.0.2/090911-10217rel-5.0.2/main/src/nde/sys/error.cpp:508
#8  0x0810954c in CStorageMgr::updateCatNewStuff (this=0x724e8200, which=PART_A, tableP=0x766a1f90, rootTable=322696) at /production/
builds/rel-5.0.2/090911-10217rel-5.0.2/main/src/nde/spu/stmcatalog.cpp:275
#9  0x080f7e75 in CIndexMgr::ThinTableDone (this=0x701c81e0, pNode=0x768548fc) at /production/builds/rel-5.0.2/090911-10217rel-5.0.2/
main/src/nde/spu/indexmgr.cpp:102
#10 0x081b23dd in ExecBindIndexDone (pNode=0x768548fc) at /production/builds/rel-5.0.2/090911-10217rel-5.0.2/main/src/nde/xp/xpbindin
dex.cpp:42
#11 0xf7917f58 in ?? ()
#12 0x768548fc in ?? ()
#13 0x00000001 in ?? ()
#14 0xf75a78f4 in ?? ()
#15 0xf75a2fb0 in ?? ()
#16 0xf75a2fc4 in ?? ()
#17 0xffe32838 in ?? ()
#18 0x76854670 in ?? ()
#19 0x01599730 in ?? ()
#20 0x766d3080 in ?? ()
#21 0x768548fc in ?? ()
#22 0x76854614 in ?? ()
#23 0x7678e2f0 in ?? ()
#24 0x7685471c in ?? ()
#25 0x7738d6d0 in ?? ()
#26 0x7685483c in ?? ()
#27 0x76854a08 in ?? ()
#28 0x7685e010 in ?? ()
#29 0x0003b722 in ?? ()
#30 0x4c00f39e in pthread_mutex_lock () from /nz/kit.5.0.2/sbin/crosscc/spu10/sysroot/lib/libpthread.so.0
--More--

Below is the help output for the nzgdb utility:

[nz@test50-1 adm]$ nzgdb -h
nzgdb -- Invoke the debugger on an active process or a core file

Usage: nzgdb [-K <kitdir>] [-D <datadir>] \
             [-pid <pid>] [-dev <devname>|-id <hwid>] [-core <corefile>] \
             [-bt] [-run]
[ ...]] -K     Kit to be debugged -D     Data directory (only necessary for generated code debugging) -pid       Attach to active process with PID     -core         Debug core file     -dev         Attach to the process on device (hostname or IP addr)     -id         Attach to the process on device with hardware ID     -run         Run a new instance of the command     -bt         Generate a backtrace and exit, rather than interactive debugging             Name of the binary to debug             Arguments to invoke with (only valid with -run)