NFS Diagnostic Data Collection Guidelines

Answer

NFS Diagnostic Data Collection Guidelines

This document describes the types of data collected for NFS diagnostic purposes and some typical data combinations required for selected problem classes.
Diagnostic Data Types:

The following types of Diagnostic data may be of value for NFS problem analysis:

• NFSS DEBUGx Trace - NFS Server activity trace log files.

For a description of the various NFSS Debug trace types see the NFS Server DEBUG Trace Types section below. The NFS Server uses two log files. It toggles to the other file whenever the currently written file is full.

This function is controlled via the startup proc for tracing during IPL, or via console commands for normal tracing control.

To capture the Log file, the current buffers must be flushed to disk and then the files must be swapped. After swapping, the desired file must be copied before it is overwritten by the automatic toggle mechanism.

NFS Server Debug Trace Commands

To start NFS Server Debug Trace from the console, enter:

MODIFY mvsnfs,LOG=level

<=== mvsnfs = name of NFS procedure in PROCLIB

<=== level = (ERROR, WARN, INFO, DEBUGn, n=1 - 9)

To flush the remaining NFS message buffer to disk, enter:

MODIFY mvsnfs,FLUSHLOG

<=== mvsnfs = name of NFS procedure in PROCLIB

To turn NFS Server Debug Trace off, issue the following console command:

MODIFY mvsnfs,LOG=INFO

To clear/reset the LOG Debug Trace Data Set, issue the following console command:

MODIFY mvsnfs,INITLOG

• NFSC Trace - NFS Client activity trace log files.

This trace can only be turned on or off. It does not have multiple debug levels. The NFS Client uses two log files. It toggles to the other file whenever the currently written file is full. The NFS Server and Client each have their own pair of files.

This function does not have any console command control capability. It is controlled from the UNIX command line (i.e. nfsstat -d xxx). It requires Superuser authority.

To capture the Log file, the current buffers must be flushed to disk and then the files must be swapped. After swapping, the desired file must be copied before it is overwritten by the automatic toggle mechanism.

NFS Client Trace Commands

To start NFS Client Trace from USS command line, enter:

/usr/lpp/NFS/nfsstat -d 1

To flush the remaining NFS message buffer to disk, enter:

/usr/lpp/NFS/nfsstat -d f

To switch NFS Client Trace to the other log file, issue the following USS command:

/usr/lpp/NFS/nfsstat -d s

To turn NFS Client Trace off, issue the following USS command:

/usr/lpp/NFS/nfsstat -d 0

To clear NFS Client Trace data, issue the following USS command:
/usr/lpp/NFS/nfsstat -d c

• OMVS CTRACE - z/OS OMVS (Unix System Services) activity trace.

It is normally maintained in a set of cyclical buffers in memory. However, it can also be written to a file via an external writer. When it is maintained in memory, a dump must be taken to capture the information for debug purposes.

OMVS CTRACE Commands

Turn on CTRACE via console command

TRACE CT,4M,COMP=SYSOMVS
R nn,OPTIONS=(ALL),JOBNAME=(NFS_server),END
Note: If JOBNAME keyword is used on Ctrace statement above, be aware that OMVS uses that parm for USERID, not for job name. So, the userid, not the jobname must be specified for this parm.


Reproduce problem in system with minimal activity

Collect dump via console dump (see Dump Commands)

Turn off CTRACE via console command

TRACE CT,OFF,COMP=SYSOMVS

• HFS CTRACE - z/OS HFS Physical File System activity trace.

It is normally maintained in a set of cyclical buffers in memory. However, it can also be written to a file via an external writer. When it is maintained in memory, a dump must be taken to capture the information for debug purposes.

HFS CTRACE Commands

To Start HFS CTRACE from the console, enter:

TRACE CT,8M,COMP=SYSSMS

R nn,JOBNAME=(OMVS),OPTIONS=(ENTRY,EXIT,EXITA,SPECIAL,CB,RRTN,COMP=(PFS)),END

Reproduce problem in system with minimal activity

Collect dump via console dump (see Dump Commands)

To turn CTRACE off, issue the following console command:

TRACE CT,OFF,COMP=SYSSMS





TCPIP Documentation for NFS related Problems

The TCP/IP CTRACE and Packet Trace are normally maintained in a set of cyclical buffers in memory. However, they can also be written to a file via an external writer. When they is maintained in memory, a dump must be taken to capture the information for debug purposes.

• TCP/IP CTRACE - z/OS TCP/IP Component trace.

To start the TCPIP Ctrace issue the following command:

TRACE CT,ON,COMP=SYSTCPIP,SUB=(tcpipprocname)
R XX,JOBNAME=(nfsprocname,tcpipprocname)
R XX,OPTIONS=(ENGINE,PFS,SOCKET,INTERNET,TCP,UDP,IOCTL),END

• TCP/IP Packet Trace - z/OS TCP/IP Packet trace.

To start the TCPIP Packet trace, issue the following command:

TRACE CT,ON,COMP=SYSTCPDA,SUB=(tcpipprocname)
V TCPIP,tcpipprocname,PKTTRACE,ON,ABBREV=152,IP=xx.xx.xx.xx
Note: xx.xx.xx.xx = client_IP_address

Recreate problem

To collect Dump of TCPIP Dataspace and TCPIP address space, issue the following MVS command from the system console:

DUMP COMM=('text')
R xx,JOBNAME=(tcpipprocname,nfsprocname),DSPNAME=('tcpipprocname'.*),
SDATA=(ALLNUC,PSA,GRSQ,SUM,CSA,LPA,LSQA,RGN,SWA,SQA,TRT),END
Note: (please see the z/OS Dump section below)

To stop the TCPIP Component trace, issue the following command:
Note: It is not necessary to stop the traces. Start them once, keep it running and flush the data
via the dump.


TRACE CT,OFF,COMP=SYSTCPIP,SUB=(tcpipprocname)
V TCPIP,tcpipprocname,PKT,OFF,IP=xx.xx.xx.xx

To stop the TCPIP Packet trace, issue the following command:

TRACE CT,OFF,COMP=SYSTCPDA,SUB=(tcpipprocname)

NOTE: The first step to collecting traces to the dataspace is to insure that the bufsize in CTIEZB00 in parmlib is set to at least 8Mb. It may need to be set higher depending on the amount of trace data desired, but 8mb should be a good starting point! Using its max bufsize of 256Mb, you will be sure that the most possible contiguous trace data is captured. Thus, we recommend to use 256Mb.
TCPIP will need to be restarted for the change in bufsize to take affect. The trace data will be captured via an MVS Dump Comm command that will dump the TCPIP Dataspace named TCPIPDS1. Be aware that this method may result in lost trace data as the possibility of wrapping is very possible. The dump command should be issued very soon after the problem happens. Following instructions could be found in II12014 or within TCPIP DIAGNOSIS GUIDE.






AIX Documentation for NFS related Problems:

rm /tmp/ibmsupt
Note: Make sure that there is enough space under /tmp for the IP-traces.
Systems with high TCP/IP traffic, may use up to 20MB per minute

snap -gkitn (takes 2-3 minutes and collects all necessary AIX informations)

• AIX IP Trace - AIX Client activity trace log file.

This trace contains a trace of the IP activity from the AIX Client's perspective.

To start AIX IP Trace, issue the following AIX command:

startsrc -s iptrace -a "-s <source address> -d <destination address> -b /tmp/ibmsupt/testcase/iptrc.bin"

cd /tmp/ibmsupt/testcase
script cmd_log (puts all keybaord actions to file cmd_log in the current directory)
Ping -c 5 (OS/390 NFS SERVER)
showmount -e <OS/390 IP address>
cd to NFS mounted OS/390 filessysem
cp /etc/filesystems /tmp/ibmsupt/testcase/filesystems.out
mount

reproduce the NFS error (mount problem, etc....)

To stop AIX IP Trace, issue the following AIX command:
stopsrc -s iptrace

To format Trace Report, issue the following AIX command:
ipreport -rns /tmp/ibmsupt/testcase/iptrc.bin > /tmp/ibmsupt/testcase/iptrc.out
Note: Please note that formatting of IP trace needs about 5 times the disk space as the raw tracefile /tmp/ibmsupt/testcase/Note:iptrc.bin

CTRL-D (to end script recording)
snap -c (generates snap.pax.Z in /tmp/ibmsupt)
Sent snap.pax.Z to AIX support

• SUN Snoop Trace - SUN Client activity trace log file.

This trace contains a trace of the IP activity from the SUN Client's perspective.

SUN Snoop Trace Commands

To start SUN Snoop Trace issue the following command:

snoop -o my.trace myclient myserver

To stop SUN Snoop Trace, issue:

<cntl c>

To format Trace Report, issue the following command:

snoop -i my.trace -v >my.trace.report

• z/OS Dump - Memory dump of the current state of the machine.

The actual contents of the dump will depend on the address spaces and data spaces selected. The dump will also include any CTRACEs running at the time the dump is taken. It is typically recommended to take a Synchronous dump, versus an Asynchronous one. This insures that the memory contents do not change between the time the dump request is made and the time the dump is actually taken by the machine.

Dump Commands

Collect dump from NFSS, NFSC, OMVS, TCPIP via console dump
DUMP COMM=(description)

R nn,JOBNAME=(OMVS,TCPproc,NFSSproc,NFSCproc,jobname*),CONT

R nn,DSPNAME=('OMVS'.*,'tcpproc'.*),CONT

R nn, SDATA=(PSA,SQA,LSQA,RGN,TRT,LPA,CSA,GRSQ,SUM,ALLNUC),CONT

R nn,END

NOTE - forked processes add a digit to end of parent jobname, hence jobname*



NFS Server DEBUG Trace Types:

The NFS Server has various trace types which control the amount of trace data recorded.

Error - Only writes error messages to the log.

Info - Writes both Error and Informational messages to the log. Some Informational messages are also written to the Console. See the NFS Messages Section (Appendix A) of the NFS Customization and Operation manual for details.

Debug x - In addition to Error and Informational messages, some NFS activity trace data is also written to the log. The specific trace data written depends on the debug level, i.e. 'x'.

Debug 1 - Request status information is recorded: requests issued and their associated replys. Also, some locking trace data is recorded.

Debug 2 - Debug 1 data, plus Network related tracing information.

Debug 3 - Debug 2 data, plus some key internal NFS trace points.

Debug 4 - Debug 3 data, plus detailed internal NFS flow information for everything, except for the HFS processing and data management functions.

Debug 5 - The Logical Layer of Data Management only.

Debug 6 - The Physical Layer of Data Management only.

Debug 8 - The HFS processing function only.

Note: This is only the NFS side of the HFS processing. It does not include any internal trace data from the HFS Physical File System itself.

Debug 9 - Everything.

Ideally, it would be best to use Debug 9. However when this is not possible because of performance impacts, typically the Debug 4 setting is required to collect the necessary NFS diagnostic information for debug. The other Debug settings, especially 5-8 are typically only used for specialized circumstances.

Note: To minimize the performance impact of recording the trace information, it is highly recommended that APAR/PTF OW53313/UW87648 should be installed on the system. This APAR moves the writing of the NFS Trace Log buffers to a separate asynchronous task eliminating the I/O delay from the main-line path.
It is recommended to have the most current NFS maintenance installed running the debug traces.




Problem Situation -> Diagnostic Data:

Needless to say, the ideal situation for any NFS failure is to get all of the above cited documentation. However for the case where that is not possible, the following table summarizes some typical NFS problem situations and the set of minimum Diagnostic Data that should be collected for their analysis. In no way is this inteneded to be a comprehensive list. It also does not imply that the cited data will always be adequate. In some instances, additional data may be required for the final problem analysis. This is only intended to provide a reasonable starting guideline when not all of the data is available.

Note: Any traces must have been turned on prior to the failure incident. Otherwise, the historical data necessary for the problem analysis will not be available. Trace data is only created when the particular trace is turned on. Therefore turning the trace on at the time of the failure will only provide trace data for the time after the failure occurrence.

Note: All traces collected for analysis of a problem must contain data for the same time period. Otherwise, the data from the various traces and the dump can not be correlated.

Situation	NFS DEBUGx Trace (1)	NFS Client Trace	TCP/IP CTRACE + PACKET Trace (2)	OMVS CTRACE (2)	HFS CTRACE (2)	AIX IP Trace or Sun Snoop Trace	z/OS Dump
Abend	Debug 4						X
NFS Server Functional Error	Debug 4
NFSS/HFS Function Error	Debug 9				X
NFS Client Error		X
Communication Problem	Debug 4		X	X		X
'Hang' Problem	Debug 4		X	X		X
‘Mount’ Problem	Debug 4	X

Notes:
(1) Debug 9 is always preferred, but if that is not possible then the cited Debug level is usually a reasonable compromise.
(2) Unless the CTRACE is written to an external writer, a z/OS Dump must be taken to capture the CTRACE data.