Reviewing cluster message log files
The PowerHA® SystemMirror® software writes the messages it generates to the system console and to several log files. Each log file contains a different subset of messages generated by the PowerHA SystemMirror software. When viewed as a group, the log files provide a detailed view of all cluster activity.
The following list describes the log files into which the PowerHA SystemMirror software writes messages and the types of cluster messages they contain. The list also provides recommendations for using the different log files. Note that the default log directories are listed here; you have the option of redirecting some log files to a chosen directory. If you have redirected any logs, check the appropriate location.
| Log file name | Description |
|---|---|
| system error log | Contains time-stamped, formatted messages from all AIX® subsystems, including scripts
and daemons. For information about viewing this log file and interpreting
the messages it contains, see the section Understanding the system
error log. Recommended Use: Because the system error log contains time-stamped messages from many other system components, it is a good place to correlate cluster events with system events. |
| /tmp/clconvert.log | Contains a record of the conversion progress when upgrading
to a recent PowerHA SystemMirror release.
The installation process runs the cl_convert utility and creates
the /tmp/clconvert.log file. Recommended Use: View the clconvert.log to gauge conversion success when running cl_convert from the command line. |
| /var/ha/log/grpglsm | Contains time-stamped messages in ASCII format. These track the execution of internal activities of the RSCT Group Services Globalized Switch Membership daemon. IBM® support personnel use this information for troubleshooting. The file gets trimmed regularly. Therefore, save it promptly if there is a chance you may need it. |
| /var/hacmp/adm/cluster.log | Contains time-stamped, formatted messages generated by PowerHA SystemMirror scripts and
daemons. Recommended Use: Because this log file provides a high-level view of current cluster status, check this file first when diagnosing a cluster problem. |
| /var/hacmp/adm/history/cluster.mmddyyyy | Contains time-stamped, formatted messages generated by PowerHA SystemMirror scripts. The
system creates a cluster history file every day, identifying each
file by its file name extension, where mm indicates the month, dd indicates
the day, and yyyy the year. For information about viewing this
log file and interpreting its messages, see the section Understanding
the cluster history log file. Recommended Use: Use the cluster history log files to get an extended view of cluster behavior over time. Note that this log is not a good tool for tracking resource groups processed in parallel. In parallel processing, certain steps formerly run as separate events are now processed differently and these steps will not be evident in the cluster history log. Use the hacmp.out file to track parallel processing activity. |
| /var/log/clcomd/clcomddiag.log | Contains time-stamped, formatted, diagnostic messages generated
by clcomd. Recommended Use: Information in this file is for IBM Support personnel. |
| /var/hacmp/log/autoverify.log | Contains any warnings or errors that occurred during the automatic cluster verification run. |
| /var/hacmp/log/clavan.log | Contains the state transitions of applications managed by PowerHA SystemMirror. For example,
when each application managed by PowerHA SystemMirror is started
or stopped and when the node stops on which an application is running.
Each node has its own instance of the file. Each record in the clavan.log file consists of a single line. Each line contains a fixed portion and a variable portion: Recommended Use: By collecting the records in the clavan.log file from every node in the cluster, a utility program can determine how long each application has been up, as well as compute other statistics describing application availability time. |
| /var/hacmp/log/clinfo.log /var/hacmp/log/clinfo.log.n, n=1,..,7 |
The clinfo.log file records the output generated by
the event scripts as they run. This information supplements and expands
upon the information in the /var/hacmp/log/hacmp.out file.
You can install Client Information (Clinfo) services on both client and server systems - client systems (cluster.es.client) will not have any HACMP ODMs (for example HACMPlogs) or utilities (for example clcycle); therefore, the Clinfo logging will not take advantage of cycling or redirection. The default debug level is 0 or "off". You can enable logging using command line flags. Use the clinfo -l flag to change the log file name. |
| /var/hacmp/log/clstrmgr.debug /var/hacmp/log/clstrmgr.debug.n, n=1,..,7 |
Contains time-stamped, formatted messages generated by the clstrmgrES subsystem.
The default messages are verbose and are typically adequate for troubleshooting
most problems, however IBM support
may direct you to enable additional debugging. Recommended Use: Information in this file is for IBM Support personnel. |
| /var/hacmp/log/clstrmgr.debug.long /var/hacmp/log/clstrmgr.debug.long.n, n=1,..,7 |
Contains high-level logging of cluster manager activity, in
particular its interaction with other components of PowerHA SystemMirror and with RSCT,
which event is currently being run, and information about resource
groups (for example, their state and actions to be performed, such
as acquiring or releasing them during an event. Recommended Use: Information in this file is for IBM Support personnel. |
| /var/hacmp/log/cspoc.log | Contains time-stamped, formatted messages generated by PowerHA SystemMirror C-SPOC commands.
The cspoc.log file resides on the node that invokes the C-SPOC
command. Recommended Use: Use the C-SPOC log file when tracing a C-SPOC command's execution on cluster nodes. |
| /var/hacmp/log/cspoc.log.long | Contains a high-level of logging for the C-SPOC utility - commands and utilities that have been invoked by C-SPOC on specified nodes and their return status. |
| /var/hacmp/log/cspoc.log.remote | Contains logging of the execution of C-SPOC commands on remote nodes with ksh option xtrace enabled (set -x). |
| /var/hacmp/log/hacmp.out /var/hacmp/log/hacmp.out.n n=1,..,7 | Contains time-stamped, formatted messages generated by PowerHA SystemMirror scripts on
the current day. In verbose mode (recommended), this log file contains a line-by-line record of every command executed by scripts, including the values of all arguments to each command. An event summary of each high-level event is included at the end of each event's details. For information about viewing this log and interpreting its messages, see the section Understanding the hacmp.out log file. Recommended Use: Because the information in this log file supplements and expands upon the information in the /var/hacmp/adm/cluster.log file, it is the primary source of information when investigating a problem. |
| /var/hacmp/log/oraclesa.log | Contains information about any Oracle specific errors that occur when using this Smart Assist and is used by the Oracle Smart Assist. |
| /var/hacmp/log/sa.log | Contains information about any general errors that occur when using the Smart Assists and is used by the Smart Assist infrastructure. |
| /var/log/clcomd/clcomddiag.log | Contains time-stamped, formatted messages generated by Cluster
Communications daemon (clcomd) activity. The log shows information
about incoming and outgoing connections, both successful and unsuccessful.
Also displays a warning if the file permissions for /usr/es/sbin/cluster/etc/rhosts are
not set correctly - users on the system should not be able to write
to the file. Recommended Use: Use this file to troubleshoot communication problems of PowerHA SystemMirror utilities. |
| /var/hacmp/log/clconfigassist.log | Contains debugging information for the Two-Node Cluster Configuration Assistant. The Assistant stores up to ten copies of the numbered log files to assist with troubleshooting activities. |
| /var/hacmp/clverify/clverify.log | The clverify.log file contains the verbose messages output by the cluster verification utility. The messages indicate the node(s), devices, command, etc. in which any verification error occurred. |
| /var/hacmp/log/clutils.log | Contains information about the date, time, results, and which
node performed an automatic cluster configuration verification. It also contains information for the file collection utility, the two-node cluster configuration assistant, and the cluster test tool. |
| /var/hacmp/log/cl_testtool.log | Includes excerpts from the hacmp.out file. The Cluster Test Tool saves up to three log files and numbers them so that you can compare the results of different cluster tests. The tool also rotates the files with the oldest file being overwritten |