Understanding call home

The details are collected from individual nodes that are marked as call home child nodes in the cluster. The details from each child node are collected by the call home node. You need to create a call home group by grouping call home child nodes. One of the nodes in the group is configured as the call home node and it performs data collection and upload.

The data gathering and upload can be configured individually on each group. Use the groups to reflect logical units in the cluster. For example, it is easier to manage when you create a group for all CES nodes and another group for all non-CES nodes. Call home groups help to distribute the data-gather and data-upload workload to prevent bottlenecks. It is recommended to limit the group size as 32. Larger groups are also possible but it might result in performance issues.

The following figure provides basic structure of call home functions.
Figure 1. Call home architecture
One call home node is defined here by using two call home child nodes and one call home server. The collected data is transferred to the IBM server.
The following are the descriptions of the important terminologies associated with the call home feature:
Call home group
A group of nodes configured by using the mmcallhome group command. A call home group consists of one call home node and multiple call home child nodes. Multiple call home groups can be configured within a GPFS™ cluster. You can automate the call home group creation by using the mmcallhome group auto [options] command.
Call home node
This node initiates the data collection within the call home group and uploads the data package to IBM® support center. A gather-send task process that runs on the call home node collects data from the child nodes and upload the data to the configured server. The gather-send configuration file includes information about what needs to be collected from the child nodes.
Call home child node
A node that the call home node collects data from. The call home node is also determined as its child node.
Start of changeThe call home feature can be configured in two ways:
  • Configuring a call home proxy and then creating the call home group:
    1. Configure the call home feature by mmcallhome proxy and mmcallhome info commands.
    2. Create the call home group by mmcallhome group auto command.
    So, in this scenario the mmcallhome group auto command creates the group and copies the global configuration to each group's local configuration. This approach might be used if you want to configure all groups homogeneously.
  • Creating a call home group and then configuring the call home proxy:
    1. Create the call home group by mmcallhome group auto command, while the global configuration is empty
    2. Perform the local call home configuration for each node individually by mmcallhome proxy and mmcallhome info commands.
    This approach might be used if you want to fine tune the configuration on each group, for the scenarios where one group is for CES nodes and the other one for AIX nodes.
End of change
The call home functionality provides the following data upload schemes to collect and upload the data:
  1. File upload: Any file can be specified for upload.
  2. Package upload: Collects predefined data package regularly. The call home feature provides weekly.conf schema to collect the package weekly and daily.conf schema to collect the package daily. These gather schemas are located at: /usr/lpp/mmfs/data/callhome/gather.d. After the upload, the data packages are stored in the data package directory for backup.
If the data collection is specified weekly, Cron is started once a week and data from call home child node is gathered by the call home node as specified in the weekly.conf file. When the gather task is finished, the data is uploaded from the call home node to the IBM Support. The following commands are issued internally to generate the data that needs to be shared with IBM Support:
  • tail -n 10000 /var/log/messages
  • mmdiag --version
  • mmdiag --config
  • mmdiag --rpc
  • mmlsnsd -L
  • mmlsnsd -X
  • mmgetstate -a
  • For each file system: mmlsdisk <fs> -L
  • For each file system: mmdf <fs>
  • callhomeutils.pl info list
  • mmces service list -a
  • curl -i http://localhost:8080/info -X GET
  • mmuserauth service list
  • /usr/bin/curl -g -i -X GET http://localhost:35357/v3/projects
If the data collection is specified daily, Cron is started once every day and data from call home child node is gathered by the call home node as specified in the daily.conf file. When the gather task is finished, the data is uploaded from the call home node to the IBM Support. The following commands are issued internally to generate the data that needs to be shared with IBM Support:
  • uname -a
  • hostname
  • sysctl -a
  • uptime
  • fdisk -l
  • df, df -i, and df -k
  • ifconfig -a
  • lspci
  • lsmod
  • dmidecode
  • cat /proc/device-tree/system-id
  • ppc64_cpu --smt;ppc64_cpu --cores-present;ppc64_cpu --cores-on
  • cat /proc/cpuinfo
  • cat /proc/meminfo
  • tail -n 10000 /var/adm/ras/mmfs.log.latest
  • tail -n 10000 /var/log/messages
  • mmdiag --rpc
  • mmlscluster
  • mmlsnode -a
  • mmlsmgr
  • mmlsnsd -L
  • mmlscallback
  • mmgetstate -a
  • mmremotecluster show all
  • mmremotefs show all
  • mmauth show
  • tsstatus
  • mmlsfs all
  • For each file system: mmlsfs <fs>
  • For each file system: mmlsfs <fs> --snc
  • For each file system: mmlsdisk <fs> -L
  • For each file system: mmlspolicy <fs> -L
  • For each file system: mmlsfileset <fs> -L
  • For each file system: mmlsfileset <fs> --afm -L
  • For each file system: mmlssnapshot <fs>
  • For each file system: mmdf <fs>
  • mmlsnode -N waiters -L
  • callhomeutils.pl info list
When an event occurs and if the system is configured to upload data to IBM Support based on that event, the call home node that is configured for capturing the details on events collects the data, creates a file, and uses the mmcallhome command to upload the data. The following events can be configured to start the relevant data collection and upload:
  • nfsd_down
  • ctdb_down
  • ctdb_state_down
  • smbd_down

To configure the call home feature, see Configuring call home to collect details.