Debug data collection

IBM Spectrum Scale Container Storage Interface driver provides the spectrum-scale-driver-snap.sh tool to collect the debug data. This tool gathers the state of required Kubernetes resources like nodes, pods, service accounts, and so on and collects Deployment and DaemonSet logs from all nodes. It collects definition of resources in the given namespace with the label, product=ibm-spectrum-scale-csi. The collected logs are stored in the given output directory.

Issue the following command and download the tool for CSI 2.7.0: 
curl -O https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.7.0/tools/spectrum-scale-driver-snap.sh

Usage of the tool

spectrum-scale-driver-snap.sh [-n namespace] [-o output-dir] [-h]

-n: Debug data for CSI resources under this namespace will be collected. If not specified, default namespace is used. The tool returns error if CSI is not running under the given namespace.
-o: Output directory where debug data will be stored. If not specified, the debug data is stored in current directory.
-h: Prints the usage

The resultant folder contains the following files with debug information:

  • ibm-spectrum-scale-csi-attacher.log
  • ibm-spectrum-scale-csi-attacher
  • ibm-spectrum-scale-csi-provisioner.log
  • ibm-spectrum-scale-csi-provisioner
  • ibm-spectrum-scale-csi-resizer.log
  • ibm-spectrum-scale-csi-resizer
  • ibm-spectrum-scale-csi-snapshotter.log
  • ibm-spectrum-scale-csi-snapshotter
  • ibm-spectrum-scale-csi-describe-CSIScaleOperator
  • ibm-spectrum-scale-csi-attacher-0.log
  • ibm-spectrum-scale-csi-attacher-0-previous.log
  • ibm-spectrum-scale-csi-attacher-1.log
  • ibm-spectrum-scale-csi-attacher-1-previous.log
  • ibm-spectrum-scale-csi-operator-xxxxxxxxxxxx-xxxxx.log
  • ibm-spectrum-scale-csi-operator-xxxxxxxxxxxx-xxxxx-previous.log
  • ibm-spectrum-scale-csi-operator-xxxxxxxxxxxx-xxxxx.log
  • ibm-spectrum-scale-csi-operator-xxxxxxxxxxxx-xxxxx-previous.log
  • ibm-spectrum-scale-csi-provisioner-0.log
  • ibm-spectrum-scale-csi-provisioner-0-previous.log
  • ibm-spectrum-scale-csi-resizer-0.log
  • ibm-spectrum-scale-csi-resizer-0-previous.log
  • ibm-spectrum-scale-csi-xxxxx.log
  • ibm-spectrum-scale-csi-xxxxx-previous.log
  • ibm-spectrum-scale-csi-snapshotter-0.log
  • ibm-spectrum-scale-csi-snapshotter-0-previous.log
  • ibm-spectrum-scale-csi-xxxxx.log
  • ibm-spectrum-scale-csi-xxxxx-previous.log
  • ibm-spectrum-scale-csi-describe-all-by-label
  • ibm-spectrum-scale-csi-get-all-by-label
  • ibm-spectrum-scale-csi-configmap
  • ibm-spectrum-scale-csi-k8snodes
  • ibm-spectrum-scale-csi-daemonsets
  • ibm-spectrum-scale-csi-versions
  • nodes.json
In addition to these logs, the following details are necessary for troubleshooting:
  • Include the following details if an issue is related to PVC.
    kubectl describe namespace < namespace name >
kubectl get pvc < pvc name > -o yaml
kubectl describe pvc < pvc name >
kubectl describe pv < pv name from above pvc >
kubectl describe StorageClass < storage class name for pvc >
  • Include the following details in addition to above if an issue is related to a pod.
    kubectl describe pod < pod name >
kubectl get pod < pod name > -o yaml
kubectl describe VolumeAttachment < volume attachment name >
  • Include the following details in addition to above if an issue is related to a snapshot.
    kubectl describe VolumeSnapshot < snapshot name >
kubectl get VolumeSnapshot < snapshot name > -o yaml
kubectl describe VolumeSnapshotContent <snapshot content name >
kubectl describe VolumeSnapshotClass < snapshot name >