Collecting support information about the cluster

The MustGather tool collects information about your cluster that is crucial for troubleshooting problems and providing support.

The MustGather tool collects support information about the IBM Cloud Paks and the clusters where they are deployed. You can use this tool to gather information that is needed and relevant to the problem before you open a case with the IBM® Support .

The MustGather tool is built on the Red Hat® OpenShift® Container Platform MustGather tool , and is automatically deployed with the IBM Cloud Pak foundational services as a part of the ibm-healthcheck-operator. However, without the ibm-healthcheck-operator you can still use the MustGather tool through the command line directly from the MustGather image.

• Using MustGather diagnostics through the command line for cluster admin
• Available data modules
  • Prerequisites
  • Quick start: Getting support information for cluster admin
  • Usage and examples
    • Available command options
    • Usage examples
• Using MustGather for non-cluster admin
  • Permission for non-cluster admin
  • Quick start guide to get support information for non-cluster admin
  • Usage and examples: non-cluster admin
    • Available command options for non-cluster admin
    • Usage examples
• Using MustGather as a part of ibm-healthcheck-operator
  • Using MustGather diagnostics from the Administration panel
  • Using MustGather diagnostics from the OpenShift cluster console

Using MustGather diagnostics through the command line for cluster admin

Important: IBM Cloud Pak foundational services version 3.18 onwards, the MustGather image for IBM Cloud Paks is available at a new location. If you use the latest tag for the MustGather image, the image from the new location is supported on all versions of foundational services.

If you need to use a specific image version, see the following notes:

• For foundational services versions 3.18 and later, use the images from icr.io/cpopen/cpfs/.
• For foundational services versions 3.17 and prior, use the images from quay.io .

All references and commands in the following sections point to the icr.io/cpopen/cpfs/ location. If your foundational services version is 3.17 or prior, and you need to use a specific image version, you can replace icr.io/cpopen/cpfs/ with quay.io/opencloudio/, where required.

Available data modules

IMPORTANT: Non-cluster admin supports Overview, Failure and Cloudpak data modules. Cluster admin supports all the available data modules.

To collect a specific type of support information only, define the data modules when you run the MustGather tool. The following table lists the details of all available modules.

Note: The default scan does not collect information about secrets as it is a user sensitive data. To collect information about secrets, you can use the dedicated data module.

Prerequisites

• Access to the cluster as a user with a cluster-admin role.
• The Red Hat® OpenShift® Container Platform CLI (oc) installed. For more information, see Installing the CLI .

• Access to the MustGather image for IBM Cloud Paks. See the previous section.

  Note: For offline environments that cannot access icr.io, the Docker image is available in the offline installation package. For more information, see Downloading the image in offline environments.

Quick start: Getting support information for cluster admin

Complete the following steps to retrieve the support information from your cluster.

1. Navigate to the directory where you want to store the MustGather data.

2. Collect the data by running the following command.

   oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:<image_version> -- gather -n common-service,ibm-common-services
   

   where the <image_version> is the version of the MustGather image that you can find at icr.io/cpopen/cpfs/.

   For example:

   oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -n common-service,ibm-common-services
   

   The collected data is compressed into a .tgz output file and stored in ./must-gather.local.<rand>.

3. Attach the .tgz output file that contains the support information when you submit the support case on the IBM Support Portal .

Usage and examples- cluster admin

Available command options

You can use the MustGather tool to collect support information about IBM Cloud Paks. You can collect all support information from your cluster, or only specified information about specified namespaces. Add options to the MustGather command to collect the data you need.

The following example shows all available command options:

  oc adm must-gather --run-namespace=<NAMESPACE_NAME> --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v

oc adm must-gather command options

Note: oc adm must-gather is a default OpenShift command. For more information, run the oc adm must-gather -h command.

gather command options:

Usage examples

The following examples show how to use the MustGather tool to collect specified support data. When you copy and run the code that is provided in the examples, the shell script is saved on the local host. You can reuse the script anytime to collect the support information.

Note: Before you run the code, edit the following parameters in the script:

• Provide a namespace where your IBM Cloud Pak is deployed in the CLOUDPAK_NAMESPACES parameter.
• For offline use, set MUST_GATHER_IMAGE to the name of your offline MustGather image.
• For online use, set the <image_version> to the version of the MustGather image that you can find at icr.io/cpopen/cpfs/.

Collecting support information with default data modules

1. Run the following command to save the must-gather-default.sh script on your local host.

   cat >must-gather-default.sh <<'EOT'
   #!/bin/bash
   export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version>
   export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES>
   export MUST_GATHER_MODULES=overview,system,failure,ocp,cloudpak
   oc adm must-gather --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES
   EOT
   
   chmod +x must-gather-default.sh
   

2. Run the following command to collect the support data.

   ./must-gather-default.sh
   

Collecting only the support information about the IBM Cloud Pak

1. Run the following command to save the must-gather-cp.sh script on your local host.

   cat >must-gather-cp.sh <<'EOT'
   #!/bin/bash
   export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version>
   export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES>
   export MUST_GATHER_MODULES=overview,cloudpak
   oc adm must-gather --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES
   EOT
   
   chmod +x must-gather-cp.sh
   

2. Run the following command to collect the support data.

   ./must-gather-cp.sh
   

Collecting logs for the resources with same labels

Prerequisite: Include the label serviceability-addon=<component-name> in all the resources.

To collect the logs only for the resources with the label serviceability-addon=<component-name> in provided namespaces, run the following command:

oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -n namespace

Using custom script to collect MustGather diagnostics that are not provided by default

To collect data that is not provided by default, run the following must-gather command with custom script:

oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -s ex

For example, if the component name is foo, run the following command:

oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c foo -s ex

From foundational services 3.17 version, you can pass the namespace to the command for custom script and use the namespace inside the custom script.

For example, if the component name is foo, and the component custom script needs to collect diagnostic specific to a namespace, you can run the following must-gather command:

oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c foo -s ex -n namespace

Creating and running the custom scripts

Before you run the custom scripts to collect the MustGather diagnostics, components need to perform the following tasks:

Note: Make sure to refer the Standard adaption guide.

1. Component creates its own custom scripts.
2. Component places the custom scripts inside the Docker image that is a part of IBM trusted registries.
3. Component provides the configmap with the image location.

After you run -c parameter with -s ex parameter, must-gather performs the following tasks:

1. must-gather fetches the image location from the component in the configmap.
2. After must-gather gets the image details from the configmap, it uses the image to run a container and copy the custom scripts.
3. must-gather validates and runs the scripts to collect the logs.

Standard adaption guide

It describes the standards that the component must follow.

The following list describes the set of rules for the Standard adaption guide:

1. Custom script standard

   When the component creates the custom script, use the following guidelines:

   • By default, the script type is bash or shell scripts.
   • If you have more than one script, name the master script as ibm-mustgather-customscript-<component-name>. A master script is a script from where the execution starts.
   • Place the scripts inside the /home/custom-scripts folder.
   • Keep the output path as ${BASE_PATH} for data collection.

   • Use only get,list, and exec commands on the resources.

     • Sample script

       oc exec -n openshift-ingress ${pod} -- cat  /lib/haproxy/haproxy.config >${BASE_PATH}/${pod}.haproxy.config &
       
       oc logs -n ${NS} ${POD} --all-containers >${BASE_PATH}/${NS}.${POD}.current.log
       

     • Sample script that uses the namespace from the must-gather command:

       if [ -n "$1" ]; then
       ns=$1
       oc get pods -n ${ns}
       

2. Dockerfile standard

   Use the following guidelines for Docker image to include custom scripts:

   1. Use the standard base image, ubi8:minimal.
   2. Install the rsync and tar packages.
   3. Create a folder /home/custom-scripts and copy the scripts in this folder. Give 777 permission to this folder.

   The following sample is of a Dockerfile:

   FROM hyc-cloud-private-edge-docker-local.artifactory.swg-devops.com/build-images/ubi8-minimal:latest
   RUN microdnf update -y && microdnf install rsync tar -y && microdnf clean all &&\
   mkdir /home/custom-scripts && chmod777 /home/custom-scripts
   COPY collection-scripts/*<custom-scripts>* /home/custom-scripts
   

3. IBM trusted registries

   Use only the following trusted registries for Docker image:

   • quay.io/opencloudio/
   • cp.icr.io/cp/icp-foundation/
   • cp.stg.icr.io/cp/icp-foundation/
   • icr.io/cpopen/

4. Configmap standard

   Use the following guidelines when the component provides configmap:

   1. Name the configmap as ibm-mustgather-customscript-<component-name> and the label as serviceability-addon=<component-name>.
   2. Set the field immutable as true in configmap.
   3. Provide the image location as key value pair, where key is the component name, value is the image location with sha256 manifest.
   4. Don't use image tags. You can use sha256 manifest.

   The following sample is of a configmap file:

   NAME : ibm-mustgather-customscript-<component-name>

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: ibm-mustgather-customscript-<component-name>
      namespace: <component specific namespace>
    labels:
      serviceability-addon:component-name
    data:
      <component-name>: icr.io/cpopen/operator@sha256:6535136...
    immutable: true
   

   Note: MustGather also supports default configmap that keeps image location of the component. However, to manage better and avoid any dependencies, MustGather recommends the component to bring its own configmap by following Standard adaption guidelines.

Using MustGather for non-cluster admin

You can use MustGather for non-cluster admin role.

Prerequisites:

• Access to the cluster as a user with non-cluster admin role. For more information, see Permissions for non-cluster admin.

• The Red Hat® OpenShift® Container Platform version 4.11 CLI is installed. For more information, see Installing the CLI.

• Access to the MustGather image for IBM Cloud Paks.

Note: A non-cluster admin can use all the must-gather commands that an admin uses. However, non-cluster admins must add the --run-namespace parameter to each command. To use the --run-namespace parameter, you need to install Red Hat OpenShift Container Platform 4.11 CLI. The 4.11 CLI also works in older versions of OpenShift clusters.

Permission for non-cluster admin

Login as a cluster admin user and assign the ClusterRole and ClusterRoleBinding roles for non-cluster admin user to collect data for modules, such as Overview, Failure, and Cloudpak.

See the following sample ClusterRole definition:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cpfs-mg-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/exec"]
  verbs: ["get", "list", "create"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]

See the following sample ClusterRoleBindingdefinition:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cpfs-mg-cluster-rolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: cpfs
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cpfs-mg-clusterrole

Collecting information for Overview module

To collect information about the Overview module, create a single YAML file with the following ClusterRole and ClusterRoleBinding definitions. Then, run the oc apply -f <file-name>.yaml command to create the ClusterRole and ClusterRoleBinding resources.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-overview-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods" ,"services", "services/finalizers", "endpoints", "persistentvolumeclaims", "events", "configmaps", "secrets", "nodes", "persistentvolumes", "resourcequotas"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses", "networkpolicies"]
  verbs: ["get", "list"]
- apiGroups: ["certmanager.k8s.io"]
  resources: ["certificates", "challenges", "clusterissuers", "issuers", "orders"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "replicasets", "statefulsets"]
  verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
  resources: ["clusterservicestatuses"]
  verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs"]
  verbs: ["get", "list"]
- apiGroups: ["operators.coreos.com"]
  resources: ["clusterserviceversions"]
  verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods" ,"nodes"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["operator.ibm.com"]
  resources: ["*"]
  verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
  resources: ["machineconfigpools"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-overview-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:{Namespace-Name}:default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-overview-clusterrole

Collecting information for failure module

To collect information about the Failure module, create a single YAML file with the following ClusterRole and ClusterRoleBinding definitions. Then, run the oc apply -f <file-name>.yaml command to create the ClusterRole and ClusterRoleBinding resources.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-failure-clusterrole
rules:
- apiGroups: [""]
  resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs", "jobs"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-failure-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:{Namespace-Name}::default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-failure-clusterrole

Collecting information for Cloud pak module

To collect information about the Cloud pak module, create a single YAML file with the following ClusterRole and ClusterRoleBinding definitions. Then, run the oc apply -f <file-name>.yaml command to create the ClusterRole and ClusterRoleBinding resources.

Sample command:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-cloudpak-clusterrole
rules:
- apiGroups: [""]
  resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints", "nodes", "secrets", "events", "configmaps", "replicationcontrollers"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses", "networkpolicies"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs", "jobs"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets", "replicasets"]
  verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
  resources: ["networks", "imagecontentsourcepolicies"]
  verbs: ["get", "list"]
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests"]
  verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
  resources: ["imagecontentsourcepolicies", "networks"]
  verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
  resources: ["machineconfigpools", "machineconfigs"]
  verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
  resources: ["volumeattachments"]
  verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
  resources: ["clusterservicestatuses"]
  verbs: ["get", "list"]
- apiGroups: ["monitoring.coreos.com"]
  resources: ["servicemonitors"]
  verbs: ["get", "list"]
- apiGroups: ["policy"]
  resources: ["endpointslices", "poddisruptionbudgets"]
  verbs: ["get", "list"]
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list"]
- apiGroups: ["build.openshift.io"]
  resources: ["buildconfigs", "builds"]
  verbs: ["get", "list"]
- apiGroups: ["apps.openshift.io"]
  resources: ["deploymentconfigs"]
  verbs: ["get", "list"]
- apiGroups: ["image.openshift.io"]
  resources: ["imagestreams"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-cloudpak-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:{Namespace-Name}::default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-cloudpak-clusterrole

Example procedure

The following example includes definitions of all resources that are required to enable non-cluster admins to run the must-gather commands. To create the resources, copy and paste the contents at the CLI or to the YAML editor in your Red Hat OpenShift Container Platform console.

cat <<EOF | tee >(oc apply -f -) | cat >/dev/null
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cpfs-mg-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/exec"]
  verbs: ["get", "list", "create"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cpfs-mg-cluster-rolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: cpfs
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cpfs-mg-clusterrole

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-overview-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods" ,"services", "services/finalizers", "endpoints", "persistentvolumeclaims", "events", "configmaps", "secrets", "nodes", "persistentvolumes", "resourcequotas"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses", "networkpolicies"]
  verbs: ["get", "list"]
- apiGroups: ["certmanager.k8s.io"]
  resources: ["certificates", "challenges", "clusterissuers", "issuers", "orders"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "replicasets", "statefulsets"]
  verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
  resources: ["clusterservicestatuses"]
  verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs"]
  verbs: ["get", "list"]
- apiGroups: ["operators.coreos.com"]
  resources: ["clusterserviceversions"]
  verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["operator.ibm.com"]
  resources: ["*"]
  verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
  resources: ["machineconfigpools"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-overview-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:ibm-common-services:default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-overview-clusterrole

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-failure-clusterrole
rules:
- apiGroups: [""]
  resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs", "jobs"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-failure-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:ibm-common-services:default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-failure-clusterrole

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-cloudpak-clusterrole
rules:
- apiGroups: [""]
  resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints", "nodes", "secrets", "events", "configmaps", "replicationcontrollers"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses", "networkpolicies"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs", "jobs"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets", "replicasets"]
  verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
  resources: ["networks", "imagecontentsourcepolicies"]
  verbs: ["get", "list"]
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests"]
  verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
  resources: ["imagecontentsourcepolicies", "networks"]
  verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
  resources: ["machineconfigpools", "machineconfigs"]
  verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
  resources: ["volumeattachments"]
  verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
  resources: ["clusterservicestatuses"]
  verbs: ["get", "list"]
- apiGroups: ["monitoring.coreos.com"]
  resources: ["servicemonitors"]
  verbs: ["get", "list"]
- apiGroups: ["policy"]
  resources: ["endpointslices", "poddisruptionbudgets"]
  verbs: ["get", "list"]
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list"]
- apiGroups: ["build.openshift.io"]
  resources: ["buildconfigs", "builds"]
  verbs: ["get", "list"]
- apiGroups: ["apps.openshift.io"]
  resources: ["deploymentconfigs"]
  verbs: ["get", "list"]
- apiGroups: ["image.openshift.io"]
  resources: ["imagestreams"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-cloudpak-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:ibm-common-services:default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-cloudpak-clusterrole

EOF

Quick start guide to get support information for non-cluster admin

1. Navigate to the directory where you want to store the MustGather data.

2. Collect the data by running the following command.

     oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=icr.io/cpopen/cpfs/must-gather:<image_version> -- gather -n {NAMESPACE-NAME}
   

   Where the is the version of the MustGather image that you can find at icr.io/cpopen/cpfs/.

   For example:

     oc adm must-gather --run-namespace=ibm-common-services --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -n ibm-common-services
   

   The collected data is compressed into a .tgz output file and stored in ./must-gather.local...

3. Attach the .tgz output file that contains the support information when you submit the support case on the IBM Support Portal.

Usage and examples: non-cluster admin

Available command options for non-cluster admin

You can use the MustGather tool to collect support information about IBM Cloud Paks. You can collect all support information from your cluster for non-cluster admin, or only specified information about specified namespaces. Add options to the MustGather command to collect the data you need.

The following example shows all available command options:

  oc adm must-gather --run-namespace=<NAMESPACE_NAME> --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v

Note: To know more about oc adm must-gather command options and gather command options, see oc adm must-gather command options and gather command options.

Usage examples: non-cluster admin

The following examples show how to use the MustGather tool to collect specified support data for non-cluster admin. When you copy and run the code that is provided in the examples, the shell script is saved on the local host. You can reuse the script anytime to collect the support information.

oc adm must-gather --run-namespace=<NAMESPACE_NAME> --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v

Where NAMESPACE_NAME parameter is the namespace name that belongs to the non-cluster admin and IBM Cloud Pak_NAMESPACES parameter is one or multiple namespaces from where the user collects to mustgather information.

For example, a non-cluster admin is assigned to ibm-common-services namespace and that non-cluster admin wants to collect mustgather logs from different namespaces, such as, ibm-common-services, iaf-demo for the Overview module, then the sample command might resemble the following command:

oc adm must-gather --run-namespace=ibm-common-services --image=icr.io/cpopen/cpfs/must-gather:4.6.14 -- gather -m overview -n ibm-common-services,iaf-demo

Note: Before you run the code, edit the following parameters in the script:

• Provide a namespace where your IBM Cloud Pak is deployed in the CLOUDPAK_NAMESPACES parameter.
• For offline use, set MUST_GATHER_IMAGE to the name of your offline MustGather image.
• For online use, set the <image_version> to the version of the MustGather image that you can find at icr.io/cpopen/cpfs/.

Note: Non-cluster admin supports Overview, Failure and Cloudpak data modules.

Collecting support information with default data modules

1. Run the following command to save the must-gather-default.sh script on your local host.

   cat >must-gather-default.sh <<'EOT'
   #!/bin/bash
   export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version>
   export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES>
   export MUST_GATHER_MODULES=overview,failure,cloudpak
   oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES
   EOT
   
   chmod +x must-gather-default.sh
   

2. Run the following command to collect the support data.

   ./must-gather-default.sh
   

Collecting only the support information about the IBM Cloud Pak

1. Run the following command to save the must-gather-cp.sh script on your local host.

   cat >must-gather-cp.sh <<'EOT'
   #!/bin/bash
   export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version>
   export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES>
   export MUST_GATHER_MODULES=overview,cloudpak
   oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES
   EOT
   
   chmod +x must-gather-cp.sh
   

2. Run the following command to collect the support data.

   ./must-gather-cp.sh
   

Collecting logs for the resources with same labels

Prerequisite: Include the label serviceability-addon=<component-name> in all the resources.

To collect the logs only for the resources with the label serviceability-addon=<component-name> in provided namespaces, run the following command:

oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -n namespace

Using MustGather as a part of ibm-healthcheck-operator

If ibm-healthcheck-operator is installed as a part of your product, you might run MustGather diagnostics from the OpenShift cluster console and Administration panel.

Using MustGather diagnostics from the OpenShift cluster console

Note: You can use MustGather diagnostics from the OpenShift cluster console only if you have ibm-healthcheck-operator deployed.

Complete the following actions to gather diagnostics from the OpenShift cluster console.

1. Log in to your OpenShift cluster console.
2. Go to Operators > Installed Operators.
3. Set the project to All Projects.
4. Find and select the ibm-healthcheck-operator.
5. Go to the IBM MustGather Jobs tab, select Create MustGather Job.
6. To run a default diagnostic scan, click Create.

7. Go to Workloads > Jobs. Find your MustGather job. The URL for downloading the results package is available in the job log, for example:

   INFO: must-gather-1600137793.tar.gz is ready. You can download it from dashboard UI or using curl -k -X GET "https://<CLUSTER_URL>/must-gather/v1alpha1/mustgather/download/must-gather-1600137793.tar.gz" -H "accept: application/json" -H "Authorization: <BEARER_TOKEN>"

Using MustGather diagnostics from the Administration panel

Note: You can use MustGather diagnostics from the Administration panel only if yo have ibm-healthcheck-operator deployed.

To run MustGather diagnostics with the console, you must use the Administration panel. This dashboard is only accessibly by Cluster Administrator users.

To run MustGather diagnostics from the Administration panel, complete the following actions:

1. Log in as a Cluster Administrative user and from the main navigation menu click Home to open the Administration panel.
2. From either the Welcome widget or the Diagnostics summary card, click Gather diagnostics. A side-panel opens where you can configure the diagnostics to gather and run the report generation.
3. Select the diagnostics that you would like to gather. You can select more than one data module.
4. If you selected to gather Cloud Pak data, provide a namespace or namespaces from which you want to collect data.
5. Click Gather diagnostics. Collecting support information can take from a few to 30 minutes.
6. Within the System overview section of the main work area, view the Diagnostics summary card, the status on the card is Report generating in progress. After the gathering process completes, the report can be downloaded.

7. To download the package with results, click the download link next to the package name on the summary card.

   Note: To view the support information about the cluster upgrade, go to Overview > ClusterUpgrade in the downloaded directory.