Collecting support information about the cluster

The MustGather tool collects information about your cluster that is crucial for troubleshooting problems and providing support.

The MustGather tool collects support information about the IBM Cloud Paks and the clusters where they are deployed. You can use this tool to gather information that is needed and relevant to the problem before you open a case with the IBM® Support .

The MustGather tool is built on the OpenShift MustGather tool .

• Using MustGather diagnostics through the command line for cluster admin
• Available data modules
  • Prerequisites
  • Quick start: Getting support information for cluster admin
  • Usage and examples
    • Available command options
    • Usage examples
• Using MustGather for non-cluster admin
  • Permission for non-cluster admin
  • Quick start guide to get support information for non-cluster admin
  • Usage and examples: non-cluster admin
    • Available command options for non-cluster admin
    • Usage examples
• Using MustGather diagnostics on a multiarchitecture cluster

Using MustGather diagnostics through the command line for cluster admin

Important: IBM Cloud Pak foundational services version 3.18 onwards, the MustGather image for IBM Cloud Paks is available at a new location. If you use the latest tag for the MustGather image, the image from the new location is supported on all versions of foundational services.

If you need to use a specific image version, see the following notes:

• For foundational services versions 3.18 and later, use the images from icr.io/cpopen/cpfs/.
• For foundational services versions 3.17 and prior, use the images from quay.io .

All references and commands in the following sections point to the icr.io/cpopen/cpfs/ location. If your foundational services version is 3.17 or prior, and you need to use a specific image version, you can replace icr.io/cpopen/cpfs/ with quay.io/opencloudio/, where required.

Available data modules

IMPORTANT: Non-cluster admin supports Overview, Failure and Cloudpak data modules. Cluster admin supports all the available data modules.

To collect a specific type of support information only, define the data modules when you run the MustGather tool. The following table lists the details of all available modules.

Note: The default scan does not collect information about secrets as it is a user sensitive data. To collect information about secrets, you can use the dedicated data module.

Prerequisites

• Access to the cluster as a user with a cluster-admin role.
• The OpenShift Container Platform CLI (oc) installed. For more information, see Installing the CLI .

• Access to the MustGather image for IBM Cloud Paks. See the previous section.

  Note: For offline environments that cannot access icr.io, the Docker image is available in the offline installation package. For more information, see Downloading the image in offline environments.

Quick start

Complete the following steps to retrieve the support information from your cluster.

1. Navigate to the directory where you want to store the MustGather data.

2. Collect the data by running the following command. The <image_version> is the version of the MustGather image that you can find at icr.io/cpopen/cpfs/.

    oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:<image_version> -- gather -n common-service,<your-foundational-services-namespace>
   

   For example:

    oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -n common-service,<your-foundational-services-namespace>
   

The collected data is compressed into a .tgz output file and stored in ./must-gather.local.<rand>.

1. Attach the .tgz output file that contains the support information when you submit the support case on the IBM Support Portal .

Usage and examples- cluster admin

Available command options

You can use the MustGather tool to collect support information about IBM Cloud Paks. You can collect all support information from your cluster, or only specified information about specified namespaces. Add options to the MustGather command to collect the data you need.

The following example shows all available command options:

oc adm must-gather --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v

oc adm must-gather command options

Note: oc adm must-gather is a default OpenShift command. Run the oc adm must-gather -h command for more information.

gather command options:

Option|Description |---|---| | -m | Define the data module that specifies what type of information is collected. For more information, see Available data modules. When this option is not set, the MustGather tool collects information that includes the following data modules: overview, system, failure, ocp, cloudpak. | -n
--namespaces= | Specify the IBM Cloud Pak namespace or namespaces from which the data is collected for the cloudpacks and secrets data modules. By default, the namespace is set to foundational-services. Separate each namespace with a comma, for example <your-foundational-services-namespace>,common-service. | | -h | Display the help message.| | -v | Display the version.| | -l | Specify the IBM Cloud Pak labels to collect logs. Separate the labels with a comma. This option requires you to set the -m cloudpak at the same time. By default, no labels are defined.| | -c
Available for IBM Cloud Pak foundational services version 3.11.0 and later. | Specify the component name to collect logs. If you pass the component-name followed by -c parameter, must-gather collects the MustGather diagnostics with the label serviceability-addon:<component-name>. Components pre-define this label for all of their resources. If you add namespaces -n parameter followed by -c parameter, it restricts the collection scope to the namespace of -n parameter. If you don't pass the namespaces, by default, the namespace is set to common-services and foundational-services. By using -c parameter, you cannot collect the MustGather diagnostics for multiple components with single execution. If you want to collect the MustGather diagnostics for different components, run the separate must-gather command with other component name. For more information, see Collecting logs for the resources with same labels. | | -s ex
Available for IBM Cloud Pak foundational services version 3.11.0 and later. | Collects MustGather diagnostics information by using custom script that is not the part of a must-gather image. The custom script is placed inside the Docker image that needs to be hosted in IBM trusted registries. To use this feature, pass the component name of -c parameter followed by -s ex parameter. For more information, see Using custom script to collect MustGather diagnostics that are not provided by default. |

Usage examples

The following examples show how to use the MustGather tool to collect specified support data. When you copy and run the code that is provided in the examples, the shell script is saved on the local host. You can reuse the script anytime to collect the support information.

Note: Before you run the code, edit the following parameters in the script:

• Provide a namespace where your IBM Cloud Pak is deployed in the CLOUDPAK_NAMESPACES parameter.
• For offline use, set MUST_GATHER_IMAGE to the name of your offline MustGather image.
• For online use, set the <image_version> to the version of the MustGather image that you can find at icr.io/cpopen/cpfs/.

Collecting support information with default data modules

1. Run the following command to save the must-gather-default.sh script on your local host.

    cat >must-gather-default.sh <<'EOT'
    #!/bin/bash
    export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version>
    export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES>
    export MUST_GATHER_MODULES=overview,system,failure,ocp,cloudpak
    oc adm must-gather --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES
    EOT
   
    chmod +x must-gather-default.sh
   

2. Run the following command to collect the support data.

    ./must-gather-default.sh
   

   Note: To collect customized diagnostic information about your IBM Cloud Pak and foundational services deployments in your cluster, see Collecting IBM Cloud Pak foundational services information for problem determination.

Collecting only the support information about the IBM Cloud Pak

1. Run the following command to save the must-gather-cp.sh script on your local host.

    cat >must-gather-cp.sh <<'EOT'
    #!/bin/bash
    export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version>
    export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES>
    export MUST_GATHER_MODULES=overview,cloudpak
    oc adm must-gather --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES
    EOT
   
    chmod +x must-gather-cp.sh
   

2. Run the following command to collect the support data.

    ./must-gather-cp.sh
   

Collecting logs for the resources with same labels

Prerequisite: Include the label serviceability-addon=<component-name> in all the resources.

To collect the logs only for the resources with the label serviceability-addon=<component-name> in provided namespaces, run the following command:

oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -n namespace

Using custom script to collect MustGather diagnostics that are not provided by default

To collect data that is not provided by default, run the following must-gather command with custom script:

oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -s ex

For example, if the component name is foo, run the following command:

oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c foo -s ex

From foundational services 3.17 version, you can pass the namespace to the command for custom script and use the namespace inside the custom script.

For example, if the component name is foo, and the component custom script needs to collect diagnostic specific to a namespace, you can run the following must-gather command:

oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c foo -s ex -n namespace

Creating and running the custom scripts

Before you run the custom scripts to collect the MustGather diagnostics, components need to perform the following tasks:

Note: Make sure to refer the Standard adoption guide.

1. Component creates its own custom scripts.
2. Component places the custom scripts inside the Docker image that is a part of IBM trusted registries.
3. Component provides the configmap with the image location.

After you run -c parameter with -s ex parameter, must-gather performs the following tasks:

1. must-gather fetches the image location from the component in the configmap.
2. After must-gather gets the image details from the configmap, it uses the image to run a container and copy the custom scripts.
3. must-gather validates and runs the scripts to collect the logs.

Components must follow the standard adoption guide, which includes the following set of rules:

Custom script standard

When the component creates the custom script, use the following guidelines:

• By default, the script type is bash or shell scripts.
• If you have more than one script, name the master script as ibm-mustgather-customscript-<component-name>. A master script is a script from where the execution starts.
  • Place the scripts inside the /home/custom-scripts folder.
  • Keep the output path as ${BASE_PATH} for data collection.
  • Use only get,list, and exec commands on the resources.

See the following sample script:

oc exec -n openshift-ingress ${pod} -- cat  /lib/haproxy/haproxy.config >${BASE_PATH}/${pod}.haproxy.config &

oc logs -n ${NS} ${POD} --all-containers >${BASE_PATH}/${NS}.${POD}.current.log

See the following sample script that uses the namespace from the must-gather command:

if [ -n "$1" ]; then
ns=$1
oc get pods -n ${ns}

Dockerfile standard

Use the following guidelines for Docker image to include custom scripts:

• Use the standard base image, ubi8:minimal.
• Install the rsync and tar packages.
• Create a folder /home/custom-scripts and copy the scripts in this folder. Give 777 permission to this folder.

The following sample is of a Dockerfile:

FROM hyc-cloud-private-edge-docker-local.artifactory.swg-devops.com/build-images/ubi8-minimal:latest
RUN microdnf update -y && microdnf install rsync tar -y && microdnf clean all &&\
mkdir /home/custom-scripts && chmod777 /home/custom-scripts
COPY collection-scripts/*<custom-scripts>* /home/custom-scripts

IBM trusted registries

Use only the following trusted registries for Docker image:

• quay.io/opencloudio/
• cp.icr.io/cp/icp-foundation/
• cp.stg.icr.io/cp/icp-foundation/
• icr.io/cpopen/

Configmap standard

Use the following guidelines when the component provides configmap:

• Name the configmap as ibm-mustgather-customscript-<component-name> and the label as serviceability-addon=<component-name>.
• Set the field immutable as true in configmap.
• Provide the image location as key value pair, where key is the component name, value is the image location with sha256 manifest.
• Don't use image tags. You can use sha256 manifest.

The following sample is of a configmap file with name ibm-mustgather-customscript-<component-name>:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ibm-mustgather-customscript-<component-name>
  namespace: <component specific namespace>
labels:
  serviceability-addon:component-name
data:
  <component-name>: icr.io/cpopen/operator@sha256:6535136...
immutable: true

Note: MustGather also supports default configmap that keeps image location of the component. However, to manage better and avoid any dependencies, MustGather recommends the component to bring its own configmap by following Standard adaption guidelines.

Using MustGather for non-cluster admin

You can use MustGather for non-cluster admin role.

Prerequisites:

• Access to the cluster as a user with non-cluster admin role. For more information, see Permissions for non-cluster admin.

• The OpenShift Container Platform version 4.11 CLI is installed. For more information, see Installing the CLI.

• Access to the MustGather image for IBM Cloud Paks.

Note: A non-cluster admin can use all the must-gather commands that an admin uses. However, non-cluster admins must add the --run-namespace parameter to each command. To use the --run-namespace parameter, you need to install Red Hat OpenShift Container Platform 4.11 CLI. The 4.11 CLI also works in older versions of OpenShift clusters.

Permission for non-cluster admin

Login as a cluster admin user and assign the ClusterRole and ClusterRoleBinding roles for non-cluster admin user to collect data for modules, such as Overview, Failure, and Cloudpak.

See the following sample ClusterRole definition:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cpfs-mg-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/exec"]
  verbs: ["get", "list", "create"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]

See the following sample ClusterRoleBindingdefinition:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cpfs-mg-cluster-rolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: cpfs
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cpfs-mg-clusterrole

Collecting information for Overview module

To collect information about the Overview module, create a single YAML file with the following ClusterRole and ClusterRoleBinding definitions. Then, run the oc apply -f <file-name>.yaml command to create the ClusterRole and ClusterRoleBinding resources.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-overview-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods" ,"services", "services/finalizers", "endpoints", "persistentvolumeclaims", "events", "configmaps", "secrets", "nodes", "persistentvolumes", "resourcequotas"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses", "networkpolicies"]
  verbs: ["get", "list"]
- apiGroups: ["certmanager.k8s.io"]
  resources: ["certificates", "challenges", "clusterissuers", "issuers", "orders"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "replicasets", "statefulsets"]
  verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
  resources: ["clusterservicestatuses"]
  verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs"]
  verbs: ["get", "list"]
- apiGroups: ["operators.coreos.com"]
  resources: ["clusterserviceversions"]
  verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods" ,"nodes"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["operator.ibm.com"]
  resources: ["*"]
  verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
  resources: ["machineconfigpools"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-overview-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:{Namespace-Name}:default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-overview-clusterrole

Collecting information for failure module

To collect information about the Failure module, create a single YAML file with the following ClusterRole and ClusterRoleBinding definitions. Then, run the oc apply -f <file-name>.yaml command to create the ClusterRole and ClusterRoleBinding resources.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-failure-clusterrole
rules:
- apiGroups: [""]
  resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs", "jobs"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-failure-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:{Namespace-Name}::default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-failure-clusterrole

Collecting information for Cloud pak module

To collect information about the Cloud pak module, create a single YAML file with the following ClusterRole and ClusterRoleBinding definitions. Then, run the oc apply -f <file-name>.yaml command to create the ClusterRole and ClusterRoleBinding resources.

Sample command:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-cloudpak-clusterrole
rules:
- apiGroups: [""]
  resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints", "nodes", "secrets", "events", "configmaps", "replicationcontrollers"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses", "networkpolicies"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs", "jobs"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets", "replicasets"]
  verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
  resources: ["networks", "imagecontentsourcepolicies"]
  verbs: ["get", "list"]
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests"]
  verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
  resources: ["imagecontentsourcepolicies", "networks"]
  verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
  resources: ["machineconfigpools", "machineconfigs"]
  verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
  resources: ["volumeattachments"]
  verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
  resources: ["clusterservicestatuses"]
  verbs: ["get", "list"]
- apiGroups: ["monitoring.coreos.com"]
  resources: ["servicemonitors"]
  verbs: ["get", "list"]
- apiGroups: ["policy"]
  resources: ["endpointslices", "poddisruptionbudgets"]
  verbs: ["get", "list"]
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list"]
- apiGroups: ["build.openshift.io"]
  resources: ["buildconfigs", "builds"]
  verbs: ["get", "list"]
- apiGroups: ["apps.openshift.io"]
  resources: ["deploymentconfigs"]
  verbs: ["get", "list"]
- apiGroups: ["image.openshift.io"]
  resources: ["imagestreams"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-cloudpak-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:{Namespace-Name}::default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-cloudpak-clusterrole

Example procedure

The following example includes definitions of all resources that are required to enable non-cluster admins to run the must-gather commands. To create the resources, copy and paste the contents at the CLI or to the YAML editor in your Red Hat OpenShift Container Platform console.

cat <<EOF | tee >(oc apply -f -) | cat >/dev/null
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cpfs-mg-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/exec"]
  verbs: ["get", "list", "create"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cpfs-mg-cluster-rolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: cpfs
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cpfs-mg-clusterrole

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-overview-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods" ,"services", "services/finalizers", "endpoints", "persistentvolumeclaims", "events", "configmaps", "secrets", "nodes", "persistentvolumes", "resourcequotas"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses", "networkpolicies"]
  verbs: ["get", "list"]
- apiGroups: ["certmanager.k8s.io"]
  resources: ["certificates", "challenges", "clusterissuers", "issuers", "orders"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "replicasets", "statefulsets"]
  verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
  resources: ["clusterservicestatuses"]
  verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs"]
  verbs: ["get", "list"]
- apiGroups: ["operators.coreos.com"]
  resources: ["clusterserviceversions"]
  verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["operator.ibm.com"]
  resources: ["*"]
  verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
  resources: ["machineconfigpools"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-overview-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:ibm-common-services:default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-overview-clusterrole

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-failure-clusterrole
rules:
- apiGroups: [""]
  resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs", "jobs"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-failure-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:ibm-common-services:default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-failure-clusterrole

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-cloudpak-clusterrole
rules:
- apiGroups: [""]
  resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints", "nodes", "secrets", "events", "configmaps", "replicationcontrollers"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses", "networkpolicies"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs", "jobs"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets", "replicasets"]
  verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
  resources: ["networks", "imagecontentsourcepolicies"]
  verbs: ["get", "list"]
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests"]
  verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
  resources: ["imagecontentsourcepolicies", "networks"]
  verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
  resources: ["machineconfigpools", "machineconfigs"]
  verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
  resources: ["volumeattachments"]
  verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
  resources: ["clusterservicestatuses"]
  verbs: ["get", "list"]
- apiGroups: ["monitoring.coreos.com"]
  resources: ["servicemonitors"]
  verbs: ["get", "list"]
- apiGroups: ["policy"]
  resources: ["endpointslices", "poddisruptionbudgets"]
  verbs: ["get", "list"]
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list"]
- apiGroups: ["build.openshift.io"]
  resources: ["buildconfigs", "builds"]
  verbs: ["get", "list"]
- apiGroups: ["apps.openshift.io"]
  resources: ["deploymentconfigs"]
  verbs: ["get", "list"]
- apiGroups: ["image.openshift.io"]
  resources: ["imagestreams"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-cloudpak-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:ibm-common-services:default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-cloudpak-clusterrole

EOF

Quick start guide to get support information for non-cluster admin

1. Navigate to the directory where you want to store the MustGather data.

2. Collect the data by running the following command.

    oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=icr.io/cpopen/cpfs/must-gather:<image_version> -- gather -n {NAMESPACE-NAME}
   

   Where the is the version of the MustGather image that you can find at icr.io/cpopen/cpfs/.

   For example:

    oc adm must-gather --run-namespace=ibm-common-services --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -n ibm-common-services
   

   The collected data is compressed into a .tgz output file and stored in ./must-gather.local...

3. Attach the .tgz output file that contains the support information when you submit the support case on the IBM Support Portal.

Usage and examples: non-cluster admin

Available command options for non-cluster admin

You can use the MustGather tool to collect support information about IBM Cloud Paks. You can collect all support information from your cluster for non-cluster admin, or only specified information about specified namespaces. Add options to the MustGather command to collect the data you need.

The following example shows all available command options:

oc adm must-gather --run-namespace=<NAMESPACE_NAME> --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v

Note: To know more about oc adm must-gather command options and gather command options, see oc adm must-gather command options and gather command options.

Usage examples: non-cluster admin

The following examples show how to use the MustGather tool to collect specified support data for non-cluster admin. When you copy and run the code that is provided in the examples, the shell script is saved on the local host. You can reuse the script anytime to collect the support information.

oc adm must-gather --run-namespace=<NAMESPACE_NAME> --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v

Where NAMESPACE_NAME parameter is the namespace name that belongs to the non-cluster admin and IBM Cloud Pak_NAMESPACES parameter is one or multiple namespaces from where the user collects to mustgather information.

For example, a non-cluster admin is assigned to ibm-common-services namespace and that non-cluster admin wants to collect mustgather logs from different namespaces, such as, ibm-common-services, iaf-demo for the Overview module, then the sample command might resemble the following command:

oc adm must-gather --run-namespace=ibm-common-services --image=icr.io/cpopen/cpfs/must-gather:4.6.14 -- gather -m overview -n ibm-common-services,iaf-demo

Note: Before you run the code, edit the following parameters in the script:

• Provide a namespace where your IBM Cloud Pak is deployed in the CLOUDPAK_NAMESPACES parameter.
• For offline use, set MUST_GATHER_IMAGE to the name of your offline MustGather image.
• For online use, set the <image_version> to the version of the MustGather image that you can find at icr.io/cpopen/cpfs/.

Note: Non-cluster admin supports Overview, Failure and Cloudpak data modules.

Collecting support information with default data modules for non-cluster admin

1. Run the following command to save the must-gather-default.sh script on your local host.

    cat >must-gather-default.sh <<'EOT'
    #!/bin/bash
    export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version>
    export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES>
    export MUST_GATHER_MODULES=overview,failure,cloudpak
    oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES
    EOT
   
    chmod +x must-gather-default.sh
   

2. Run the following command to collect the support data.

    ./must-gather-default.sh
   

Collecting only the support information about the IBM Cloud Pak for non-cluster admin

1. Run the following command to save the must-gather-cp.sh script on your local host.

    cat >must-gather-cp.sh <<'EOT'
    #!/bin/bash
    export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version>
    export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES>
    export MUST_GATHER_MODULES=overview,cloudpak
    oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES
    EOT
   
    chmod +x must-gather-cp.sh
   

2. Run the following command to collect the support data.

    ./must-gather-cp.sh
   

Collecting logs for resources with the same labels for non-cluster admin

Prerequisite: Include the label serviceability-addon=<component-name> in all the resources.

To collect the logs only for the resources with the label serviceability-addon=<component-name> in provided namespaces, run the following command:

oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -n namespace

Using MustGather diagnostics on a multiarchitecture cluster

In a multiarchitecture cluster, use the must-gather options --node-name or —node-sector to choose non-ARM nodes to run MustGather diagnostics.

See the following example:

oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:<image_version> -- gather -n {NAMESPACE-NAME} --node-selector 'node-role.kubernetes.io/worker'

The --node-selector argument and example value specifies to perform the packet captures on the worker nodes. As an alternative, you can specify the --node-name argument to run the packet capture on a single node.