Collecting support information about the cluster
The MustGather tool collects information about your cluster that is crucial for troubleshooting problems and providing support.
The MustGather tool collects support information about the IBM Cloud Paks and the clusters where they are deployed. You can use this tool to gather information that is needed and relevant to the problem before you open a case with the IBM® Support .
The MustGather tool is built on the Red Hat® OpenShift® Container Platform MustGather tool ,
and is automatically deployed with the IBM Cloud Pak foundational services as a part of the
ibm-healthcheck-operator
. However, without the ibm-healthcheck-operator
you can still use the MustGather tool through the command
line directly from the MustGather image.
- Using MustGather diagnostics through the command line for cluster admin
- Available data modules
- Using MustGather for non-cluster admin
- Using MustGather as a part of
ibm-healthcheck-operator
Using MustGather diagnostics through the command line for cluster admin
Important: IBM Cloud Pak foundational services version 3.18 onwards, the MustGather image for IBM Cloud Paks is available at a new location. If you use the latest
tag for the MustGather image, the image from the new
location is supported on all versions of foundational services.
If you need to use a specific image version, see the following notes:
- For foundational services versions 3.18 and later, use the images from
icr.io/cpopen/cpfs/
. - For foundational services versions 3.17 and prior, use the images from quay.io
.
All references and commands in the following sections point to the icr.io/cpopen/cpfs/
location. If your foundational services version is 3.17 or prior, and you need to use a specific image version, you can replace icr.io/cpopen/cpfs/
with quay.io/opencloudio/
, where required.
Available data modules
IMPORTANT: Non-cluster admin supports Overview, Failure and Cloudpak data modules. Cluster admin supports all the available data modules.
To collect a specific type of support information only, define the data modules when you run the MustGather tool. The following table lists the details of all available modules.
Note: The default scan does not collect information about secrets as it is a user sensitive data. To collect information about secrets, you can use the dedicated data module.
Data module | Collected cluster data | Included in the default scan? |
---|---|---|
overview |
|
Yes |
system |
system data module because of the restricted
Security Context Constraints. |
Yes |
failure |
|
Yes |
ocp |
|
Yes |
cloudpak |
|
Yes |
secrets |
|
No |
etcd Available for OpenShift version 4.5 and higher |
|
No |
route |
|
No |
Prerequisites
- Access to the cluster as a user with a cluster-admin role.
- The OpenShift Container Platform CLI (oc) installed. For more information, see Installing the CLI
.
-
Access to the MustGather image for IBM Cloud Paks. See the previous section.
Note: For offline environments that cannot access icr.io, the Docker image is available in the offline installation package. For more information, see Downloading the image in offline environments.
Quick start: Getting support information for cluster admin
Complete the following steps to retrieve the support information from your cluster.
- Navigate to the directory where you want to store the MustGather data.
-
Collect the data by running the following command.
oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:<image_version> -- gather -n common-service,ibm-common-services
where the
<image_version>
is the version of the MustGather image that you can find aticr.io/cpopen/cpfs/
.For example:
oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -n common-service,ibm-common-services
The collected data is compressed into a .tgz output file and stored in
./must-gather.local.<rand>
. -
Attach the .tgz output file that contains the support information when you submit the support case on the IBM Support Portal
.
Usage and examples- cluster admin
Available command options
You can use the MustGather tool to collect support information about IBM Cloud Paks. You can collect all support information from your cluster, or only specified information about specified namespaces. Add options to the MustGather command to collect the data you need.
The following example shows all available command options:
oc adm must-gather --run-namespace=<NAMESPACE_NAME> --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v
oc adm must-gather
command options
Option | Description |
---|---|
--image |
A MustGather plug-in image to run. Specify the image name and version, for example, icr.io/cpopen/cpfs/must-gather:4.5.16 . |
Note: oc adm must-gather
is a default OpenShift command. For more information, run the oc adm must-gather -h
command.
gather
command options:
Option | Description | |
---|---|---|
-m |
Define the data module that specifies what type of information is collected. For more information, see Available data modules. When this option is not set, the MustGather tool collects information that includes the following data modules: overview, system, failure, ocp, cloudpak. | |
-n --namespaces= |
Specify the IBM Cloud Pak namespace or namespaces from which the data is collected for the cloudpacks and secrets data modules. By default, the namespace is set to ibm-common-services . Separate each namespace
with a comma, for example ibm-common-services,common-service . |
|
-h |
Display the help message. | |
-v |
Display the version. | |
-l |
Specify the IBM Cloud Pak labels to collect logs. Separate the labels with a comma. This option requires you to set the -m cloudpak at the same time. By default, no labels are defined. |
|
-c Available for IBM Cloud Pak foundational services version 3.11.0 and later. |
Specify the component name to collect logs. If you pass the component-name followed by -c parameter, must-gather collects the MustGather diagnostics with the label serviceability-addon:<component-name> .
Components pre-define this label for all of their resources. If you add namespaces -n parameter followed by -c parameter, it restricts the collection scope to the namespace of -n parameter. If you don't
pass the namespaces, by default, the namespace is set to common-services and ibm-common-services . By using -c parameter, you cannot collect the MustGather diagnostics for multiple components with single
execution. If you want to collect the MustGather diagnostics for different components, run the separate must-gather command with other component name. For more information, see Collecting logs for the resources with same labels. |
|
-s ex Available for IBM Cloud Pak foundational services version 3.11.0 and later. |
Collects MustGather diagnostics information by using custom script that is not the part of a must-gather image. The custom script is placed inside the Docker image that needs to be hosted in IBM trusted registries. To use this
feature, pass the component name of -c parameter followed by -s ex parameter. For more information, see Using custom script to collect MustGather diagnostics that are not provided by default. |
Usage examples
The following examples show how to use the MustGather tool to collect specified support data. When you copy and run the code that is provided in the examples, the shell script is saved on the local host. You can reuse the script anytime to collect the support information.
Note: Before you run the code, edit the following parameters in the script:
- Provide a namespace where your IBM Cloud Pak is deployed in the
CLOUDPAK_NAMESPACES
parameter. - For offline use, set
MUST_GATHER_IMAGE
to the name of your offline MustGather image. - For online use, set the
<image_version>
to the version of the MustGather image that you can find aticr.io/cpopen/cpfs/
.
Collecting support information with default data modules
-
Run the following command to save the
must-gather-default.sh
script on your local host.cat >must-gather-default.sh <<'EOT' #!/bin/bash export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version> export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES> export MUST_GATHER_MODULES=overview,system,failure,ocp,cloudpak oc adm must-gather --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES EOT chmod +x must-gather-default.sh
-
Run the following command to collect the support data.
./must-gather-default.sh
Note: To collect customized diagnostic information about your IBM Cloud Pak and foundational services deployments in your cluster, see Collecting IBM Cloud Pak foundational services information for problem determination.
Collecting only the support information about the IBM Cloud Pak
-
Run the following command to save the
must-gather-cp.sh
script on your local host.cat >must-gather-cp.sh <<'EOT' #!/bin/bash export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version> export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES> export MUST_GATHER_MODULES=overview,cloudpak oc adm must-gather --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES EOT chmod +x must-gather-cp.sh
-
Run the following command to collect the support data.
./must-gather-cp.sh
Collecting logs for the resources with same labels
Prerequisite: Include the label serviceability-addon=<component-name>
in all the resources.
To collect the logs only for the resources with the label serviceability-addon=<component-name>
in provided namespaces, run the following command:
oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -n namespace
Using custom script to collect MustGather diagnostics that are not provided by default
To collect data that is not provided by default, run the following must-gather
command with custom script:
oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -s ex
For example, if the component name is foo
, run the following command:
oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c foo -s ex
From foundational services 3.17 version, you can pass the namespace to the command for custom script and use the namespace inside the custom script.
For example, if the component name is foo
, and the component custom script needs to collect diagnostic specific to a namespace, you can run the following must-gather
command:
oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c foo -s ex -n namespace
Creating and running the custom scripts
Before you run the custom scripts to collect the MustGather diagnostics, components need to perform the following tasks:
Note: Make sure to refer the Standard adaption guide.
- Component creates its own custom scripts.
- Component places the custom scripts inside the Docker image that is a part of IBM trusted registries.
- Component provides the configmap with the image location.
After you run -c
parameter with -s ex
parameter, must-gather
performs the following tasks:
must-gather
fetches the image location from the component in the configmap.- After
must-gather
gets the image details from the configmap, it uses the image to run a container and copy the custom scripts. must-gather
validates and runs the scripts to collect the logs.
Standard adaption guide
It describes the standards that the component must follow.
The following list describes the set of rules for the Standard adaption guide:
-
Custom script standard
When the component creates the custom script, use the following guidelines:
- By default, the script type is bash or shell scripts.
- If you have more than one script, name the master script as
ibm-mustgather-customscript-<component-name>
. A master script is a script from where the execution starts. - Place the scripts inside the
/home/custom-scripts
folder. - Keep the output path as
${BASE_PATH}
for data collection. -
Use only
get
,list
, andexec
commands on the resources. -
Sample script
oc exec -n openshift-ingress ${pod} -- cat /lib/haproxy/haproxy.config >${BASE_PATH}/${pod}.haproxy.config & oc logs -n ${NS} ${POD} --all-containers >${BASE_PATH}/${NS}.${POD}.current.log
-
Sample script that uses the namespace from the
must-gather
command:if [ -n "$1" ]; then ns=$1 oc get pods -n ${ns}
-
Dockerfile standard
Use the following guidelines for Docker image to include custom scripts:
- Use the standard base image,
ubi8:minimal
. - Install the rsync and tar packages.
-
Create a folder
/home/custom-scripts
and copy the scripts in this folder. Give 777 permission to this folder.The following sample is of a Dockerfile:
FROM hyc-cloud-private-edge-docker-local.artifactory.swg-devops.com/build-images/ubi8-minimal:latest RUN microdnf update -y && microdnf install rsync tar -y && microdnf clean all &&\ mkdir /home/custom-scripts && chmod777 /home/custom-scripts COPY collection-scripts/*<custom-scripts>* /home/custom-scripts
- Use the standard base image,
-
IBM trusted registries
Use only the following trusted registries for Docker image:
- quay.io/opencloudio/
- cp.icr.io/cp/icp-foundation/
- cp.stg.icr.io/cp/icp-foundation/
- icr.io/cpopen/
-
Configmap standard
Use the following guidelines when the component provides configmap:
- Name the configmap as
ibm-mustgather-customscript-<component-name>
and the label asserviceability-addon=<component-name>
. - Set the field
immutable
astrue
in configmap. - Provide the image location as key value pair, where key is the component name, value is the image location with
sha256
manifest. -
Don't use image tags. You can use
sha256
manifest.The following sample is of a configmap file:
NAME :
ibm-mustgather-customscript-<component-name>
apiVersion: v1 kind: ConfigMap metadata: name: ibm-mustgather-customscript-<component-name> namespace: <component specific namespace> labels: serviceability-addon:component-name data: <component-name>: icr.io/cpopen/operator@sha256:6535136... immutable: true
Note: MustGather also supports default configmap that keeps image location of the component. However, to manage better and avoid any dependencies, MustGather recommends the component to bring its own configmap by following Standard adaption guidelines.
- Name the configmap as
Using MustGather for non-cluster admin
You can use MustGather for non-cluster admin role.
Prerequisites:
-
Access to the cluster as a user with non-cluster admin role. For more information, see Permissions for non-cluster admin.
-
The OpenShift Container Platform version 4.12 CLI is installed. For more information, see Installing the CLI.
-
Access to the MustGather image for IBM Cloud Paks.
Note: A non-cluster admin can use all the must-gather
commands that an admin uses. However, non-cluster admins must add the --run-namespace
parameter to each command. To use the --run-namespace
parameter, you need to install Red Hat OpenShift Container Platform 4.12 CLI. The 4.12 CLI also works in older versions of OpenShift clusters.
Permission for non-cluster admin
Login as a cluster admin user and assign the ClusterRole
and ClusterRoleBinding
roles for non-cluster admin user to collect data for modules, such as Overview, Failure, and Cloudpak.
See the following sample ClusterRole
definition:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cpfs-mg-clusterrole
rules:
- apiGroups: [""]
resources: ["pods", "pods/log", "pods/exec"]
verbs: ["get", "list", "create"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
See the following sample ClusterRoleBinding
definition:
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cpfs-mg-cluster-rolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: cpfs
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cpfs-mg-clusterrole
Collecting information for Overview module
To collect information about the Overview
module, create a single YAML file with the following ClusterRole
and ClusterRoleBinding
definitions. Then, run the oc apply -f <file-name>.yaml
command
to create the ClusterRole
and ClusterRoleBinding
resources.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-overview-clusterrole
rules:
- apiGroups: [""]
resources: ["pods" ,"services", "services/finalizers", "endpoints", "persistentvolumeclaims", "events", "configmaps", "secrets", "nodes", "persistentvolumes", "resourcequotas"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "create"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["get", "list"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses", "networkpolicies"]
verbs: ["get", "list"]
- apiGroups: ["certmanager.k8s.io"]
resources: ["certificates", "challenges", "clusterissuers", "issuers", "orders"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "daemonsets", "replicasets", "statefulsets"]
verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
resources: ["clusterservicestatuses"]
verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["cronjobs"]
verbs: ["get", "list"]
- apiGroups: ["operators.coreos.com"]
resources: ["clusterserviceversions"]
verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
resources: ["horizontalpodautoscalers"]
verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
resources: ["pods" ,"nodes"]
verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
resources: ["routes"]
verbs: ["get", "list"]
- apiGroups: ["operator.ibm.com"]
resources: ["*"]
verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
resources: ["machineconfigpools"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-overview-clusterrolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccount:{Namespace-Name}:default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: mg-overview-clusterrole
Collecting information for failure module
To collect information about the Failure
module, create a single YAML file with the following ClusterRole
and ClusterRoleBinding
definitions. Then, run the oc apply -f <file-name>.yaml
command
to create the ClusterRole
and ClusterRoleBinding
resources.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-failure-clusterrole
rules:
- apiGroups: [""]
resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "create"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["cronjobs", "jobs"]
verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
resources: ["routes"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "daemonsets", "statefulsets"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-failure-clusterrolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccount:{Namespace-Name}::default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: mg-failure-clusterrole
Collecting information for Cloud pak module
To collect information about the Cloud pak
module, create a single YAML file with the following ClusterRole
and ClusterRoleBinding
definitions. Then, run the oc apply -f <file-name>.yaml
command to create the ClusterRole
and ClusterRoleBinding
resources.
Sample command:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-cloudpak-clusterrole
rules:
- apiGroups: [""]
resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints", "nodes", "secrets", "events", "configmaps", "replicationcontrollers"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "create"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses", "networkpolicies"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
resources: ["routes"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["cronjobs", "jobs"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "daemonsets", "statefulsets", "replicasets"]
verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
resources: ["networks", "imagecontentsourcepolicies"]
verbs: ["get", "list"]
- apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests"]
verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
resources: ["imagecontentsourcepolicies", "networks"]
verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
resources: ["machineconfigpools", "machineconfigs"]
verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
resources: ["clusterservicestatuses"]
verbs: ["get", "list"]
- apiGroups: ["monitoring.coreos.com"]
resources: ["servicemonitors"]
verbs: ["get", "list"]
- apiGroups: ["policy"]
resources: ["endpointslices", "poddisruptionbudgets"]
verbs: ["get", "list"]
- apiGroups: ["discovery.k8s.io"]
resources: ["endpointslices"]
verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
resources: ["horizontalpodautoscalers"]
verbs: ["get", "list"]
- apiGroups: ["build.openshift.io"]
resources: ["buildconfigs", "builds"]
verbs: ["get", "list"]
- apiGroups: ["apps.openshift.io"]
resources: ["deploymentconfigs"]
verbs: ["get", "list"]
- apiGroups: ["image.openshift.io"]
resources: ["imagestreams"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-cloudpak-clusterrolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccount:{Namespace-Name}::default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: mg-cloudpak-clusterrole
Example procedure
The following example includes definitions of all resources that are required to enable non-cluster admins to run the must-gather
commands. To create the resources, copy and paste the contents at the CLI or to the YAML editor in your
Red Hat OpenShift Container Platform console.
cat <<EOF | tee >(oc apply -f -) | cat >/dev/null
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cpfs-mg-clusterrole
rules:
- apiGroups: [""]
resources: ["pods", "pods/log", "pods/exec"]
verbs: ["get", "list", "create"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cpfs-mg-cluster-rolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: cpfs
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cpfs-mg-clusterrole
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-overview-clusterrole
rules:
- apiGroups: [""]
resources: ["pods" ,"services", "services/finalizers", "endpoints", "persistentvolumeclaims", "events", "configmaps", "secrets", "nodes", "persistentvolumes", "resourcequotas"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "create"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["get", "list"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses", "networkpolicies"]
verbs: ["get", "list"]
- apiGroups: ["certmanager.k8s.io"]
resources: ["certificates", "challenges", "clusterissuers", "issuers", "orders"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "daemonsets", "replicasets", "statefulsets"]
verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
resources: ["clusterservicestatuses"]
verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["cronjobs"]
verbs: ["get", "list"]
- apiGroups: ["operators.coreos.com"]
resources: ["clusterserviceversions"]
verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
resources: ["horizontalpodautoscalers"]
verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
resources: ["routes"]
verbs: ["get", "list"]
- apiGroups: ["operator.ibm.com"]
resources: ["*"]
verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
resources: ["machineconfigpools"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-overview-clusterrolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccount:ibm-common-services:default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: mg-overview-clusterrole
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-failure-clusterrole
rules:
- apiGroups: [""]
resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "create"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["cronjobs", "jobs"]
verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
resources: ["routes"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "daemonsets", "statefulsets"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-failure-clusterrolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccount:ibm-common-services:default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: mg-failure-clusterrole
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-cloudpak-clusterrole
rules:
- apiGroups: [""]
resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints", "nodes", "secrets", "events", "configmaps", "replicationcontrollers"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "create"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses", "networkpolicies"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
resources: ["routes"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["cronjobs", "jobs"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "daemonsets", "statefulsets", "replicasets"]
verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
resources: ["networks", "imagecontentsourcepolicies"]
verbs: ["get", "list"]
- apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests"]
verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
resources: ["imagecontentsourcepolicies", "networks"]
verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
resources: ["machineconfigpools", "machineconfigs"]
verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
resources: ["clusterservicestatuses"]
verbs: ["get", "list"]
- apiGroups: ["monitoring.coreos.com"]
resources: ["servicemonitors"]
verbs: ["get", "list"]
- apiGroups: ["policy"]
resources: ["endpointslices", "poddisruptionbudgets"]
verbs: ["get", "list"]
- apiGroups: ["discovery.k8s.io"]
resources: ["endpointslices"]
verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
resources: ["horizontalpodautoscalers"]
verbs: ["get", "list"]
- apiGroups: ["build.openshift.io"]
resources: ["buildconfigs", "builds"]
verbs: ["get", "list"]
- apiGroups: ["apps.openshift.io"]
resources: ["deploymentconfigs"]
verbs: ["get", "list"]
- apiGroups: ["image.openshift.io"]
resources: ["imagestreams"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-cloudpak-clusterrolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccount:ibm-common-services:default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: mg-cloudpak-clusterrole
EOF
Quick start guide to get support information for non-cluster admin
-
Navigate to the directory where you want to store the MustGather data.
-
Collect the data by running the following command.
oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=icr.io/cpopen/cpfs/must-gather:<image_version> -- gather -n {NAMESPACE-NAME}
Where the
is the version of the MustGather image that you can find at icr.io/cpopen/cpfs/. For example:
oc adm must-gather --run-namespace=ibm-common-services --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -n ibm-common-services
The collected data is compressed into a
.tgz
output file and stored in./must-gather.local..
. -
Attach the
.tgz
output file that contains the support information when you submit the support case on the IBM Support Portal.
Usage and examples: non-cluster admin
Available command options for non-cluster admin
You can use the MustGather tool to collect support information about IBM Cloud Paks. You can collect all support information from your cluster for non-cluster admin, or only specified information about specified namespaces. Add options to the MustGather command to collect the data you need.
The following example shows all available command options:
oc adm must-gather --run-namespace=<NAMESPACE_NAME> --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v
Note: To know more about oc adm must-gather
command options and gather
command options, see oc adm must-gather
command options and gather
command options.
Usage examples: non-cluster admin
The following examples show how to use the MustGather tool to collect specified support data for non-cluster admin. When you copy and run the code that is provided in the examples, the shell script is saved on the local host. You can reuse the script anytime to collect the support information.
oc adm must-gather --run-namespace=<NAMESPACE_NAME> --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v
Where NAMESPACE_NAME parameter is the namespace name that belongs to the non-cluster admin and IBM Cloud Pak_NAMESPACES parameter is one or multiple namespaces from where the user collects to mustgather information.
For example, a non-cluster admin is assigned to ibm-common-services
namespace and that non-cluster admin wants to collect mustgather logs from different namespaces, such as, ibm-common-services
, iaf-demo
for
the Overview
module, then the sample command might resemble the following command:
oc adm must-gather --run-namespace=ibm-common-services --image=icr.io/cpopen/cpfs/must-gather:4.6.14 -- gather -m overview -n ibm-common-services,iaf-demo
Note: Before you run the code, edit the following parameters in the script:
- Provide a namespace where your IBM Cloud Pak is deployed in the
CLOUDPAK_NAMESPACES
parameter. - For offline use, set
MUST_GATHER_IMAGE
to the name of your offline MustGather image. - For online use, set the
<image_version>
to the version of the MustGather image that you can find aticr.io/cpopen/cpfs/
.
Note: Non-cluster admin supports Overview, Failure and Cloudpak data modules.
Collecting support information with default data modules for non-cluster admin
-
Run the following command to save the
must-gather-default.sh
script on your local host.cat >must-gather-default.sh <<'EOT' #!/bin/bash export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version> export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES> export MUST_GATHER_MODULES=overview,failure,cloudpak oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES EOT chmod +x must-gather-default.sh
-
Run the following command to collect the support data.
./must-gather-default.sh
Collecting only the support information about the IBM Cloud Pak for non-cluster admin
-
Run the following command to save the
must-gather-cp.sh
script on your local host.cat >must-gather-cp.sh <<'EOT' #!/bin/bash export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version> export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES> export MUST_GATHER_MODULES=overview,cloudpak oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES EOT chmod +x must-gather-cp.sh
-
Run the following command to collect the support data.
./must-gather-cp.sh
Collecting logs for the resources with same labels
Prerequisite: Include the label serviceability-addon=<component-name>
in all the resources.
To collect the logs only for the resources with the label serviceability-addon=<component-name>
in provided namespaces, run the following command:
oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -n namespace
Using MustGather as a part of ibm-healthcheck-operator
If ibm-healthcheck-operator
is installed as a part of your product, you might run MustGather diagnostics from the OpenShift cluster console and Administration panel.
Using MustGather diagnostics from the OpenShift cluster console
Note: You can use MustGather diagnostics from the OpenShift cluster console only if you have ibm-healthcheck-operator
deployed.
Complete the following actions to gather diagnostics from the OpenShift cluster console.
- Log in to your OpenShift cluster console.
- Go to Operators > Installed Operators.
- Set the project to All Projects.
- Find and select the ibm-healthcheck-operator.
- Go to the IBM MustGather Jobs tab, select Create MustGather Job.
- To run a default diagnostic scan, click Create.
-
Go to Workloads > Jobs. Find your MustGather job. The URL for downloading the results package is available in the job log, for example:
INFO: must-gather-1600137793.tar.gz is ready. You can download it from dashboard UI or using curl -k -X GET "https://<CLUSTER_URL>/must-gather/v1alpha1/mustgather/download/must-gather-1600137793.tar.gz" -H "accept: application/json" -H "Authorization: <BEARER_TOKEN>"
Using MustGather diagnostics from the Administration panel
Note: You can use MustGather diagnostics from the Administration panel only if yo have ibm-healthcheck-operator
deployed.
To run MustGather diagnostics with the console, you must use the Administration panel. This dashboard is only accessibly by Cluster Administrator users.
To run MustGather diagnostics from the Administration panel, complete the following actions:
- Log in as a Cluster Administrative user and from the main navigation menu click Home to open the Administration panel.
- From either the Welcome widget or the Diagnostics summary card, click Gather diagnostics. A side-panel opens where you can configure the diagnostics to gather and run the report generation.
- Select the diagnostics that you would like to gather. You can select more than one data module.
- If you selected to gather Cloud Pak data, provide a namespace or namespaces from which you want to collect data.
- Click Gather diagnostics. Collecting support information can take from a few to 30 minutes.
- Within the System overview section of the main work area, view the Diagnostics summary card, the status on the card is Report generating in progress. After the gathering process completes, the report can be downloaded.
- To download the package with results, click the download link next to the package name on the summary card.
Note: To view the support information about the cluster upgrade, go to Overview > ClusterUpgrade in the downloaded directory.