Question & Answer
Question
MustGather for all Operational Decision Manager (ODM) on IBM Cloud Pak for Automation (CP4BA) on Red Hat OpenShift.
Gathering this information before calling IBM® support helps familiarize you with the troubleshooting process and saves you time.
Answer
For IBM Operational Decision Manager (ODM) on IBM Cloud Pak for Business Automation (CP4BA) on Red Hat OpenShift, gather the following information for any problem related to ODM on CP4BA.
General Information
General Information
Conventions used in this document
- -n <ODM-NAMESPACE> is specified for most ODM MustGather commands. Namespace is optional when the oc project is already set to the ODM-Namespace.
- The kubectl command can generally be used in place of the oc command.
- For any oc or kubectl command, append --help to view help and command options.
- All oc and kubectl commands can be extended to send the stdout to the console and saved to a file by piping the stdout to tee cmd:
Example: to view the operator log and save to operator.log file
$ oc logs -f deployment/ibm-cp4a-operator -c operator | tee operator.log
Automated Collection:
Run the following automated collection script to collect all of the required ODM logs and system information-
Procedure:
- Download script to the RHEL host you where you access your Red Hat OpenShift cluster with either oc or kubectl command.
- Make script executable:
$ chmod +x collectODM_K8S_MustGather.sh
- Run the script by specifying the namespace where ODM/CP4BA is installed:
$ collectODM_K8S_MustGather.sh namespace
- Attach the generated MustGatherODM_K8S.xxxxx.tar to support case.
For more command options, such as limiting collection to a specific deployment or collecting Java™ dumps, run:
$ collectODM_K8S_MustGather.sh options
Manual Collection:
System information:
- Run the following commands to manually collect system information and configuration-related info:
cat /etc/os-release > system-info.log
kubectl version >> system-info.log
oc version >> system-info.log
sudo podman version >> system-info.log
skopeo -v >> system-info.log
kubectl get configmap >> system-info.log
kubectl get csv >> system-info.log
kubectl get route >> system-info.log
kubectl get mcp >> system-info.log
kubectl get icp4acluster -o yaml >> system-info.log
kubectl get pvc >> system-info.log
kubectl get secrets >> system-info.log
kubectl get csv -n ibm-common-services >> system-info.log.
sudo podman images --digests > container_images.log
kubectl get pods > odm_pods.log
kubectl get events > odm_ events.log
- Collect detailed information for configmaps
touch configmaps.log
configmaps=$(kubectl get configmap | grep -i odm | awk '{print $1}')
for cfgmap in $configmaps ; do kubectl get configmap $cfgmap -o yaml >> configmaps.log ; done
Collecting runtime logs manually:
Run the following commands to capture the ODM runtime logs for a specific ODM pod or several pods:
Obtain runtime log for a pod
oc log -n <ODM-NAMESPACE> POD_NAME > POD_NAME.log
Obtain detailed description of a pod
oc describe pod -n <ODM-NAMESPACE> POD_NAME > POD_NAME_describe.log
Example for Decision Center:
$ oc get log cp4acluster-odm-decisioncenter-5b77b7d4ff-jldrj > DC.log
$ oc describe pod -n dba ibm-cp4a-operator-84c548fc8-nxl9h > cp4a_operator_describe.log
$ oc describe pod -n dba ibm-cp4a-operator-84c548fc8-nxl9h > cp4a_operator_describe.log
Note: When there are many pods for a deployment and you have chosen to not run the automated collection script provided,
run the following shell commands to collect logs for each pod in the deployment.
This can reduce the effort of manually collecting the logs when there are multiple pods for a given deployment.
Example for decisionserverruntime pods, where there are usually several pods running:
$ pods=$(oc get pods --field-selector=status.phase=Running | grep decisionserverruntime | awk '{print $1}')
$ for pod in $pods ; do oc get pod $pod -o yaml ; done
The two commands generate a separate log for each decisionserverruntime pod in the deployment.
You can use this pattern for other commands in order to generate a log for each pod in the deployment by replacing
the oc (or kubectl) command after the 'do' statement with a different oc command.
For example, to generate the yaml for each pod in the deployment:
$ for pod in $pods ; do oc get pod $pod -o yaml > $pod.yaml ; done
Detailed logging and tracing:
In some cases, support may request a change to the default logging for a container or several containers in order to diagnose an issue further or to apply a specific trace spec in Liberty.
In this case, the configmap needs to be modified, and a custom resources (CR) yaml applied and the cp4a-operator applies the change to the running ODM container(s). When applying the CR's, collect the cp4a-operator logs in addition the Liberty logs that required the log level change.
Typically, this is done only for a short duration in order to collect the needed MustGather and attach to the support case, then modify the configmap and apply a CR to set the logging level back to the default levels, as it could have an impact on performance.
Executing commands/bash shell inside a pod:
In addition to the oc command described above which runs outside of the container, you can also view/collect the logs from within a container:
oc exec [flags] POD [-c CONTAINER] -- COMMAND [args...]
Example: to run a bash shell within the decisioncenter pod and tail the liberty messages.log:
oc exec -ti cp4acluster-odm-decisioncenter-5b77b7d4ff-jldrj -- /bin/bash
bash-4.2$ tail -f /logs/messages.log
oc exec [flags] POD [-c CONTAINER] -- COMMAND [args...]
Example: to run a bash shell within the decisioncenter pod and tail the liberty messages.log:
oc exec -ti cp4acluster-odm-decisioncenter-5b77b7d4ff-jldrj -- /bin/bash
bash-4.2$ tail -f /logs/messages.log
Having the capability to inspect data and run commands from within containers and pods provides additional diagnostics not easily performed from outside the container and its associated mounted volumes.
-------------------------------------------------------------------------------------------------------------------
[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS2JQC","label":"IBM Cloud Pak for Automation"},"ARM Category":[{"code":"a8m0z0000001iUhAAI","label":"Operate-\u003EODM App Management"},{"code":"a8m50000000L36nAAC","label":"Operate-\u003EODM App Management-\u003EDecision Center"},{"code":"a8m0z0000001iUcAAI","label":"Operate-\u003EODM Install\\Upgrade\\Setup"},{"code":"a8m50000000CcsOAAS","label":"Use-\u003EODM App Usage-\u003EDecision Server"},{"code":"a8m50000000L1ZJAA0","label":"Use-\u003EODM App Usage-\u003EDecision Server-\u003ERES Console"},{"code":"a8m0z0000001htmAAA","label":"Use-\u003EODM App Usage-\u003EDecision Server-\u003ETesting-\u003EDecisionRunner"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"20.0.0;20.0.3;21.0.2;21.0.3"}]
Product Synonym
ODM; CP4BA
Was this topic helpful?
Document Information
Modified date:
18 October 2022
UID
ibm16208668