IBM Support

Mustgather: Collecting data to diagnose issues with IBM Automation Decision Services

Troubleshooting


Problem

This document describes the general information and diagnostic data needed to start troubleshooting issues related to IBM Automation Decision Services.
The document applies for the Standalone edition and the Cloud Pak for Automation edition.
When you open a case for problems related to Automation Decision Services, include the diagnostics gathered following this document to expedite its resolution.

Resolving The Problem

Overview of Automation Decision Services diagnostic information
Collect diagnostic information - must provide for every case
Component specific diagnostic configuration - provide for the specific component you are having an issue with
Collect diagnostic information

Everything needed to collect Automation Decision Services diagnostic information is detailed here. 
Problem and environment description
  • Provided a detailed description of your issue. Include screen captures and steps to recreate the issue if possible.
  • Is the issue intermittent or recreatable? Has there always been an issue or did it start only after a change occurred?
  • What is the business impact? Is the issue impacting any deadlines?
  • Provide a reference to the documentation used when the issue happens.
  • Which platform are you using: OpenShift, managed OpenShift, other Kubernetes platform?
  • Provide version (and vendor as needed) for the following products:
    • Automation Decision Services deployment version
    • Database vendor and version.
    • LDAP vendor and version.
    • If this installation is an IBM Cloud Pak for Automation deployment, what version are you using?
  • For deployment issues, were images downloaded from Entitled Registry or Passport Advantage?
Collecting data
Automatic collection
To collect information automatically follow this steps:
  • Download the Automation Decision Services repository
    It provides shell scripts to collect data automatically.
  • Unzip the repository
  • Move to the must-gather directory.
  • Follow the instructions in the readme file to gather diagnostics automatically
Manual collection
If you prefer gathering information manually follow these steps.
Note:
  • Run diagnostic commands from an empty directory to make it easy to package the files collected.
  • Run the commands from the project or namespace containing Automation Decision Services or use the -n <namespace> flag with all kubectl commands.
Steps:
  • For installation on OpenShift provide the output of this command: 
    oc version > version.txt
  • Provide the output of this command: 
    kubectl version >> version.txt
    Note:
    • Running the two previous commands in order produces a single versions.txt file containing both versions
    • For installation on other Kubernetes technology provide version and architecture
  • Provide the Custom Resource (CR) .yaml file used by the operator to set up the environment
    For Cloud Pak installation: 
    kubectl get icp4acluster -o yaml > config.yaml
    For Standalone installation: 
    kubectl get AutomationDecisionService-o yaml > config.yaml
    Collect information about the nodes.
  • kubectl get nodes -o wide > nodes.txt
  • Collect information about the pod statuses 
    kubectl get pods > pods.txt
  • Collect information about the pod containers 
    kubectl get pods -o jsonpath="{..image}" > containerInfo.txt
  • On OpenShift gather route configuration 
    oc get route > routes.txt
    Note: If needed, more detailed route configuration information can be gotten with -o yaml option
  • For installation on other technologies, get the ingresses 
    kubectl get ingresses > ingresses.txt
    Collect the defined secrets
  • kubectl get secrets > secrets.txt
  • Collect the defined persistent volume claims 
    kubectl get pvc > pvcs.txt
  • Collect the description and log of any pod that you are having issues with: 
    kubectl describe pod <pod-name> > describe-<podname>.txt
kubectl logs <podname> > log-<podname>.log
  • On Openshift gather the operator log 
    namespace=$(oc project -q)
icp4a_instance=$(oc get icp4acluster -o=jsonpath='{.items[0].metadata.name}')
operator_pod=$(oc get pod | grep operator | awk '{print $1}')
oc cp ${operator_pod}:/tmp/ansible-operator/runner/icp4a.ibm.com/v1/ICP4ACluster/${namespace}/${icp4a_instance}/artifacts/latest/stdout operator.log
  • Tar the collection directory and send it to the support team.
Component specific diagnostic information
Decision Designer
Use the undermentioned trace strings and follow steps in the section  How to set trace strings and collect logs? to generate the data.
The name of the pod on which the generated logs are available is provided in each section.
  • REST API
    CR format Configuration Trace location
    standalone spec.rest_api.log_trace_specification In the ads-rest-api pod
    Cloud Pak spec.ads_configuration.rest_api.log_trace_specification In the icp4adeploy-ads-rest-api pod
     
    Trace String Description
    com.ibm.odm.expresso.web=finer Show debug level logs about the rest api operations
    com.ibm.odm.foundation.adapters=finer Show debug level logs about operations on decision artifacts
    org.apache.httpcomponents=finer Log all http traffic going through the httpclient with debug level
  • Git Service
     
    CR format Configuration Trace location
    standalone spec.git_service.log_trace_specification In the ads-git-service pod
    Cloud Pak spec.ads_configuration.git_service.log_trace_specification In the icp4adeploy-ads-git-service pod
     
    Trace String Description
    com.ibm.odm.expresso=finer Show debug level logs about git operations
    com.ibm.odm.gitservice=finer Show debug level logs about the service global operation and mongo db related logs
    For enterprise GitHub, confirm that the Git id and Git Access Token are used for making the connection. Do not use your email address that is used for SSO.
    • Parsing Service
       
      CR format Configuration Trace location
      standalone spec.parsing_service.log_trace_specification In the ads-parsing-service pod
      Cloud Pak spec.ads_configuration.parsing_service.log_trace_specification In the icp4adeploy-ads-parsing-service pod
       
      Trace String Description
      com.ibm.odm.parsing=finer Show debug level logs about the parsing service global operation
      com.ibm.odm.foundation.adapters=finer Show debug level logs about build and parsing of decisions
  • Run Service
     
    CR format Configuration Trace location
    standalone spec.run_service.log_trace_specification In the ads-run-service pod
    Cloud Pak spec.ads_configuration.run_service.log_trace_specification In the icp4adeploy-ads-run-service pod
     
    Trace String Description
    com.ibm.odm.runservice=finer Show debug level logs about the run service global operation
    com.ibm.odm.foundation.adapters=finer Show debug level logs about build and running of decisions
Deployment
  • If the deploy link is unavailable in the Deploy tab, confirm that:
    • the project is connected to a Git repository,
    • changes are shared,
    • a version is created.
  • If a failure is seen while building or deploying a decision service, check the errors by clicking the View logs icon next to the deployment status.
    Copy and save this log to a file deployment.log  in the collection directory.
Runtime
To include the exception stack trace in the response payload set a property to true in the CR
CR format Configuration Trace location
standalone spec.decision_runtime_service.stack_trace_enabled: true In the response payload
Cloud Pak spec.ads_configuration.decision_runtime_service.stack_trace_enabled: true In the response payload
How to set trace strings and collect logs?
Follow these steps to enable tracing in the Automation Decision Services containers.
  • Edit the CR .yaml file used by the operator to create the Automation Decision Services pods. You can use the file gathered earlier in this document. Modify the log_trace_specification property in the Automation Decision Services logs section of the yaml file. Follow the documentation syntax defined for the traceSpecification attribute .
    spec:
      ...
      ads_configuration:
        component_as_informed_by_support_team:
          log_trace_specification: '*=info:component_specific_trace_string_as_requested_by_support_team'
  • Update the operator with the new configuration. 
    kubectl apply -f my_cr.yaml --overwrite=true
    Notes:
    • For Cloud Pak installation, the operator can take up to 30 minutes to recognize the change and update the configuration.
    • You can grep the log file for traceSpecification to see when the trace settings change.
  • By default, the logs are saved in the standard output of the respective pods in JSON format. Collect the logs, FFDC, and trace by tarring the files on the respective pod.
  • Disable the trace by setting log_trace_specification back to "*=info" and applying the CR changes again.

Document Location

Worldwide

[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS2JQC","label":"IBM Cloud Pak for Automation"},"ARM Category":[{"code":"a8m3p000000GnuZAAS","label":"Automation Decision Service-\u003EADS App Development"},{"code":"a8m3p000000GnuaAAC","label":"Automation Decision Service-\u003EADS App Management"},{"code":"a8m3p000000GnuVAAS","label":"Automation Decision Service-\u003EADS Install\\Upgrade\\Setup"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)"},{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSD1E4E","label":"IBM Automation Decision Services"},"ARM Category":[{"code":"a8m3p000000hCCwAAM","label":"ADS standalone"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
01 June 2023

UID

ibm16405888

Page Feedback