IBM Support

MustGather: Collecting data to diagnose issues with User Management Service (UMS)

Troubleshooting


Problem

This document describes the general information and diagnostic data needed to start troubleshooting issues related to User Management Service, which is included in IBM Cloud Pak for Automation. Include the diagnostics retrieved from using this document when opening a case for User Management Service or Cloud Pak for Automation.

Resolving The Problem

Overview of User Management Service diagnostic information

General diagnostic information

As needed diagnostic information

Detailed diagnostic collection steps

These are the detailed steps to gather different types of data for User Management Service. When running diagnostic commands, run them from an empty collection directory to make it easy to package the files. Run the commands from the project or namespace containing User Management Service or use the -n <namespace> flag with all kubectl commands.
For a detailed explanation of the logs used by UMS and how to retrieve, you may reference the related document
Where to find logs in User Management Service (UMS)
 

1: Provide a detailed description of the problem and your environment

  • Provided a detailed description of your issue. Include screen captures and re-create steps if possible.
  • Is it an intermittent or re-creatable issue? Has this always been an issue or an issue that started only after a change occurred?
  • What is the business impact? Should we be aware of any deadlines impacted by the issue?
  • Provide a reference to the documentation being followed for the failing operation
  • Which platform are you using (OpenShift, managed OpenShift,  other Kubernetes platform)?
  • What database is used by UMS [Derby, Db2, Oracle]?

2: Gather the following configuration information

  •  If using OpenShift, provide the output of this command: 
    oc version > version.txt
  • Provide the output of this command: 
    kubectl version >> version.txt
    Note: Running the two commands above in order will produce a single versions.txt file containing both versions
  • Provide the Custom Resource(CR) .yaml file used by the operator to set up the environment 
    kubectl get icp4acluster -o yaml > config.yaml
  • Collection information about the nodes. 
    kubectl get nodes -o wide > nodes.txt
  • Collect information about the pod statuses 
    kubectl get pods > pods.txt
  • Collect information about the pod containers
    kubectl get pods -o jsonpath="{..image}" > containerInfo.txt
  • On OpenShift gather route configuration 
    kubectl get route > routes.txt
    Note: If needed, more detailed route config info can be gotten with -o yaml option
  • Collect the defined secrets 
    kubectl get secrets > secrets.txt
  • Collect the defined persistent volume claims 
    kubectl get pvc > pvc.txt
  • Collect the description and log of any pod you are having issues with (If your UMS is connected to DB2 collect log from the DB2 pod as well).
    • For 21.0.x: 
      pod = <podname>
kubectl describe pod $pod > $pod-description.txt
kubectl get pod $pod -o yaml > $pod-configuration.txt
kubectl cp $pod:/logs/UMS/$pod/  $pod-logs
    • For 20.0.x: 
      pod = <podname>
kubectl describe pod $pod > $pod-description.txt
kubectl get pod $pod -o yaml > $pod-configuration.txt

Depending on the pod (sso, scim, teams or allinone) execute:
kubectl cp $pod:/logs/application/UMS/sso/$pod/  $pod-logs
kubectl cp $pod:/logs/application/UMS/teams/$pod/  $pod-logs
kubectl cp $pod:/logs/application/UMS/scim/$pod/  $pod-logs
kubectl cp $pod:/logs/application/UMS/allinone/$pod/  $pod-logs
  • Collect the configuration mounted to the containers 
    kubectl get cm icp4adeploy-ums-configmap -o yaml > UMSConfigMap.yaml
    icp4adeploy is the name of the deployment. replace this value with your deployment name if not using default.
  • Collect the status of problematic pods. From within the terminal of the pod itself: 
    curl -sk https://localhost:9443/umshealth/rest/health/ready | jq

3: Log and Tracing data for WebSphere Liberty

Once you finish data gathering, remove the trace strings in the Custom Resource and apply the changes.

  1. Edit your Custom Resource

  2. Configure tracing in section ums_configuration.logs.trace_specification. If you are using IBM Cloud Pak for Automation 20.0.3 and you have dedicated_pods enabled, then configure tracing for each of the individual capabilities in the corresponding configuration section.

  3. To enable security tracing specify the following trace string

    • If dedicated_pods option is disabled 

      ums_configuration:
    ...
    logs:
      ...
      trace_specification: "com.ibm.ws.security.*=all:com.ibm.ws.webcontainer.security.*=all:com.ibm.oauth.*=all:com.ibm.wsspi.security.oauth20.*=all:org.apache.http.client.*=all:org.openid4java.*=all:io.openliberty.security.*=all"

    • If dedicated_pods option is enabled 

      ums_configuration:
    ...
    <pod>
      logs:
        ...
        trace_specification: "com.ibm.ws.security.*=all:com.ibm.ws.webcontainer.security.*=all:com.ibm.oauth.*=all:com.ibm.wsspi.security.oauth20.*=all:org.apache.http.client.*=all:org.openid4java.*=all:io.openliberty.security.*=all"
      where <pod> is <sso>, <scim> or <teamserver> - based on the capability that requires tracing


  4. To enable UMS tracing specify the following trace string

    • If dedicated_pods option is disabled 

        ums_configuration:
    ...
    logs:
      ...
      trace_specification: "com.ibm.dba.ums.*=all"

    • If dedicated_pods option is enabled 

        ums_configuration:
    ...
    <pod>
      logs:
        ...
        trace_specification: "com.ibm.dba.ums.*=all"
      where <pod> is <sso>, <scim> or <teamserver> - based on the capability that requires tracing
  5. To collect both, security and UMS data, combine the above trace strings  

    • If dedicated_pods option is disabled 

      ums_configuration:
    ...
    logs:
      ...
      trace_specification: "com.ibm.ws.security.*=all:com.ibm.ws.webcontainer.security.*=all:com.ibm.oauth.*=all:com.ibm.wsspi.security.oauth20.*=all:org.apache.http.client.*=all:org.openid4java.*=all:io.openliberty.security.*=all:com.ibm.dba.ums.*=all"

    • If dedicated_pods option is enabled 
       

      ums_configuration:
    ...
    <pod>
      logs:
        ...
        trace_specification: "com.ibm.ws.security.*=all:com.ibm.ws.webcontainer.security.*=all:com.ibm.oauth.*=all:com.ibm.wsspi.security.oauth20.*=all:org.apache.http.client.*=all:org.openid4java.*=all:io.openliberty.security.*=all:com.ibm.dba.ums.*=all"
      where <pod> is <sso>, <scim> or <teamserver> - based on the capability that requires tracing
      See reference MustGather: Web Single Sign-on problems with WebSphere Application Server for more details on collecting data for WebSphere Liberty.

  6. Save the Custom Resource & apply your changes

    kubectl apply -f my-cr.yaml

  7. Reproduce the issue.

  8. For every UMS pod collect log and traces by running:

    pod = <podname>
kubectl logs $pod > $pod.log
  9. Collect the trace.  In addition to using the `kubectl logs` command, if persistent storage has been setup (available starting in version 20.0.3), you may obtain the logs and trace from the persistent volume specified in the `ums_configuration.existing_claim_name` property of your configuration, under the UMS directory.  See UMS parameters for more details.  More details can be found at Where to find logs in User Management Service (UMS)
    The trace may be collected by copying using the oc command as well as long as the pod is available (where $pod refers to any pod with persistent storage enabled):
    • For 21.0.x: 
      pod = <podname>
kubectl describe pod $pod > $pod-description.txt
kubectl get pod $pod -o yaml > $pod-configuration.txt
kubectl cp $pod:/logs/UMS/$pod/  $pod-logs
    • For 20.0.x: 
      pod = <podname>
kubectl describe pod $pod > $pod-description.txt
kubectl get pod $pod -o yaml > $pod-configuration.txt

Depending on the pod (sso, scim, teams or allinone) execute:
kubectl cp $pod:/logs/application/UMS/sso/$pod/  $pod-logs
kubectl cp $pod:/logs/application/UMS/teams/$pod/  $pod-logs
kubectl cp $pod:/logs/application/UMS/scim/$pod/  $pod-logs
kubectl cp $pod:/logs/application/UMS/allinone/$pod/  $pod-logs

4: Collect the following information in case of OpenID Connect issues

  • Get a specific client: 
        curl -k -v -u <umsadmin>:<umspassword> -s https://<ums-host>/oidc/endpoint/ums/registration/yourclient
  • Get all connected clients: 
        curl -k -v -u <umsadmin>:<umspassword> -s https://<ums-host>/oidc/endpoint/ums/registration

5: User Management Service as an on-prem component

When running User Management Service in an on-prem environment, see reference Set up trace and get a full dump for WebSphere Liberty
As an additional specification, setup UMS tracing:
com.ibm.dba.ums.*=all

6: Collect Operator Logs

If you are having issues during the deployment by the operator then collect the operator logs described in the installation troubleshooting page.

7. Downloadable MustGather script

The following downloadable .zip file provides a shell script together with a readme file that have been created for you to collect data automatically. Download the  script, extract it into a location of your choice. Read the readme file for further instructions.
For UMS 21.0.x:  210x-ums-mustgather.tar
For UMS 20.0.x:  2003-ums-mustgather.tar

What to do next

  1. Review the log files and traces at the time of the problem to try to determine the source of the problem.
  2. Check these locations for known issues:
  3. Once you completed gathering all the needed info and diagnostics, you can add them to your case. Alternatively, you can upload files to ECURep. For more information, see Enhanced Customer Data Repository (ECuRep) - Overview.

Document Location

Worldwide

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7JTW","label":"IBM Digital Business Automation"},"Component":"User Management Service, UMS","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Product Synonym

CP4A;UMS

Document Information

Modified date:
06 October 2023

UID

ibm11076031

Page Feedback