Mustgather: Collecting data to diagnose issues with Workflow Process Service (WfPS) runtime

Troubleshooting

Problem

This document describes the general information and diagnostic data needed to start troubleshooting issues related to Workflow Process Service (WfPS) containers. Include the diagnostics retrieved from using this document when you open a case for problems related to Workflow Process Service.

Resolving The Problem

Detailed diagnostic collection steps

These steps are the detailed steps to gather different types of data for WfPS. When you run the diagnostic commands, run them from an empty collection directory to make it easy to package the files. Run the commands from the project or namespace containing WfPS or use the -n <namespace> flag with all oc commands.

Note: oc commands are interchangeable with kubectl.

1: Provide a detailed description of the problem and your environment

Provided a detailed description of your issue. Include screen captures and re-create steps if possible.
Is it an intermittent or recreatable issue? Has this issue always been a problem or one that started only after a change occurred?
What is the business impact? Are there any deadlines impacted by the issue?
Provide a reference to the documentation being followed for the failing operation
Which platform are you using (Red Hat OpenShift, managed Red Hat OpenShift, other Kubernetes platform)?
What is the database type and version?

2: Gather the configuration information

Gather the general configuration data

oc get icp4acluster -o yaml > CP4BAconfig.yaml
oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -m automationfoundation -n <cloud pak namespace>

The -n parameter is required and must be a single namespace. If you are using an air gap setup, ensure you push the latest version of the must-gather image into your local repository. The command requires cluster admin access to execute. Generally, this collection takes 5 - 10 minutes and produces a 25 - 50MB gzip file.

If you have trouble with this see item 2 of the main Cloud Pak MustGather for more details.

Preferred (23.0.1 or higher): If you want to gather the configuration data from the WfPS pattern in CP4BA, execute below command:

oc adm must-gather --image=icr.io/cpopen/cp4ba/icp4a-must-gather:23.0.2 -- gather -m cp4ba –p wfps -n <cloud pak namespace>

3: Log and Tracing data for WebSphere Liberty

Follow these steps to enable IBM WebSphere Liberty tracing on the WfPS containers.

Edit the WfPSRuntime CR yaml used by the WfPS operator create the WfPS pods.
Modify the node.logging.traceSpecification property in the CR yaml and set the trace string or a trace that fits your problem.
```
spec:
  ...
  node:
    ...
    logging:
      traceSpecification: '*=info:WLE.*=all:com.ibm.bpm.*=all:com.ibm.workflow.*=all'
```
Update the WfPSRuntime CR with the new configuration by using your preferred method. For example, the edit command can be used
```
oc edit WfPSRuntime <wfps instance name>
```
Note: It can take some of time to recognize the change (length of an operator reconcile) and update the configuration. You can grep the log file for traceSpecification to see when the trace settings change.
The following command can be used to gather the WfPS logs where pod name is one of the WfPS pods.
```
oc cp <pod-name>:/logs/application/ ./WfPS
```
Disable the trace by setting traceSpecification back to "*=info" and applying the changes again.

4: Export of your application

If the issue is specific to a certain application, provide an export of that application.

For more information, see Importing and exporting projects.

5: Collect Browser data for UI issues

For console or web application usage issues, capture the following browser data:

Network traffic capture export in .har or .saz format. See Collect a HTTP traffic capture with Fiddler or your web browser.
Export the browser console log.
Open the browser developer tools (can be accessed by pressing F12), and copy the contents of the console tab.

6: Gathering javacores and heap dumps

For issue related to performance, hangs, jvm crashes, or memory issues, we likely need to get dumps from the liberty servers.

Determine the names of the WfPS server pods by using the get pods command
```
oc get pods | grep wfps-runtime-server
```
If dumps need to be generated, you can use the Liberty server dump commands to create them. Use the javadump command to generate javacores for each WfPS server pod. Include the option --include=heap or --include=system to generate heap dumps or system core dumps. For example, the following command generates a javacore and heap dump for the pod.
```
oc exec <podname> -- bash -c "server javadump defaultServer --include=heap" 
```
Note: If a WfPS Liberty server JVM crashes, then you might also see dumps get generated.
The following command can be used to gather the WfPS dumps where pod name is one of the WfPS pods.
```
oc cp <pod-name>:/opt/ibm/wlp/output/defaultServer/dump ./WfPS/dumps/
```

What to do next

Review the log files and traces at the time of the problem to try to determine the source of the problem.
Check these locations for known issues:
- Search in the Cloud Pak for Automation Support Page.
- Review the Business Automation Workflow documentation.
Once you completed gathering all the needed information and diagnostics, you can add them to your case. Alternatively, you can upload files to ECURep. For more information, see Enhanced Customer Data Repository (ECuRep) - Overview.

Related Information

Cloud Pak for Automation MustGather

On-Prem Business Automation Workflow MustGathers

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB10","label":"Data and AI"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS2JQC","label":"IBM Cloud Pak for Automation"},"ARM Category":[{"code":"a8m3p000000hBFVAA2","label":"Operate-\u003EWfPS Install\\Upgrade\\Setup"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Tips