Trace reference

Use this reference to enable and disable trace for integration servers or integration runtimes by creating and deleting trace objects in the Red Hat® OpenShift® web console or CLI, or the CLI for a Kubernetes environment.

Availability: Tracing is available only for integration servers whose spec.version value resolves to 12.0.2.0-r2 or later, and integration runtimes whose spec.version value resolves to 12.0.7.0-r4 or later.

Introduction
Prerequisites
Red Hat OpenShift SecurityContextConstraints requirements
Enabling trace by creating a trace object after deploying an integration
Collecting trace from an integration server or integration runtime container that is crashing
Disabling trace by deleting a trace object
Custom resource values
Supported platforms

Introduction

The Trace API enables you to enable and manage trace for a deployed integration server or integration runtime if you cannot get enough information about a particular problem from the entries that are available in the log. Trace provides more details about what is happening while code runs, and can be analyzed to discover the cause of your problem.

Two types of trace are available for an integration server or integration runtime:

User trace: Enable user trace to help you debug your integration solutions; you can trace integration servers or integration runtimes and their deployed message flows.
Service trace: Enable service trace only when you receive an error message that instructs you to do so, or when you are directed to do so by IBM® support. When active, service trace performs more comprehensive integration server or integration runtime tracing and tracks the execution of commands.

When you start user or service trace, additional processing occurs for activities in the integration server or integration runtime that you are tracing, so there might be some effect on performance while the trace is active. You can limit this additional processing by restricting how long the trace is active for.

When you deploy a trace custom resource (CR) to create a trace object, you must specify a running integration server or integration runtime on which you want to enable and run the trace. Only one trace can be enabled at a time on an integration server or integration runtime.

Tip:

If you want to capture trace on or before startup of an integration server or integration runtime, you cannot use a trace object, but must instead use the server.conf.yaml file or an environment variable to enable trace. For more information, see Enabling trace before deploying an integration from the Red Hat OpenShift web console or CLI, or the Kubernetes CLI.
You can also enable and manage user and service trace from an App Connect Dashboard instance that hosts your deployed integration servers or integration runtimes. For more information, see Enabling and managing trace for a deployed integration server and Enabling and managing trace for a deployed integration runtime. For information about deploying integration servers or integration runtimes by using the App Connect Dashboard, see Deploying Designer and Toolkit integrations to integration servers and Deploying Designer and Toolkit integrations to integration runtimes.

Prerequisites

If using Red Hat OpenShift, Red Hat OpenShift Container Platform 4.12 or 4.14 is required.
Note: For information about the custom resource (or operand) versions that are supported for each Red Hat OpenShift version, see spec.version values.
If using a Kubernetes environment, Kubernetes 1.25.x–1.29.x is required.
The IBM App Connect Operator must be installed in your cluster either through an independent deployment or an installation of IBM Cloud Pak for Integration. For further details, see the following information:
Ensure that you have cluster administrator authority or have been granted the appropriate role-based access control (RBAC).
If you want to use the command-line interface (CLI) to log in to your cluster and run commands to create and manage your IBM App Connect instances and other resources, ensure that the required CLI tools are installed on your computer. For more information, see Installing tools for managing the cluster, containers, and other resources (on Red Hat OpenShift), or Installing tools for managing the cluster, containers, and other resources (on Kubernetes).
You must have a Kubernetes pull secret called ibm-entitlement-key in the namespace before creating the instance. For more information, see Obtaining and applying your IBM Entitled Registry entitlement key.

Red Hat OpenShift SecurityContextConstraints requirements

IBM App Connect runs under the default restricted SecurityContextConstraints.

Enabling trace by creating a trace object after deploying an integration

To enable (or start) user or service trace on a deployed integration server or integration runtime, you can create a trace object by using the Red Hat OpenShift web console or CLI, or the CLI for a Kubernetes environment.

Restriction:

After you create a trace object, you cannot update its custom resource settings. If you want to change the type of trace that is currently active for an integration server or integration runtime, you must delete the existing trace object and then create a new trace object of the required type.
You cannot use a trace CR to capture trace for a serverless integration runtime deployment. If you require trace, you can deploy the serverless integration runtime with a configuration object of type server.conf.yaml with defined settings to enable trace, or deploy the serverless integration runtime with the MQSI_FORCE_DEBUG_TRACING environment variable. For more information, see Enabling trace before deploying an integration from the Red Hat OpenShift web console or CLI, or the Kubernetes CLI.
If a pod is restarted or a new replica is added while trace is active, trace is started on that pod within 60 seconds because the trace reconcile loop is scheduled to run every minute. However, there is a possibility that some trace data might be lost during this interval.

Before you begin

Ensure that the Prerequisites are met.

Namespace restriction for an instance, server, configuration, or trace:

The namespace in which you create an instance or object must be no more than 40 characters in length.

Create options:

Creating a trace object from the Red Hat OpenShift web console
Creating a trace object from the Red Hat OpenShift or Kubernetes CLI

Creating a trace object from the Red Hat OpenShift web console

To create a trace object by using the Red Hat OpenShift web console, complete the following steps:

From a browser window, log in to the Red Hat OpenShift Container Platform web console. Ensure that you are in the Administrator perspective .
From the navigation, click Operators > Installed Operators.
If required, select the namespace (project) in which you installed the IBM App Connect Operator.
From the Installed Operators page, click IBM App Connect.
From the Operator details page for the App Connect Operator, click the Trace tab. Any previously created trace objects are displayed in a table.
Click Create Trace.
From the Details tab on the Operator details page, you can also locate the Trace tile and click Create instance to specify custom resource settings for the trace object.
To use the Form view, complete the fields as follows:
- Name: Specify a unique short name by which the trace object can be identified.
- Trace type: From the drop-down list, select the type of trace that you want to collect. Valid values are user and service. For more information about each trace type, see Introduction.
- Integration server name: Specify the name of a deployed integration server that you want to collect trace for. This name is the metadata.name value in the integration server CR.
  Note: Leave this field blank if you are creating a trace object for an integration runtime.
- Integration runtime name: Specify the name of a deployed integration runtime that you want to collect trace for. This name is the metadata.name value in the integration runtime CR.
  Note: Leave this field blank if you are creating a trace object for an integration server.
Optional: If you prefer to use the YAML view, click YAML view and then update the contents of the YAML editor with the parameters and values that you require for this trace object.
To view the full set of parameters and values available, see Custom resource values.
The following YAML code shows an example of what the custom resource settings for your trace object should look like for an integration server user trace.
```
apiVersion: appconnect.ibm.com/v1beta1
kind: Trace
metadata:
  name: trace-user-is-fd-toolkit
  namespace: mynamespace
spec:
  integrationServerName: is-fd-toolkit
  type: user
```
The following YAML code shows an example of what the custom resource settings for your trace object should look like for an integration server service trace.
```
apiVersion: appconnect.ibm.com/v1beta1
kind: Trace
metadata:
  name: trace-service-is-fd-toolkit
  namespace: mynamespace
spec:
  integrationServerName: is-fd-toolkit
  type: service
```
The following YAML code shows an example of what the custom resource settings for your trace object should look like for an integration runtime user trace.
```
apiVersion: appconnect.ibm.com/v1beta1
kind: Trace
metadata:
  name: trace-user-ir-fd-toolkit
  namespace: mynamespace
spec:
  integrationRuntimeName: ir-fd-toolkit
  type: user
```
The following YAML code shows an example of what the custom resource settings for your trace object should look like for an integration runtime service trace.
```
apiVersion: appconnect.ibm.com/v1beta1
kind: Trace
metadata:
  name: trace-service-ir-fd-toolkit
  namespace: mynamespace
spec:
  integrationRuntimeName: ir-fd-toolkit
  type: service
```
Click Create. An entry for the trace object is shown in the Traces table with a Ready status.
Click the trace object name to view information about its definition. On the Events tab, you should see a notification that the trace has started for the integration server or integration runtime pod as shown in the following example.

You can use the breadcrumb trail to return to the (previous) Operator details page for Traces. The status for the trace object is shown as Ready in the Traces table.

Tip: If you enabled trace for an integration server and then access the Servers page of your App Connect Dashboard instance after the trace object is created, you should notice that the integration server tile shows a Ready (trace enabled) status. Similarly, if you enabled trace for an integration runtime and then access the Runtimes page of your App Connect Dashboard instance after the trace object is created, the integration runtime tile shows the status as Ready (trace enabled).

Integration server tile in the App Connect Dashboard with a "Ready (trace enabled)" status

What to do next

When you have collected some trace information, you can download the trace to your computer for analysis. You can do so from your App Connect Dashboard instance as described in Downloading the user or service trace log files (for an integration server) or Downloading the user or service trace log files (for an integration runtime).
If you want to erase the trace information that has been written to an integration server or integration runtime's trace log files, you can use a reset option. You can do so from your App Connect Dashboard instance as described in Clearing user or service trace information (for an integration server) or Clearing user or service trace information (for an integration runtime).

Creating a trace object from the Red Hat OpenShift or Kubernetes CLI

To create a trace object from the Red Hat OpenShift CLI, complete the following steps.

Note: These instructions relate to the Red Hat OpenShift CLI, but can be applied to a Kubernetes environment by using the relevant command to log in to the cluster, and substituting oc with kubectl where appropriate.

From your local computer, create a YAML file that contains the custom resources settings for the trace object that you want to create. Include the metadata.namespace parameter to identify the namespace in which you want to create the trace; this should be the same namespace where the integration server or integration runtime is created.
To view the full set of parameters and values available, see Custom resource values.
The following YAML code shows an example of what the custom resource settings for your trace object should look like for an integration server user trace.
```
apiVersion: appconnect.ibm.com/v1beta1
kind: Trace
metadata:
  name: trace-user-is-fd-toolkit
  namespace: mynamespace
spec:
  integrationServerName: is-fd-toolkit
  type: user
```
The following YAML code shows an example of what the custom resource settings for your trace object should look like for an integration server service trace.
```
apiVersion: appconnect.ibm.com/v1beta1
kind: Trace
metadata:
  name: trace-service-is-fd-toolkit
  namespace: mynamespace
spec:
  integrationServerName: is-fd-toolkit
  type: service
```
The following YAML code shows an example of what the custom resource settings for your trace object should look like for an integration runtime user trace.
```
apiVersion: appconnect.ibm.com/v1beta1
kind: Trace
metadata:
  name: trace-user-ir-fd-toolkit
  namespace: mynamespace
spec:
  integrationRuntimeName: ir-fd-toolkit
  type: user
```
The following YAML code shows an example of what the custom resource settings for your trace object should look like for an integration runtime service trace.
```
apiVersion: appconnect.ibm.com/v1beta1
kind: Trace
metadata:
  name: trace-service-ir-fd-toolkit
  namespace: mynamespace
spec:
  integrationRuntimeName: ir-fd-toolkit
  type: service
```
Save this file with a .yaml extension; for example, myintserver_usertrc_cr.yaml.
From the command line, run the following command to create the trace object in your Red Hat OpenShift cluster. (Use the name of the .yaml file that you created.)
```
oc apply -f myintserver_usertrc_cr.yaml
```

Run the following command to check the status of the trace object and verify that it is ready:

oc get traces -n namespace

You should see output similar to this.

NAME                       AGE    TRACETYPE   INTEGRATIONSERVER   STATUS
trace-user-is-fd-toolkit   139m   user        is-fd-toolkit       Ready

What to do next

When you have collected some trace information, you can download the trace to your computer for analysis. You can do so from your App Connect Dashboard instance as described in Downloading the user or service trace log files (for an integration server) or Downloading the user or service trace log files (for an integration runtime).
If you want to erase the trace information that has been written to an integration server or integration runtime's trace log files, you can use a reset option. You can do so from your App Connect Dashboard instance as described in Clearing user or service trace information (for an integration server) or Clearing user or service trace information (for an integration runtime).

Collecting trace from an integration server or integration runtime container that is crashing

If an integration server or integration runtime container is crashing, you can use an environment variable to prevent the container from stopping so that the trace can be collected as normal. To do so, you must edit the integration server or integration runtime CR to set an environment variable as follows:

Integration server: Use the spec.env parameter to set an environment variable named MQSI_PREVENT_CONTAINER_SHUTDOWN to true.
```
spec:
  env:
    - name: MQSI_PREVENT_CONTAINER_SHUTDOWN
      value: 'true'
```
Integration runtime: Use the spec.template.spec.containers[].env parameter (for a named container; for example, runtime) to set an environment variable named MQSI_PREVENT_CONTAINER_SHUTDOWN to true.
```
spec:
  template:
    spec:
      containers:
        - name: runtime
          env:
            - name: MQSI_PREVENT_CONTAINER_SHUTDOWN
              value: 'true'
          ...
```

For information about editing an integration server or integration runtime's CR, see Integration Server reference: Updating the custom resource settings for an instance or Integration Runtime reference: Updating the custom resource settings for an instance.

Disabling trace by deleting a trace object

To disable (or stop) an active user or service trace, you can delete the trace object by using the Red Hat OpenShift web console or CLI, or the CLI for a Kubernetes environment. When you delete a trace object, the trace log files are retained in the system. For information about the location of these files, see Downloading the user or service trace log files (for an integration server) or Downloading the user or service trace log files (for an integration runtime). Also note that if you delete an integration server or integration runtime, any trace object that references the integration server or integration runtime (in spec.integrationServerName or spec.integrationRuntimeName) is also automatically deleted.

Before you begin

Ensure that you have cluster administrator authority or have been granted the appropriate role-based access control (RBAC).

Delete options:

Deleting a trace object from the Red Hat OpenShift web console
Deleting a trace object from the Red Hat OpenShift or Kubernetes CLI

Deleting a trace object from the Red Hat OpenShift web console

To delete a trace object by using the Red Hat OpenShift web console, complete the following steps:

From a browser window, log in to the Red Hat OpenShift Container Platform web console. Ensure that you are in the Administrator perspective .
From the navigation, click Operators > Installed Operators.
If required, select the namespace (project) in which you installed the IBM App Connect Operator.
From the Installed Operators page, click IBM App Connect.
From the Operator details page for the App Connect Operator, click the Trace tab.
Locate the trace object that you want to delete.
Click the options icon () to open the options menu, and then click the Delete option.
Confirm the deletion.

Deleting a trace object from the Red Hat OpenShift or Kubernetes CLI

To delete a trace object from the Red Hat OpenShift CLI, complete the following steps.

From the command line, log in to your Red Hat OpenShift cluster by using the oc login command.
From the namespace where the trace object is deployed, run the following command to delete this object, where traceName is the value of the metadata.name parameter.
```
oc delete trace traceName
```

Custom resource values

The following table lists the configurable parameters and default values for trace objects.

Parameter	Description	Default
apiVersion	The API version that identifies which schema is used for this object.	`appconnect.ibm.com/v1beta1`
kind	The resource type.	`Trace`
metadata.name	A unique short name by which the trace object can be identified. If the trace object is created from the Red Hat OpenShift web console or CLI, or the CLI for a Kubernetes environment, you can specify a preferred name. If the trace object is created by starting a trace from an integration server or integration runtime's tile in the App Connect Dashboard, the integration server or integration runtime's name is automatically assigned to the trace.
metadata.namespace	The namespace (project) in which the trace object and the referenced integration server or integration runtime are created. The namespace in which you create an instance or object must be no more than 40 characters in length.
spec.integrationRuntimeName	The name of a deployed integration runtime that you want to collect trace for.
spec.integrationServerName	The name of a deployed integration server that you want to collect trace for.
spec.type	The type of trace. Valid values are `user` and `service`. For more information about each trace type, see Introduction.	`service`

Supported platforms

Red Hat OpenShift: Supports the amd64, s390x, and ppc64le CPU architectures. For more information, see Supported platforms.

Kubernetes environment: Supports only the amd64 CPU architecture. For more information, see Supported operating environment for Kubernetes.