IBM MQ Tracing Support (IBM-MQ-TRACING-USER-EXITS)

• mqtracingexit_r (Linux and AIX) / mqtracingexit.dll (Windows): The IBM MQ user exits, which intercept the IBM MQ API calls and invokes the wrapped OpenTelemetry C++ client library to create spans.
• tracelibrary.so (Linux and AIX) / tracelibrary.dll (Windows): The wrapped OpenTelemetry C++ client, which provides functions to manage the lifecycle of spans, and sends spans to the target-tracing system.
• mqtracingexit.conf: The configuration file to specify the target-tracing system (Jaeger or Instana) and the parameters for span generation.

You need to edit mqs.ini or qm.ini file to configure user exits. The mqs.ini file contains information relevant to all the queue managers on a particular node. The qm.ini file contains information relevant to a specific queue manager. The mqs.ini file is located at /var/mqm on Linux and AIX, on Windows it is located in C:\ProgramData\IBM\MQ. The qm.ini file is located at /var/mqm/qmgrs/<QMNAME> on Linux and AIX, on Windows it is located in C:\ProgramData\IBM\MQ\qmgrs\<QMNAME>.

Complete the following steps to enable tracing for all the queue managers on a particular node:

Add the following content to enable user exits for all queue managers on this node. When any queue manager starts, the attributes in this stanza are read, and then overridden by the user exits defined in qm.ini.
Linux and AIX:

ApiExitCommon:
   Sequence=100
   Function=EntryPoint
   Module=/var/mqm/exits64/mqtracingexit
   Name=TracingApiExit

Windows:

ApiExitCommon:
   Sequence=100
   Function=EntryPoint
   Module=C:\ProgramData\IBM\MQ\exits64\mqtracingexit
   Name=TracingApiExit

Complete the following steps to enable tracing for a particular queue manager or to change user exits settings for a particular queue manager.

Add the following content on Linux and AIX:

ApiExitLocal:
   Sequence=100
   Function=EntryPoint
   Module=/var/mqm/exits64/mqtracingexit
   Name=TracingApiExit

Add the following content on Windows:

ApiExitCommon:
   Sequence=100
   Function=EntryPoint
   Module=C:\ProgramData\IBM\MQ\exits64\mqtracingexit
   Name=TracingApiExit

Where:

• Sequence identifies the sequence of this exit in relation to other exits. An exit with a low sequence number is called before an exit with a higher sequence number.
• Function identifies the name of the function entry point into the module containing the user exits code.
• Module contains the user exits code. If this field contains the full path name of the module, it is used as is. If this field contains only the module name, the module is located by using the ExitsDefaultPath attribute in the ExitPath in qm.ini.
• Name is the descriptive name of the user exits.

For more information, see Configuring API exits in IBM MQ document library.

You need to save changes to the configuration files, and restart queue managers for the changes to take effect.

Configuring IBM MQ tracing

2. Edit mqtracingexit.conf file.

LOG_LEVEL="info" #Log level: info, warn, error, debug
SPAN_FORMAT="instana" #specify opentracing span format as instana or jaeger
MONITOR_LEVEL="normal" #MQ tracing level: normal, quiet, off, debug
JAEGER_ENDPOINT = "http://127.0.0.1:14268/api/traces"
JAEGER_SAMPLER_TYPE = "const"
JAEGER_SAMPLER_PARAM = 1.0
IBMMQ_PUBSUB_SUPPORT = "off" #The switch for the publish/subscribe support: on, off.
#IBMMQ_RFH2_HEADERS_TO_EXPOSE = "" #A list of message headers that should be included in the span payload, with each header separated by commas.
#JAEGER_AGENT_HOST = "127.0.0.1"
#JAEGER_AGENT_PORT = 6831 
#INSTANA_AGENT_HOST = "localhost" # instana agent host, localhost will be used by default
#INSTANA_AGENT_PORT = 42699 # instana agent port, 42699 will be used by default
#INSTANA_AGENT_PROTO = "http" # connection type between mq tracing exit and instana agent, http and https are supported, http will be used by default
#HOST_ALIAS="<YOUR-HOST-NAME>" # host name to match with instana agent configuration.yaml

Where:

• LOG_LEVEL specifies the log level, which can be one of info, warn, error and debug. The log files are located at /tmp/ibmmqtrace directory on Linux and AIX. The log files are located at C:\ProgramData\IBM\MQ\trace directory on Windows.
• SPAN_FORMAT specifies where the span data is sent to. The SPAN format can be set to instana or jaeger. If SPAN_FORMAT is set to instana, IBM MQ Tracing user exits sends span data to the Instana host agent endpoint http://localhost:42699 by default. In such a case, you can ignore JAEGER_* parameters in mqtracingexit.conf file. Update the configuration items INSTANA_AGENT_HOST and INSTANA_AGENT_PROTO, if you want to send the span data to a remote host agent by using https protocol. If SPAN_FORMAT is set to Jaeger, you need to configure JAEGER_* parameters. In such a case, you can ignore INSTANA_* parameters. You must use the same SPAN_FORMAT setting for all IBM MQ nodes.
• MONITOR_LEVEL specifies the tracing level of IBM MQ, which can be one of off, quiet, normal and debug. If MONITOR_LEVEL is set to off, no tracing data will be collected. If it is set to quiet, the trace correlation information will not be added to all the messages, but the messages can still be traced in the MQ server. If it is set to normal, the trace correlation information will be added to the messages with RFH2 headers or named properties, but will not be added to text messages. If it is set to debug, the trace correlation information will be added to all traced messages. If you encounter the MQRC_HEADER_ERROR (MQRC 2142) error in your message consumer application, you can resolve it by changing the MONITOR_LEVEL property to normal or quiet, or by changing the PROPCTL property of the message destination queues to NONE. Alternatively, you can modify your application to ignore the additional RFH2 header with trace correlation information added by Instana, to avoid the message header error.
• JAEGER_ENDPOINT specifies Jaeger collector endpoint. You can choose to send tracing data to Jaeger collector or to the Jaeger agent. If you want to send tracing data to Jaeger agent, comment this line and specify values for configuration item JAEGER_AGENT_HOST and JAEGER_AGENT_PORT.
• JAEGER_SAMPLER_TYPE specifies the sample type. The value can be one of const, probabilistic, ratelimiting, or remote.
• JAEGER_SAMPLER_PARAM is interpreted based on the value of JAEGER_SAMPLER_TYPE. See more details at https://www.jaegertracing.io/docs/1.22/sampling/#client-sampling-configuration.
• JAEGER_AGENT_HOST specifies Jaeger agent host.
• JAEGER_AGENT_PORT specifies Jaeger agent port.
• IBMMQ_PUBSUB_SUPPORT controls whether the support for tracing publish and subscribe messages is enabled. Set it to 'on' to enable the support, and set it to 'off' to disable the support.
• IBMMQ_RFH2_HEADERS_TO_EXPOSE specifies a list of message headers that should be included in the span payload, with each header separated by commas. For example, IBMMQ_RFH2_HEADERS_TO_EXPOSE="rfh2_folder1.header1, rfh2_folder2.header2, header3". To avoid creating a span size that is too large and negatively affecting overall performance, any message header with a size larger than 256 bytes will be truncated.  Additionally, the total length of all collected message headers in a span cannot exceed 3K bytes. It is recommended that only the most important headers be configured for collection.
• INSTANA_AGENT_HOST specifies the agent host where the Instana format span data is sent to, and localhost is used by default. If you specify a remote agent host, you also need to add a line http.listen=* in <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg for the remote host agent first as the host agent is not reachable from other hosts by default.
• INSTANA_AGENT_PORT specifies the port where the Instana agent is listening  on, and 42699 is used by default. You only need to change the port when the agent is listening on another port.
• INSTANA_AGENT_PROTO specifies the connection type between IBM MQ tracing exits and the host agent. http and https are supported, and http is used by default. If you want to change it to https, you need to follow https://www.ibm.com/docs/en/obi/current?topic=agent-host-configuration#setting-up-tls-encryption-for-agent-endpoint to secure the agent endpoint first.
• HOST_ALIAS specifies a host alias for the span data collected by IBM MQ Tracing user exits, so we can link calls to IBM MQ to the infrastructure entity if the Queue Manager is also monitored by the IBM MQ sensor. The FQDN of IBM MQ host is used by default. The host alias value must match with the IBM MQ host specified in host agent configuration yaml. You only need to specify a host alias if the FQDN of IBM MQ host is not used in IBM MQ sensor configuration when the host agent is not on the local IBM MQ host. The IBM MQ sensor can discover the FQDN for the local Queue Managers.

Trace correlation

Upgrade