Monitoring IBM MQ

Supported operating systems

• The supported operating systems of IBM MQ sensor are consistent with host agents requirements, which can be checked in the Supported operating systems section of the Windows, Unix, or Linux host agent:

  • Supported operating systems for Linux
  • Supported operating systems for Unix
  • Supported operating systems for Windows

• The following table outlines the privileged users by platform for local binding:

  Table 1. Privileged users by platform

  Platform	Privileged users
  Windows	- SYSTEM - Members of the mqm group - Members of the Administrators group
  Linux	Members of the mqm group

• For the supported operating systems of IBM MQ Tracing, see Supported operating systems for IBM MQ Tracing.

Supported versions and support policy

The sensor supports IBM MQ 8.0.0 and later.

Instana IBM MQ Tracing supports IBM MQ 9.0.0 and later.

The following table shows the latest supported version and support policy:

For more information about the support policy, see Support strategy for sensors.

Supported client-side tracing

Instana supports client-side tracing for Java.

The configuration steps in the Configuring section are for monitoring and collecting IBM MQ metrics. You can continue to use IBM MQ Tracing on your IBM MQ hosts as you need.

To use IBM MQ server-side tracing, you must install and enable IBM MQ Tracing first. For detailed steps, see IBM MQ Tracing. For IBM i servers, see IBM MQ Tracing on IBM i.

Replicated data queue manager (RDQM) support

You need to install the Instana host agent on each RDQM host. The host must be a virtual machine or physical machine of the Linux platform. Then, Instana automatically discovers and monitors the running and standby Queue Manager instances through local monitoring. The Instana UI displays only 1 RDQM queue manager name with the aggregated data of 3 Queue Manager instances. You can view the information about HA Type, HA Role, HA Preferred Location, HA Floating IP address, Active Node, and Standby Nodes values on the dashboard of the running queue manager in the Instana UI.

Multi-instance queue managers support

All the distributed platforms and Kubernetes environments are supported. For the distributed platform, you need to install the Instana host agent on each Queue Manager instance of the multi-instance host. For the Kubernetes environment, you need to install the Instana host agent in the Kubernetes environment. Then, Instana automatically discovers and monitors queue managers that are in running, standby, and elsewhere status through local monitoring. The Instana UI displays only 1 multi-instance queue manager name with the aggregated data of the other Queue Manager instances. You can view the information about HA Type, Active Node, Standby Nodes, and Elsewhere Nodes values on the dashboard of the running queue manager in the Instana UI.

Native HA queue manager support

You need to install the Instana host agent in the Kubernetes environment. Then, Instana automatically discovers and monitors queue managers that are in running and replica status through local monitoring. The Instana UI display only 1 Native HA queue manager name with the aggregated data of the other Queue Manager instances. You can view the information about HA Type, Active Node, and Standby Nodes on the dashboard of the running queue manager in the Instana UI.

Single-instance queue manager in Red Hat OpenShift Container Platform support

You need to install the Instana host agent on the Kubernetes environment. Then, Instana automatically discovers and monitors the running Queue Manager instance through local monitoring. The Instana UI displays the information about HA Type and Active Node of this Queue Manager instance on the dashboard.

Local binding mode and client binding mode

Local and client binding modes refer to the connection mode that IBM MQ sensor connects to queue manager.

Local monitoring and remote monitoring

Local and remote monitoring methods show whether IBM MQ sensor is in the same environment as queue manager. If IBM MQ sensor and queue manager are in the same environment, this is called local monitoring. If they are not in the same environment, this is called remote monitoring.

The collection of monitoring metrics is decided by the connection modes: local binding mode or client binding mode. The local binding mode can collect more metrics in local files.

Support matrix of IBM MQ connection mode

The following table outlines the support matrix for IBM MQ connection mode:

Monitoring IBM MQ in a virtual or physical machine

When the Instana host agent and IBM MQ run in the same machine, two connection modes are supported: local binding mode and client binding mode.

Monitoring IBM MQ in a Kubernetes cluster

When the Instana host agent and IBM MQ run in the same Kubernetes cluster, only local monitoring with the client binding mode is supported.

com.instana.plugin.ibmmq:
  enabled: true
  poll_rate: 60
  queueManagers:
    QMGR03:    # Your Queue Manager name here. If there are queue managers with the same name, it is required to append '-<instance>' in the queue manager name to distinguish them. You can select any string for <instance>.
      channel: '<INSERT_CHANNEL_HERE>' # Remote administration channel

Monitoring IBM MQ in Docker

When the Instana host agent and IBM MQ run in different Docker containers in the same machine, only local monitoring with the client binding mode is supported because Instana host agent and IBM MQ run in different Docker operating systems. In such scenario, you need to configure IBM MQ connection parameters in the agent configuration file. Because IBM MQ sensor discovers IBM MQ Queue Manager instances automatically, you don't need to set the host and port in the agent configuration file.

Extra IBM MQ configurations

To collect specific IBM MQ metrics, you must enable queue statistics, queue manager performance events, or channel events. Follow these steps to configure the necessary settings:

1. To collect the queue metrics Last Put Time, Last Get Time, Oldest Message Time, and On Queue Message Time, enable queue statistic at either the queue level or the queue manager level.

• At the queue level, run the following RUNMQSC command:

  alter ql(QUEUE_NAME) MONQ(LOW)
  

• At the queue manager level, run the following RUNMQSC command:

  alter qmgr MONQ(LOW)
  

  For more information about the alter qmgr command, see IBM MQ documents.

2. To collect the IBM MQ performance built-in events, such as Queue Depth High, Queue Full, and Queue Service Interval High, you must enable the queue manager performance events (PERFMEV) by running the following RUNMQSC command:

   alter qmgr PERFMEV(ENABLED) CHLEV(ENABLED)
   

   1. To collect built-in events that are related to queue manager, such as Alias Base Queue Type Error, Default Transmission Queue Type Error, Default Transmission Queue Usage Error, Get Inhibited, Not Authorized, Put Inhibited, Queue Manager Not Active, Queue Type Error, Remote Queue Name Error, Transmission Queue Type Error, Transmission Queue Usage Error, Unknown Alias Base Queue, Unknown Default Transmission Queue, Unknown Object Name, Unknown Remote Queue Manager, and Unknown Transmission Queue, you must enable the following channel events (CHLEV) events:

      • AUTHOREV (Authorization events)
      • INHIBTEV (Inhibition events)
      • LOCALEV (Local events)
      • REMOTEEV (Remote events)
      • STRSTPEV (Storage events)

      Run the following RUNMQSC command:

      alter qmgr AUTHOREV(ENABLED) INHIBTEV(ENABLED) LOCALEV(ENABLED) REMOTEEV(ENABLED) STRSTPEV(ENABLED)
      

      For more information about controlling queue manager events, see Controlling queue manager events.

   2. To collect built-in events that are related to channel, such as Bridge stopped, Channel Auto-definition Error, Channel Blocked, Channel Conversion Error, Channel Not Activated, Channel Not Available, Channel SSL Error, Channel SSL Warning, Channel Stopped, and Channel Stopped By User, you must enable the following channel-related events (CHLEV) events:

      • CHLEV (Channel events)
      • BRIDGEEV (Bridge events)
      • SSLEV (SSL events)
      • CHADEV (Channel auto-definition events)

      Run the following RUNMQSC command:

      alter qmgr CHLEV(ENABLED) BRIDGEEV(ENABLED) SSLEV(ENABLED) CHADEV(ENABLED)
      

      For more information about controlling channel events, see Controlling channel and bridge events.

   For more information about the alter qmgr command, see IBM MQ documents.

3. To collect MQ queue statistics metrics such as Persistent Put Bytes, Nonpersistent Put Bytes, Persistent Get Bytes, Nonpersistent Get Bytes, Expired Msg Count, Put Fail Count, Put1 Fail Count, and Get Fail Count, the following steps are necessary from IBM MQ:

   1. At the queue manager level, enable queues statistics data collection for all queues by running the following command:

      alter qmgr STATQ(ON)
      

      At the queue level, enable statistics for a specific queue by running the following Queue_Name command:

      alter qlocal(Queue_Name) STATQ(ON)
      

      You can also change the queue statistics interval by using the IBM MQ Queue Manager parameter STATINT, which is 1800 seconds by default. For example, to set it to 900 seconds, run the following command:

      alter qmgr STATINT(900)
      

   2. To collect statistics data from the SYSTEM.ADMIN.STATISTICS.QUEUE, you need to grant the get authority to the Instana agent user because the statistics data is collected from this queue. For example, run the following command:

      setmqaut -m QmgrName -n SYSTEM.ADMIN.STATISTICS.QUEUE -t q -p user +get
      

      For more information about the setmqaut command, see IBM MQ documents.

Configuring IBM MQ authority

• Using privileged user for the Instana agent

  If the Instana agent is installed by using a privileged user, it has all the necessary authorities. Therefore, you can view all the monitoring data in the Instana UI.

• Using non-privileged user for the Instana agent

  When the Instana agent is installed by using a non-privileged user to connect to queue manager, you must grant appropriate Object Authority Manager (OAM) authorities to the user to query all the monitoring data.

  To provide the IBM MQ OAM authorities to a non-privileged user, use the setmqaut control command. The user that you use to issue the setmqaut command must be a privileged user. The setmqaut command is in the $MQ_INSTALLED_DIR/bin directory.

  The user that you use to issue the setmqaut command must be a privileged user.

1. To grant the user the appropriate authorities to access queue manager that you want to monitor, run the following command:

   setmqaut -m QMGR -t qmgr -p UserID +inq +connect +dsp
   

   where:

   • QMGR: Name of Queue Manager
   • UserID: UserID of the user.

   You must specify the fully qualified user for the -p option. For example, -p user@domain or -p user@host.

   To specify a user group name, replace the -p option with the -g option in the command.

2. To provide the appropriate authorities to the user to access the system queues of queue manager, run the following commands:

   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.COMMAND.QUEUE -p UserID +inq +dsp +chg +get +put
   setmqaut -m QMGR -t q -n SYSTEM.DEFAULT.MODEL.QUEUE -p UserID +dsp +get
   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.QMGR.EVENT -p UserID +inq +dsp +chg +get
   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.PERFM.EVENT -p UserID +inq +dsp +chg +get
   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.CHANNEL.EVENT -p UserID +inq +dsp +chg +get
   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.STATISTICS.QUEUE -p UserID +inq +dsp +chg +get
   setmqaut -m QMGR -t q -n SYSTEM.AUTH.DATA.QUEUE -p UserID +dsp
   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.LOGGER.EVENT -p UserID +inq +dsp +chg
   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.CONFIG.EVENT -p UserID +inq +dsp +chg
   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.COMMAND.EVENT -p UserID +inq +dsp +chg
   

3. For the Instana UI to display data, the user specified by UserID needs the display access to various objects. To grant the appropriate authorities, run the following commands:

   setmqaut -m QMGR -t q -n "**" -p UserID +dsp +chg
   setmqaut -m QMGR -t channel -n "**" -p UserID +dsp
   setmqaut -m QMGR -t clntconn -n "**" -p UserID +dsp
   setmqaut -m QMGR -t listener -n "**" -p UserID +dsp
   setmqaut -m QMGR -t topic -n "**" -p UserID +dsp
   

4. To ensure all the commands work, restart queue manager or refresh security:

   echo "REFRESH SECURITY" | runmqsc QMGR
   

These permissions cover all the necessary access rights for IBM MQ sensors to query and gather monitoring data. Typically, read-only access is adequate for most monitoring data needs. However, for some of the following metrics, additional permissions are required. The permissions must be adjusted accordingly to accommodate these requirements based on your required monitoring metrics.

• Queue reset statistics data:

  To retrieve queue reset statistics data, queue objects require the change authority (+chg).

• Statistics data from SYSTEM.ADMIN.STATISTICS.QUEUE queue:

  Access to statistics data from SYSTEM.ADMIN.STATISTICS.QUEUE queue requires the get authority (+get) because the statistics data is collected from this queue.

• IBM MQ events:

  To obtain IBM MQ events, such as queue manager Events, Performance Events and Channel Events, grant get authority (+get) to the SYSTEM.ADMIN.PERFM.EVENT, SYSTEM.ADMIN.CHANNEL.EVENT, and SYSTEM.ADMIN.QMGR.EVENT system queues.

• MQ monitoring data from the PCF interface:

  The IBM MQ sensor calls the PCF interface to inquire data. The MQ management mechanism uses the put authority (+put) for SYSTEM.ADMIN.COMMAND.QUEUE queue to put the PCF command. To create a temporary queue based on the queue template and to save the PCF command, queue manager needs get authority to retrieve the queue template from SYSTEM.DEFAULT.MODEL.QUEUE queue.

Problems with the local binding mode

For the queue managers that are running, after you make the user privileged, make sure that the security settings work. You must either refresh security by running the following IBM MQ runmqsc command or restart queue manager to make authority that you granted take effective.

REFRESH SECURITY

Monitoring might stop after a sensor update due to a known issue in local binding mode. To resume monitoring, restart the agent.

Connection or authority problems with the client binding mode

For the client binding mode, the IBM MQ sensor acts as a client application to connect to queue manager. Therefore, the sensor requires the same authority as any other applications to connect to queue manager and inquire data, such as MQ explorer, amqsputc MQ sample application. When queue manager does not get connected with your configured parameters, you can try your connection parameters with MQ explorer to connect to queue manager and check whether you can connect queue manager.

See the following issues for reference:

1. RC 2538 - MQRC_HOST_NOT_AVAILABLE

   Instana agent log message: "Listener not started on{host}:{port} ({exceptionCode}) {message}".

   • Listener is not started.

     Solution: Start the listener. To start the listener, run the following IBM MQ runmqsc command:

     START LISTENER($Listener_Name)
     

   • Qmgr@host can’t be found. Check if the queue manager and host names are correct and can be connected.

     Solution: Check if the connection parameters for queue manager and host names are correct and can be connected. Or check if a firewall is blocking the connection.

   • The host and port are configured in the agent configuration file in the Kubernetes cluster.

     In the Kubernetes cluster, when queue manager is restarted, the host IP changes. Therefore, the IBM MQ sensor fails to connect to queue manager.

     Solution: To automatically discover the host and port information, delete the host and port in the agent configuration file for the host agent in the Kubernetes cluster.

2. RC 2540 - Channel is not defined.

   Instana agent log message: "Channel {channel} is not defined ({exceptionCode}). {message}".

   Solution: Check if the correct SVRCONN channel is configured in the configuration.yaml agent configuration file.

3. RC 2035 - Authority problem.

   Instana agent log message: "Channel {channel} authorization failed for user {username} ({exceptionCode}). {message}".

   The problem is caused by authority problem and might be caused by different reasons according to different MQ configurations. To debug the authority problem, check whether channel security is enabled:

   • If channel security is disabled, but reports no authority with the user, then the MQ Application asserts user has no authority. Check which user is used and provide the correct authority to this user or change to another proper user to connect to queue manager.
   • If channel security is enabled. Check whether the provided user and password have the correct authority to connect to queue manager. If the TLS is enabled, you need to provide the correct keystore, keystorePassword, and the corresponding cipherSuite to connect queue manager.

The following image shows the debug flow chart:

The following problems that are frequently encountered might cause 2035 error:

1. Channel security is disabled, but no authority with “root” is reported.

   Usually when CHLAUTH parameter is disabled and CONNAUTH parameter is not set, then the MQ channel security is disabled. You need to determine which user is used for authorization. Here is the order of precedence for security features.

   If you do not configure Security exit, channel record USERSRC(MAP), CLNTUSER, MCAUSER, then the application asserted user, which is the operating system user in a remote connection is used. In this scenario, “root” is used as the application asserted user because the Instana agent is running as root. If root doesn’t have authority, you obtain no authority with “root” in the IBM MQ error log. For more user authority priority details, see https://www.ibm.com/docs/en/ibm-mq/9.3?topic=objects-determining-which-user-is-used-authorization

   Solution: You can configure the channel record and define CLNTUSER or MCAUSER for the channel. Define MCAUSER as a user with MQ access authority for your server connection channel, and then use this user to connect queue manager.

   For example, alter channel(SVRCONN) chltype(SVRCONN) MCAUSER(‘mqmtest‘)

2. No authority is granted to connect to the SYSTEM server connection channel.

   Some system server connection channels like SYSTEM.AUTO.SVRCONN are blocked by default, here are MQ BLOCKUSERS rules: There are three default rules for CHLAUTH processing:

   • NO ACCESS to all channels by any MQ-admin* users
   • NO ACCESS to all SYSTEM.* channels by all users
   • ALLOW access to SYSTEM.ADMIN.SVRCONN channel (non MQ-admin users)

   The first two rules block access to all channels. The third rule is more specific and takes preference over the other two, that is, CHLAUTH allows access only to SYSTEM.ADMIN.SVRCONN channel. For more information, see Resolving CHLAUTH access issues.

   Solution: Unblock the user for system server connection channel before the user is used, or define your own server connection channel for connection, which can workaround this block user problem.

3. Security is enabled or TLS is enabled

   When channel security is enabled--CHLAUTH(ENABLED), or TLS is enabled, but the corresponding username and password or keystore parameters (keystore, keystorePassword, and cipherSuite) are not provided, then provide the credentials. After you change MQ security-related configurations, you need to run refresh security type(CONNAUTH) from the runmqsc prompt to make it work.

   Solution: Provide the corresponding username and password or keystore parameters in the agent configuration file.

4. Security is enabled for both CHLAUTH and CONNAUTH, but user still has authority problem to connect to queue manager.

   Solution: CHLAUTH and CONNAUTH configurations in queue manager, and then check CHLAUTH and CONNAUTH interactions flow to confirm the priority to see which security record works, and correct the problem. For more information, see Interaction of CHLAUTH and CONNAUTH.

5. MQ connection has no problem, but object security is not enough for the user to get other monitoring data.

   Solution: Check the MQ error log to see which object and which user have an authority problem, and provide the correct authority to the object to solve the problem. For more information, see Configuring IBM MQ authority.

Insufficient authority to access SYSTEM.AUTH.DATA.QUEUE

You encounter the following error when the IBM MQ has insufficient authority to obtain the queue reset statistics data. To obtain queue reset statistics data, you need the change authority to access SYSTEM.AUTH.DATA.QUEUE. But SYSTEM.AUTH.DATA.QUEUE is a special queue for which you cannot provide permission to change authority.

AMQ8077W: Entity 'user' has insufficient authority to access object
'SYSTEM.AUTH.DATA.QUEUE'.

EXPLANATION:
The specified entity is not authorized to access the required object. The
following requested permissions are unauthorized: chg
ACTION:
Ensure that the correct level of authority has been set for this entity against
the required object, or ensure that the entity is a member of a privileged
group.

Solution: If you want to stop such error messages from appearing in the IBM MQ log for SYSTEM.AUTH.DATA.QUEUE, make the user of the Instana agent privileged.