Monitoring IBM MQ

Supported operating systems

• The supported operating systems of IBM MQ sensor are consistent with host agents requirements, which can be checked in the Supported operating systems section of the Windows, Unix, or Linux host agent:

  • Supported operating systems for Linux
  • Supported operating systems for Unix
  • Supported operating systems for Windows

• For the supported operating systems of IBM MQ Tracing, see Supported operating systems for IBM MQ Tracing.

Supported IBM MQ versions

• Instana IBM MQ sensor supports monitoring metrics and configuration data for IBM MQ 8.0.0 and later.

• Instana IBM MQ Tracing supports IBM MQ 9.0.0 and later.

Supported client-side tracing

Instana supports client-side tracing for Java.

The configuration steps in the Configuring section are for monitoring and collecting IBM MQ metrics. You can continue to use IBM MQ Tracing on your IBM MQ hosts as you need.

To use IBM MQ server-side tracing, you must install and enable IBM MQ Tracing first. For detailed steps, see IBM MQ Tracing.

Replicated data queue manager (RDQM) support

You need to install the Instana host agent on each RDQM host. The host must be a virtual machine or physical machine of the Linux platform. Then, Instana automatically discovers and monitors the running and standby Queue Manager instances through local monitoring. The Instana UI displays only 1 RDQM Queue Manager name with the aggregated data of 3 Queue Manager instances. You can view the information about HA Type, HA Role, HA Preferred Location, HA Floating IP address, Active Node, and Standby Nodes values on the dashboard of the running Queue Manager in the Instana UI.

Multi-instance Queue Managers support

All the distributed platforms and Kubernetes environments are supported. For the distributed platform, you need to install the Instana host agent on each Queue Manager instance of the multi-instance host. For the Kubernetes environment, you need to install the Instana host agent in the Kubernetes environment. Then, Instana automatically discovers and monitors Queue Managers that are in running, standby, and elsewhere status through local monitoring. The Instana UI displays only 1 multi-instance Queue Manager name with the aggregated data of the other Queue Manager instances. You can view the information about HA Type, Active Node, Standby Nodes, and Elsewhere Nodes values on the dashboard of the running Queue Manager in the Instana UI.

Native HA Queue Manager support

You need to install the Instana host agent in the Kubernetes environment. Then, Instana automatically discovers and monitors Queue Managers that are in running and replica status through local monitoring. The Instana UI display only 1 Native HA Queue Manager name with the aggregated data of the other Queue Manager instances. You can view the information about HA Type, Active Node, and Standby Nodes on the dashboard of the running Queue Manager in the Instana UI.

Single-instance queue manager in Red Hat OpenShift Container Platform support

You need to install the Instana host agent on the Kubernetes environment. Then, Instana automatically discovers and monitors the running queue manager instance through local monitoring. The Instana UI displays the information about HA Type and Active Node of this queue manager instance on the dashboard.

Local binding mode and client binding mode

These 2 modes refer to the connection mode that IBM MQ sensor connects to Queue Manager.

• By using the local binding mode, IBM MQ sensor can do fully automatic discovery and monitoring. By using the client binding mode, IBM MQ sensor can do only partially automatic discovery. In addition, you need to provide some connection parameters.
• By using local binding mode, more monitoring metrics including metrics from local files can be provided. By using client binding mode, IBM MQ sensor can provide only metrics that are retrieved from IBM MQ PCF interface and IBM MQ queues.
• If you use the local binding mode, you need to only add the Instana agent user to the mqm group of IBM MQ, and then all the IBM MQ data can be monitored.
• For client binding mode, you need to configure Queue Manager name, channel name, and related authority to connect to Queue Manager as a client application.

Local monitoring and remote monitoring

These 2 monitoring methods show whether IBM MQ sensor is in the same environment as Queue Manager.

• If IBM MQ sensor and Queue Manager are in the same environment, this is called local monitoring. If they are not in the same environment, this is called remote monitoring.
• The locally monitored Queue Manager instance is shown as one layer in the separated host or Kubernetes cluster node in the Infrastructure Map page of the Instana UI.
• The remotely monitored Queue Manager instances are shown in the Infrastructure Map page as separate boxes in an availability zone, and these instances are grouped by the availabilityZone configuration property or IBM MQ cluster name if present. If a Queue Manager instance is not running in the cluster mode and the availabilityZone configuration property is not defined, Queue Manager is shown in the Undefined zone.
• By using the local monitoring, IBM MQ sensor can connect to Queue Manager by either the local binding mode or client binding mode.
• By using the remote monitoring, IBM MQ sensor can connect to Queue Manager only by the client binding mode.
• The collection of monitoring metrics is decided by the connection modes: local binding mode or client binding mode. The local binding mode can collect more metrics in local files.

Support matrix of IBM MQ connection mode

See the following support matrix for IBM MQ connection mode:

Monitoring IBM MQ in a virtual or physical machine

When the Instana host agent and IBM MQ run in the same machine, two connection modes are supported: local binding mode and client binding mode.

Monitoring IBM MQ in a Kubernetes cluster

When the instana host agent and IBM MQ run in the same Kubernetes cluster, only local monitoring with the client binding mode is supported.

com.instana.plugin.ibmmq:
  enabled: true
  poll_rate: 60
  queueManagers:
    QMGR03:    # Your Queue Manager name here. If there are queue managers with the same name, it is required to append '-<instance>' in the queue manager name to distinguish them. You can select any string for <instance>.
      channel: '<INSERT_CHANNEL_HERE>' # Remote administration channel

Monitoring IBM MQ in Docker

When the Instana host agent and IBM MQ run in different Docker containers in the same machine, only local monitoring with the client binding mode is supported because Instana host agent and IBM MQ run in different Docker operating systems. In such scenario, you need to configure IBM MQ connection parameters in the agent configuration file. Because IBM MQ sensor discovers IBM MQ Queue Manager instances automatically, you don't need to set the host and port in the agent configuration file.

Extra IBM MQ configurations

Some MQ metrics require that the queue statistics data, Queue Manager performance or channel events from IBM MQ side are enabled. If you want to collect these metrics, you need to do the following actions:

1. To collect the queue metrics Last Put Time, Last Get Time, Oldest Message Time, and On Queue Message Time, enable the queue statistic by running the RUNMQSC command alter ql(QUEUE_NAME) MONQ(LOW) by the queue level or by running the command alter qmgr MONQ(LOW) by the Queue Manager level.

   For more information, see IBM MQ documents.

2. To collect the built-in events Bridge stopped, Channel conversion error, Channel SSL Error and Queue Service Interval High, the Queue Manager performance events (PERFMEV) and channel events (CHLEV) need to be enabled by running the following RUNMQSC command:

   alter qmgr PERFMEV(ENABLED) CHLEV(ENABLED)
   

   For more information, see IBM MQ documents.

3. To collect MQ queue statistics metrics as Persistent Put Bytes, Nonpersistent Put Bytes, Persistent Get Bytes, Nonpersistent Get Bytes, Expired Msg Count, Put Fail Count, Put1 Fail Count, Get Fail Count, the following steps are required from IBM MQ:

   1. If you want to get all the queues statistics data, you need to enable queues statistics data collection by Queue Manager level as follows:

      alter qmgr STATQ(ON)
      

      If you want to get only a specific queue statistics data, you need enable it by queue level with your specific Queue_Name as follows:

      alter qlocal(Queue_Name) STATQ(ON)
      

      You can also change the queue statistics interval by using the IBM MQ Queue Manager parameter STATINT, which is 1800 seconds by default. For example, run the command alter qmgr STATINT(900).

   2. To get statistics data from SYSTEM.ADMIN.STATISTICS.QUEUE, you need to grant the get authority to the Instana agent user because the statistics data is collected from this queue. For example, run the command setmqaut -m QmgrName -n SYSTEM.ADMIN.STATISTICS.QUEUE -t q -p user +get. For more information, see IBM MQ documents.

Configuring IBM MQ authority

• Using privileged user for the Instana agent

  If the Instana agent is installed by using a user in the mqm group, it has all the necessary authorities. Therefore, you can view all the monitoring data in the Instana UI.

• Using non-privileged user for the Instana agent

  When the Instana agent is installed by using a non-privileged user (the user not in the mqm group) to connect to Queue Manager, you must grant appropriate Object Authority Manager (OAM) authorities to the user to query all the monitoring data.

  To provide the IBM MQ OAM authorities to a non-privileged user, use the setmqaut control command. The user that you use to issue the setmqaut command must be a member of the mqm group. The setmqaut command is in the $MQ_INSTALLED_DIR/bin directory.

  The user that you use to issue the setmqaut command must be a member of the mqm group.

1. To grant the user the appropriate authorities to access Queue Manager that you want to monitor, run the following command:

   setmqaut -m QMGR -t qmgr -p UserID +inq +connect +dsp
   

   where:

   • QMGR: Name of Queue Manager
   • UserID: UserID of the user.

   You must specify the fully qualified user for the -p option. For example, -p user@domain or -p user@host.

   To specify a user group name, replace the -p option with the -g option in the command.

2. To provide the appropriate authorities to the user to access the system queues of Queue Manager, run the following commands:

   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.COMMAND.QUEUE -p UserID +inq +dsp +chg +get +put
   setmqaut -m QMGR -t q -n SYSTEM.DEFAULT.MODEL.QUEUE -p UserID +dsp +get
   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.QMGR.EVENT -p UserID +inq +dsp +chg +get
   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.PERFM.EVENT -p UserID +inq +dsp +chg +get
   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.CHANNEL.EVENT -p UserID +inq +dsp +chg +get
   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.STATISTICS.QUEUE -p UserID +inq +dsp +chg +get
   setmqaut -m QMGR -t q -n SYSTEM.AUTH.DATA.QUEUE -p UserID +dsp
   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.LOGGER.EVENT -p UserID +inq +dsp +chg
   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.CONFIG.EVENT -p UserID +inq +dsp +chg
   setmqaut -m QMGR -t q -n SYSTEM.ADMIN.COMMAND.EVENT -p UserID +inq +dsp +chg
   

3. For the Instana UI to display data, the user specified by UserID needs the display access to various objects. To grant the appropriate authorities, run the following commands:

   setmqaut -m QMGR -t q -n "**" -p UserID +dsp +chg 
   setmqaut -m QMGR -t channel -n "**" -p UserID +dsp
   setmqaut -m QMGR -t clntconn -n "**" -p UserID +dsp
   setmqaut -m QMGR -t listener -n "**" -p UserID +dsp
   setmqaut -m QMGR -t topic -n "**" -p UserID +dsp
   

4. To ensure all the commands work, restart Queue Manager or refresh security:

   echo "REFRESH SECURITY" | runmqsc QMGR
   

These permissions cover all the necessary access rights for IBM MQ sensors to query and gather monitoring data. Typically, read-only access is adequate for most monitoring data needs. However, for some of the following metrics, additional permissions are required. The permissions must be adjusted accordingly to accommodate these requirements based on your required monitoring metrics.

• Queue reset statistics data:

  To retrieve queue reset statistics data, queue objects require the change authority (+chg).

• Statistics data from SYSTEM.ADMIN.STATISTICS.QUEUE queue:

  Access to statistics data from SYSTEM.ADMIN.STATISTICS.QUEUE queue requires the get authority (+get) because the statistics data is collected from this queue.

• IBM MQ events:

  To obtain IBM MQ events, such as Queue Manager Events, Performance Events and Channel Events, grant get authority (+get) to the SYSTEM.ADMIN.PERFM.EVENT, SYSTEM.ADMIN.CHANNEL.EVENT, and SYSTEM.ADMIN.QMGR.EVENT system queues.

• MQ monitoring data from the PCF interface:

  The IBM MQ sensor calls the PCF interface to inquire data. The MQ management mechanism uses the put authority (+put) for SYSTEM.ADMIN.COMMAND.QUEUE queue to put the PCF command. To create a temporary queue based on the queue template and to save the PCF command, Queue Manager needs get authority to retrieve the queue template from SYSTEM.DEFAULT.MODEL.QUEUE queue.

Problems with the local binding mode

For the Queue Managers that are running, after you add the user of the Instana agent to the mqm group, ensure that the security settings work. You need to either refresh security by running the following IBM MQ runmqsc command or restart Queue manager to make the mqm authority that you granted take effective.

REFRESH SECURITY

Connection or authority problems with the client binding mode

For the client binding mode, the IBM MQ sensor acts as a client application to connect to Queue Manager. Therefore, the sensor requires the same authority as any other applications to connect to Queue Manager and inquire data, such as MQ explorer, amqsputc MQ sample application. When Queue Manager does not get connected with your configured parameters, you can try your connection parameters with MQ explorer to connect to Queue Manager and check whether you can connect Queue Manager.

See the following issues for reference:

1. RC 2538 - MQRC_HOST_NOT_AVAILABLE

   Instana agent log message: "Listener not started on{host}:{port} ({exceptionCode}) {message}".

   • Listener is not started.

     Solution: Start the listener. To start the listener, run the following IBM MQ runmqsc command:

     START LISTENER($Listener_Name)
     

   • Qmgr@host can’t be found. Check if the Queue Manager and host names are correct and can be connected.

     Solution: Check if the connection parameters for Queue Manager and host names are correct and can be connected. Or check if a firewall is blocking the connection.

   • The host and port are configured in the agent configuration file in the Kubernetes cluster.

     In the Kubernetes cluster, when Queue Manager is restarted, the host IP changes. Therefore, the IBM MQ sensor fails to connect to Queue Manager.

     Solution: To automatically discover the host and port information, delete the host and port in the agent configuration file for the host agent in the Kubernetes cluster.

2. RC 2540 - Channel is not defined.

   Instana agent log message: "Channel {channel} is not defined ({exceptionCode}). {message}".

   Solution: Check if the correct SVRCONN channel is configured in the configuration.yaml agent configuration file.

3. RC 2035 - Authority problem.

   Instana agent log message: "Channel {channel} authorization failed for user {username} ({exceptionCode}). {message}".

   The problem is caused by authority problem and might be caused by different reasons according to different MQ configurations. To debug the authority problem, check whether channel security is enabled:

   • If channel security is disabled, but reports no authority with the user, then the MQ Application asserts user has no authority. Check which user is used and provide the correct authority to this user or change to another proper user to connect to Queue Manager.
   • If channel security is enabled. Check whether the provided user and password have the correct authority to connect to Queue Manager. If the TLS is enabled, you need to provide the correct keystore, keystorePassword, and the corresponding cipherSuite to connect Queue Manager.

The following problems that are frequently encountered might cause 2035 error:

1. Channel security is disabled, but no authority with “root” is reported.

   Usually when CHLAUTH parameter is disabled and CONNAUTH parameter is not set, then the MQ channel security is disabled. You need to determine which user is used for authorization. Here is the order of precedence for security features.

   If you do not configure Security exit, channel record USERSRC(MAP), CLNTUSER, MCAUSER, then the application asserted user which is the operating system user in a remote connection is used. In this scenario, “root” is used as the application asserted user because the Instana agent is running as root. If root doesn’t have mqm authority, you obtain no authority with “root” in the MQ error log. For more user authority priority details, see https://www.ibm.com/docs/en/ibm-mq/9.3?topic=objects-determining-which-user-is-used-authorization

   Solution: You can configure the channel record and define CLNTUSER or MCAUSER for the channel. Define MCAUSER as a user with MQ access authority for your server connection channel, and then use this user to connect Queue Manager.

   For example, alter channel(SVRCONN) chltype(SVRCONN) MCAUSER(‘mqmtest‘)

2. No authority is granted to connect to the SYSTEM server connection channel.

   Some system server connection channels like SYSTEM.AUTO.SVRCONN are blocked by default, here are MQ BLOCKUSERS rules: There are three default rules for CHLAUTH processing:

   • NO ACCESS to all channels by any MQ-admin* users
   • NO ACCESS to all SYSTEM.* channels by all users
   • ALLOW access to SYSTEM.ADMIN.SVRCONN channel (non MQ-admin users)

   The first two rules block access to all channels. The third rule is more specific and takes preference over the other two, that is, CHLAUTH allows access only to SYSTEM.ADMIN.SVRCONN channel. For more information, see Resolving CHLAUTH access issues.

   Solution: Unblock the user for system server connection channel before the user is used, or define your own server connection channel for connection, which can workaround this block user problem.

3. Security is enabled or TLS is enabled

   When channel security is enabled--CHLAUTH(ENABLED), or TLS is enabled, but the corresponding username and password or keystore parameters (keystore, keystorePassword, and cipherSuite) are not provided, then provide the credentials. After you change MQ security-related configurations, you need to run refresh security type(CONNAUTH) from the runmqsc prompt to make it work.

   Solution: Provide the corresponding username and password or keystore parameters in the agent configuration file.

4. Security is enabled for both CHLAUTH and CONNAUTH, but user still has authority problem to connect to Queue Manager.

   Solution: Check CHLAUTH and CONNAUTH configurations in Queue Manager, and then check CHLAUTH and CONNAUTH interactions flow to confirm the priority to see which security record works, and correct the problem.
   For more information, see Interaction of CHLAUTH and CONNAUTH.

5. MQ connection has no problem, but object security is not enough for the user to get other monitoring data.

   Solution: Check the MQ error log to see which object and which user have an authority problem, and provide the correct authority to the object to solve the problem. For more information, see Configuring IBM MQ authority.

Insufficient authority to access SYSTEM.AUTH.DATA.QUEUE

You encounter the following error when the IBM MQ has insufficient authority to obtain the queue reset statistics data. To obtain queue reset statistics data, you need the change authority to access SYSTEM.AUTH.DATA.QUEUE. But SYSTEM.AUTH.DATA.QUEUE is a special queue for which you cannot provide permission to change authority.

AMQ8077W: Entity 'user' has insufficient authority to access object
'SYSTEM.AUTH.DATA.QUEUE'.

EXPLANATION:
The specified entity is not authorized to access the required object. The
following requested permissions are unauthorized: chg
ACTION:
Ensure that the correct level of authority has been set for this entity against
the required object, or ensure that the entity is a member of a privileged
group.

Solution: If you want to stop such error messages from appearing in the IBM MQ log for SYSTEM.AUTH.DATA.QUEUE, add the user of the Instana agent to the mqm group.