Integrating with IBM OMEGAMON

Summary

The Integration of IBM OMEGAMON with Instana is accomplished by configuring IBM Z OMEGAMON Data Provider and by using the Integration Proxy to communicate. The Integration Proxy sends Instana converted Omegamon content to a Host Agent’s proxy sensor called com.instana.plugin.ibmapmproxy. The Instana host agent then sends the payloads to the Instana server where the payloads are processed, creating entities in the Instana server, and saving the metrics for display in the Instana UI.

Architecture diagram

Prerequisites

  • Make sure you have the host agent installed for Instana. For more information, see Installing the host agent on Linux.

  • Activate OMEGAMON support by following the steps in Activating OMEGAMON support.

  • In order for the sensor to gather metrics from OMEGAMON, the IBM Z OMEGAMON Data Provider needs to be configured to push data to IBM OMEGAMON Instana Integration Proxy. For information about configuring IBM Z OMEGAMON Data Provider, see IBM OMEGAMON Documentation.

  • IBM OMEGAMON Instana Integration Proxy must meet the following prerequisites:

    • Kubernetes cluster that runs on Intel 64 bit (amd64) Linux OS
      • Supported Kubernetes versions: 1.19, 1.20, 1.21, 1.22, 1.23
    • At least 8 CPU cores and 16 GB of memory that can be allocated to the IBM OMEGAMON Instana Integration Proxy
    • An ingress controller (such as nginx) that is configured to enable external network access to the cluster
    • A default persistent storage class that has 30 GB available
  • The supported platforms for the IBM OMEGAMON Instana Integration Proxy are:

    • MicroK8s 1.23/stable

    • Red Hat OpenShift Container Platform 4.6.52

    • Red Hat OpenShift Container Platform 4.8.27

    • Red Hat OpenShift Container Platform 4.9.13

      For all the Red Hat OpenShift Container Platform versions, if there is an Instana agent monitoring the cluster, it is recommended that the agent be configured in Infrastructure mode rather than APM mode.

Activating OMEGAMON support

Metric ingestion by using the proxy sensor in the host agent is deactivated by default. To activate this support, you need to do the following steps:

  1. Download the deployment package for the IBM OMEGAMON Instana Integration Proxy from https://github.com/instana/ibm-monitoring-integration and follow the instructions to deploy the service.

    Note: The deployment package provides the ability to deploy the IBM OMEGAMON Instana Integration Proxy on a system that does not have a connection to the internet (also known as "air-gapped"). The deployment supports High Availability (HA) mode for new deployments. When HA mode is enabled, three pod instances are created for statefulset components (zookeeper and kafka). For stateless components, the number of replicas can be specified (the minimum number of replicas is 2).

  2. Enable the proxy sensor in your Instana host agent to receive IBM OMEGAMON data.
    Instana collects data from monitored systems by using a single agent on each host. The agent runs on your hosts to collect and aggregate data from various sensors before it sends the data to the Instana backend. For more information, see Instana Host Agent documentation.

Configuration

To enable the proxy sensor, locate the ibmapmproxy section of the agent configuration file, and uncomment the lines for the sensor. Provide the host, protocol, and port information for your IBM OMEGAMON Instana Integration Proxy in the corresponding fields: <agent_install_dir>/etc/instana/configuration.yaml:

com.instana.plugin.ibmapmproxy:
  enabled: true
  # host: Provide the hostname or IP address of the system where the APM-Proxy server is running.  No quotation marks are needed.
  host: '<INSERT_HOST_HERE>' # APM-Proxy host
  # port: provide the port defined for contacting the server. No quotation marks are needed.
  port: 443 # APM-Proxy port Default
  datainterval: 1 # Default
  heartbeatinterval: 15 # Default  
  # protocol: indicate whether to connect with the APM-Proxy server using the secure (SSL) port or the insecure port.  Values are either "http" or "https".  Default is "https".  No quotation marks are needed. Ensure the value provided for the "port" variable is the appropriate port for your server.
  protocol: https # Default

Note: The IBM OMEGAMON Instana Integration Proxy acts as the APM-Proxy server.

Metrics collection

The metrics that are related to z/OS, Db2 for z/OS, CICS, JVM, MQ for z/OS, and IMS are collected from IBM OMEGAMON and visualized within Instana.

z/OS

See the configuration and performance metrics that are related to z/OS as follows.

Configuration data

  • Workload Management Mode
  • Sysplex Name
  • SMF ID
  • Managed System
  • Product Code
  • Physical CPU Count

Performance metrics

See the performance metrics that are related to System CPU Utilization, Real Storage Utilization History and Common Storage Utilization History as follows.

System CPU Utilization

This table provides information about CPU usage for the system.

Metric Description Granularity
Average CPU Percent Current/latest sample of the system CPU, as seen by the LPAR
Undispatched Tasks The number of tasks or address spaces that have not been dispatched by the SRM due to constraints
MVS Overhead The CPU utilization percentage that is not attributable to any user or address space
Average IFA Percent Average IFA Percent
Average IFA on CP Percent Average IFA on CP Percent
4 Hour MSUs The four-hour rolling average value of the millions of service units (MSUs) per hour
Average zIIP Percent The average percentage of IBM Z Integrated Information Processor (zIIP) time
Average zIIP on CP Percent The average percentage of regular processor time the system consumed for zIIP
Percent LPAR MSU Capacity The percentage of the Entitled or Defined Capacity millions of service units (MSUs) of this LPAR
Average Unused Group MSUs The number of millions of service units (MSUs) that were unused

Note: Granularity depends on OMEGAMON and IBM OMEGAMON Instana Integration Proxy configuration.

Real Storage Utilization History

The historical real storage utilization attribute group provides information about the real storage that is allocated to an address space in terms of various types of frame counts and slot counts, as well as the management status of a given address space.

Metric Description Granularity
Total Frames The total number of frames occupied by the address space
Active Frames The number of active frames occupied by the address space
Idle Frames The number of idle frames occupied by the address space
Auxiliary Storage Slots The number of auxiliary storage slots occupied by this address space
Memory Objects Allocated The number of memory objects allocated by this address space
1 Meg Frames In Real The number of 1-Megabyte frames backed in real storage
Active Frames Working Set The number of active frames contained in the working set
Active Frames Fixed The number of fixed frames actively used by the address space
Active Frames DIV The number of active Data-in-Virtual frames actively used by the address space
Page-In Rate Page-In rate (Page-ins per second)
Shared Page-In Rate Shared page-in rate from auxiliary storage (Page-ins per second)
Shared Pages Total Views The total shared page views
Shared Pages Total Valid The total valid shared pages
Shared Pages Validation Rate The shared page validation rate of the address space

Note: Granularity depends on OMEGAMON and IBM OMEGAMON Instana Integration Proxy configuration.

Common Storage Utilization History

The historical Common Storage attribute group displays historical information about four important areas of common storage: common service area (CSA), extended CSA, system queue area (SQA), and extended SQA.

Metric Description Granularity
Percentage CSA In Use The average percentage of Common Service Area (CSA) storage used by the job
Percentage ECSA In Use The average percentage of Extended Common Service Area (ECSA) storage used by the job
Percentage SQA In Use The average percentage of System Queue Area (SQA) storage used by the job
Percentage ESQA In Use The average percentage of Extended System Queue Area (ESQA) storage used by the job
Amount CSA In Use The amount of total Common Service Area (CSA) used by this address space
Amount ECSA In Use The amount of total Extended Common Service Area (ECSA) used by this address space
Amount SQA In Use The amount of total System Queue Area (SQA) used by this address space
Amount ESQA In Use The amount of total Extended System Queue Area (ESQA) used by this address space

Note: Granularity depends on OMEGAMON and IBM OMEGAMON Instana Integration Proxy configuration.

Db2 for z/OS

See the configuration and performance metrics that are related to Db2 for z/OS as follows.

Configuration data

  • MVS System
  • Origin Node
  • Db2 Subsystem
  • Product Code

Performance metrics

See the performance metrics that are related to Db2 System States, Db2 CICS Exceptions, and Db2 IMS Connections as follows.

Db2 System States

This table provides information about Db2 system states attributes.

Metric Description Granularity
Lock Conflict Count The total number of owners and waiters that are in lock conflict
Current Thread Count The current number of active threads
Transactions Per Second The number of successful requests for commit phase 1 in a two-phase commit environment such as CICS or IMS
Pages read from BPs The number of Getpage requests including conditional and unconditional requests
Pages Read From DASD The number of synchronous read I/O operations performed by DB2 for applications and utilities
ECSA Used By Db2 ECSA Used By Db2
Real Storage Used By Db2 Real Storage Used By Db2

Note: Granularity depends on OMEGAMON and IBM OMEGAMON Instana Integration Proxy configuration.

Db2 CICS Exceptions

This table provides information about Db2 CICS Exceptions attributes from connected CICS subsystems.

Metric Description Granularity
Total Thread Utilization The utilization of threads based on the current total threads in use divided by the maximum allowed threads
Pool Thread Utilization The utilization of pool threads based on the current pool thread in use count that is divided by the maximum allowed pool threads
Total Threads Maximum The maximum number of threads that can be used concurrently for this CICS connection
Total Threads In Use The count of active threads for this CICS connection
Pool Threads In Use The count of pool threads that are currently active
Pool Thread Waits The count of pool threads that are waiting because the maximum has been reached
Pool Thread Maximum The maximum number of pool threads that are allowed

Note: Granularity depends on OMEGAMON and IBM OMEGAMON Instana Integration Proxy configuration.

Db2 IMS Connections

This table provides information about Db2 IMS Connections attributes from connected IMS subsystems.

Metric Description Granularity
Active Threads The count of threads currently active for this IMS region
Defined Dependent Regions The count of defined dependent regions for this IMS connection
Connected Dependent Regions The count of dependent regions currently connected to this Db2 system
Unconnected Regions The number of regions not connected

Note: Granularity depends on OMEGAMON and IBM OMEGAMON Instana Integration Proxy configuration.

Db2 Lock Contentions

This table provides information about Db2 Lock Contentions.

Metric Description Granularity
Lock Elapsed Time Lock Elapsed Time

Note: Granularity depends on OMEGAMON and IBM OMEGAMON Instana Integration Proxy configuration.

CICS

See the configuration and performance metrics that are related to CICS as follows.

Configuration data

  • Origin Node
  • System ID
  • CICS Region Name
  • CICS Version
  • CICSPlex Name

Performance metrics

See the performance metrics that are related to CICSplex Region Overview as follows.

CICSplex Region Overview

The CICSplex Region Overview attributes report on the internal resources of CICS regions. These resources include storage, files, queues, and enqueues.

Metric Description Granularity
Transaction Rate The average number of transactions executed in one minute of elapsed time
Maximum Tasks Percent The total number of tasks within a CICS region, expressed as a percentage of the MXT limit
CPU Utilization The percentage of accumulated CPU time for the CICS region
I/O Rate The rate at which I/O operations are being performed in a specified CICS region per second of elapsed time
Page Rate The number of page-in operations in the CICS region per CPU second
Storage Violations The number of storage violations that have occurred within the last hour
Enqueue Waits The number of CICS enqueues exclusively controlling a resource that tasks are also waiting for
AIDs The number of automatic initiate descriptors (AIDs) found on the AID chain within a given CICS region
ICEs The number of interval control elements (ICEs) found on the ICE chain within a given CICS region
Worst Region Performance Index Indicator as to which CICS region's response time might be starting to degrade
Queued Remote Requests The CICS region's total number of queued allocation requests for MRO, ISC, and IP sessions
Highest Pct Class MaxT The highest percentage of transactions with respect to the class limit for any one transaction class
Total Queued Transactions The total number of transactions queued for Maximum task or transaction class limits

Note: Granularity depends on OMEGAMON and IBM OMEGAMON Instana Integration Proxy configuration.

JVM metrics

See the following JVM metrics that are collected by IBM OMEGAMON Instana Integration Proxy.
Note: You must install version 1.2.0 or a later version of the OMEGAMON Instana Integration Proxy to get the JVM metrics.

Metric Description
Collector Ids Collector ID is the identifier of a JVM collector instance
Memory Pool Allocated Bytes The number of BYTES allocated (NOT including children)
Application Title Name Name of the job or started task hosting this JVM
JVM Free Memory The sum of free memory across 64-bit virtual memory (MB) remaining above the 2 GB bar in the address space, 31-bit virtual memory remaining above the 16 MB line in the address space and 24-bit virtual memory remaining below the 16 MB line in the address space.
JVM Used Memory The sum of used memory across 64-bit virtual memory (MB) remaining above the 2 GB bar in the address space, 31-bit virtual memory remaining above the 16 MB line in the address space and 24-bit virtual memory remaining below the 16 MB line in the address space.
JVM Total Memory Allocated The total memory across 64-bit virtual memory (MB) remaining above the 2 GB bar in the address space, 31-bit virtual memory remaining above the 16 MB line in the address space and 24-bit virtual memory remaining below the 16 MB line in the address space.
JVM Total Blocked Threads Current Number of Blocked Threads in the JVM.
JVM Total Thread Count The total number of active threads in the JVM.
Garbage Collection Time Total time performing garbage collection.
Garbage collection Invocations Nursury Garbage Collection Count.
Maximum Heap Size Reached Maximum heap size (bytes) reached during interval.
Minimum Heap Size Reached Minimum heap size (bytes) during interval.
Maximum Heap Used Maximum used heap reached during interval.
Minimum Heap Used Minimum used heap (bytes) during interval.

MQ for z/OS metrics

See the following IBM MQ metrics that are collected by the IBM OMEGAMON Instana Integration Proxy.
Note: You must install version 1.2.0 or a later version of the OMEGAMON Instana Integration Proxy to get the MQ for z/OS metrics.

Metric Description
Queue Manager Name The name of the Queue Manager.
Queue Manager Start Date Queue Manager start date and time.
Queue Manager Status The current execution status of the queue manager. One of Starting, Running, Quiescing, Stopping, Standby, Stopped
Queue Manager Connections Indicates the current number of connection to the queue manager.
Queue Name(s) Queues that have had queue depth at some point in the past or currently. (Active Queues)
Queue Puts Enabled Indicates whether the current queue is enabled for puts (that is, whether applications can call WebSphere MQ API routines MQPUT or MQPUT1 for this queue).
Queue Gets Enabled Indicates whether the current queue is enabled for gets (that is, whether applications can call WebSphere MQ API routine MQGET for this queue).
Current Queue Depth The number of messages that are currently on the queue.
Maximum Queue Depth Highest recorded Current Depth value from the collection of sampling intervals that comprise the entire historical period.
Messages Placed into Queue The number of messages that are put to the queue during the sampling interval.
Messages Read from Queue The number of messages that are read (and removed) from the queue during the sampling interval.
Queue Monitoring Level The current level of monitoring data collection for the queue.
Current Queue Output Handles The number of handles that are currently open for output.
Current Queue Input Handles The number of handles that are currently open for input (either shared or exclusive), this does not include opens for browsing.
Channel Name(s) The name of the channel.
Connection Name(s) Name of the connection. This is the connection name that is obtained using the MQSC DIS CHS command (see WebSphere MQ documentation for further information about this command).
The Channel Status Status of the channel. One of Binding, Starting, Running, Stopping, Retrying, Stopped, Requesting, Paused, Initializing, Switching.
The Channel Sub-Status Current action that is being performed by the channel.
Channel In Doubt The number of monitored channel connections that are currently in doubt for this queue manager. A sender channel is in doubt when a logical unit of work (LUW) has been sent and the channel is waiting for an acknowledgment that the LUW has been successfully received (in other words, when a syncpoint has been requested but not yet performed).
Channel Start Date and Time The date and time at which the channel is started.
Channel Last Message Date and Time Date and time at which the last message is sent or MQI call is handled.
Channel Buffers Sent Number of buffers that are sent.
Channel Buffers Received Number of buffers that are received.
Channel Bytes Received Number of bytes that are received.
Channel Sharing Conversations It is blank for all channel types other than server-connection channels. For each instance of a server-connection channel, it gives the number of conversations that are currently running over that channel instance.
Channel Message Count The number of messages that are sent or received on this channel during the current interval or aggregation period.
Channel Message Sequence Number The number of the last message sent for the logical unit of work (LUW) that the in-doubt channel is currently processing. This attribute is not available for CICS channels.
Channel Sequence Number Last The number of the last-committed message within the last-committed logical unit of work (LUW).

IMS

See the following IMS metrics and configuration data that are collected by IBM OMEGAMON Instana Integration Proxy.
Note: You must install Instana Self-Hosted release 243 or a later version to get the IMS metrics.

Dependent Regions

The Dependent Regions table is used to view status information for dependent regions.

Configuration data
Configuration data Description
MVS System The MVS System Management Facility (SMF) identifier (mvs_system)
IMS Id The IMS Subsystem identifier (ims_id)
Region Name The job name of the subject IMS address space (region_name)
Transaction Name The transaction id that is currently running in the subject region (transaction_name)
IMSPlex Name The name of the IMSplex that this dependent region belongs to (imsplex_name)
User Id Indicates the user identification currently active in the region (user_id)
Originating System Identifier Fully identifies the originating IMS subsystem (originating_system_identifier)
PSB Name The PSB name that is currently running in the subject region (psb_name)
Metrics
Metric Description
logical_terminal Indicates the LTERM name for message type regions.
type Indicates the IMS dependent region type. Valid values include BMP, DBCTL, FastPath, HSSP-BMP, Java, Message, andJava Batch.
locks_held_count Indicates the number of IRLM or PI locks held by subject IMS region.
region_status Indicates the current status of the region.
region_occupancy_percentage The dependent region occupancy percentage.
external_subsystem_calls The external subsystem status, provided when the application is processing an external subsystem request as indicated by the region status value of Exec-DB2, Exec-MQ, or Exec-ESS.
call_pcb_sensitive_segments PCB number of sensitive segments from the current or most recently run call.
external_subsystem_calls For IMS V10 and later versions, the current count of all external subsystem (ESAF) calls for the running transaction (UOR); prior to IMS V10, N/A is displayed.
call_pcb_status PCB status code from the current or most recently run call.

VSAM Sub Pools

Configuration data
Configuration data Description
Pool Name The pool identifier name assigned to this subpool (pool_name)
MVS System The MVS System Management Facility (SMF) identifier (mvs_system)
IMS Id The IMS Subsystem identifier (ims_id)
Originating System Identifier Fully identifies the originating IMS subsystem (originating_system_identifier)
Metrics
Metric Description
using_hiperspace Indicates whether the subpool is using hiperspace.
subpool_hit_ratio The percentage of total record requests that were satisfied from the subpool.
buffer_size The subpool buffer size, in bytes.
blocks_pagefixed Indicates whether the blocks for this subpool are page fixed.
subpool_number The subpool identification number.
pool_type Indicates the pool type for this subpool. Valid value is Data or Index.
buffers_pagefixed Indicates whether the buffers for this subpool are page fixed.

Pool Utilization

Configuration data
Configuration data Description
Originating System Identifier Fully identifies the originating IMS subsystem (originating_system_identifier)
IMS Id The IMS Subsystem identifier (ims_id)
Pool Name The pool identifier name assigned to this subpool (pool_name)
Pool Group Indicates the group of the subject IMS storage pool. Valid values include Communications, Database, and Program (pool_group)
Metrics
Metric Description
free_blocks Indicates the current count of free blocks for the subject IMS pool.
largest_free_block Indicates the largest free blocks for the subject IMS pool in bytes.
pool_size Indicates the size of the subject IMS storage pool in bytes.
free_space Indicates the current free space for the subject IMS pool in bytes.

Sub Pool Statistics

Configuration data
Configuration data Description
Originating System Identifier Fully identifies the originating IMS subsystem (originating_system_identifier)
MVS System The MVS System Management Facility (SMF) identifier (mvs_system)
IMS Id The IMS Subsystem identifier (ims_id)
Metrics
Metric Description
subpool_hit_ratio Indicates the percentage of total locate calls that were satisfied from the subpool.
prefix_pagefixed Indicates whether the prefix is page fixed or not. Valid values include Yes, No, All, Mixed, and None.
subpool_number Identifies the subpool.
total_storage Indicates the total amount of storage allocated for subpool.

Address Spaces

Configuration data
Configuration data Description
Originating System Identifier Fully identifies the originating IMS subsystem (originating_system_identifier)
IMSPlex Name The name of the IMSplex that this dependent region belongs to (imsplex_name)
MVS System The MVS System Management Facility (SMF) identifier (mvs_system)
IMS Id The IMS Subsystem identifier (ims_id)
Job Name The job name of the subject IMS address space (job_name)
Metrics
Metric Description
type Type of IMS address space, including BMP, Control, FastPath, HSSP-BMP, IRLM, Track, TMS, Message, DBRC, DL/I_SAS, Java, and Java_Batch.
cpu_time Total amount of CPU time consumed by this address space.
cpu_percentage Percentage of CPU time consumed by this address space.
working_set_size Current working set size for address space, expressed in number of frames.
common_page_in_rate Page-in rate of common area storage in pages per second for address space.
swapped_out Indicates whether this address space is currently swapped out.
excp_rate EXCP rate for this address space in EXCPs per second.

IMS Health

Configuration data
Configuration data Description
SQGroup Name Indicates the name of the Shared Queues group for this IMS system if defined (sqgroup_name)
IMSPlex Name The name of the IMSplex that this dependent region belongs to (imsplex_name)
MVS System The MVS System Management Facility (SMF) identifier (mvs_system)
IMS Id The IMS Subsystem identifier (ims_id)
Sysplex Name Indicates the originating sysplex for this request (sysplex_name)
Metrics
Metric Description
region_count The number of dependent regions running under this IMS.
total_transaction_rate The total full function and Fast Path transaction rate.
lock_waiters The number of regions waiting on global or local locks.
shared_transaction_queue The number of transactions on the shared queue. This includes messages on the transaction and fast path program queues.
total_io_rate The total I/O rate for IMS control type regions and all dependent regions.
longest_lock Indicates the elapsed time, in microseconds, that the region has been waiting on a database lock.
total_cpu_percent The total CPU percentage for IMS control type regions and all dependent regions.
affinity_count The number of transactions on the shared queue that have an affinity to this IMS system.
total_enq_rate The total message enqueue rate for full function and Fast Path.
highest_r0_time The highest R0 response time in microseconds for collected transactions.
total_transaction_queue The total number of transactions in the input queue for all full function and Fast Path.
total_paging_rate The total paging rate for IMS control type regions and all dependent regions.

Monitoring Issues

Missing Configuration

Monitoring issue type: apmproxy_missing_config

There are one or more required items missing in the configuration. Make sure that your host is set in the configuration.yaml file.

Connection Error

Monitoring issue type: apmproxy_connection_error

The proxy sensor can't connect to the IBM OMEGAMON Instana Integration Proxy after many retries. Make sure that your configuration properties are correctly set in the configuration.yaml file. Make sure that the protocol and port values are aligned, such as port 80 with protocol HTTP or port 443 with protocol HTTPS.

Other Exception

Monitoring issue type: apmproxy_exception_error

The proxy sensor encounters an uncommon exception that is not yet documented. This might mean that there is an issue with the Instana Agent or IBM OMEGAMON Instana Integration Proxy. Make sure that Instana Agent and IBM OMEGAMON Instana Integration Proxy are correctly configured and running. You can raise a ticket to Instana support team if you encounter this problem.