Viewing the status of Native HA queue managers

For custom-built containers, you can view the status of the Native HA instances by using the dspmq command.

About this task

You can use the dspmq command in one of the running Pods to view the operational status of a queue manager instance. The information returned depends on whether the instance is active or a replica. The information supplied by the active instance is definitive, information from replica instances might be out of date.

For a Native HA Cross-Region Replication (CRR) configuration, the information supplied also depends on whether the instance is part of a Live or a Recovery group. The information from the Live group is more definitive, but can still be out of date as the Recovery group sends back only periodic information.

You can perform the following actions:

• View whether the queue manager instance on the current node is active or a replica.
• View the Native HA operational status of the instance on the current node.
• View the operational status of all three instances in a Native HA configuration.
• View the status of the groups that the instance belongs to or is connected to.

The following status fields are used to report Native HA and Native HA Cross-Region Replication (CRR) configuration status of a queue manager instance. These fields always appear:

ROLE

Specifies the current role of the instance. Is one of Active, Replica, Unknown, (for a recovery group) Leader, or Not Configured.

INSTANCE

The name provided for this instance of the queue manager when it was created using the -lr option of the crtmqm command.

INSYNC

Indicates whether the instance is able to take over as the active instance if required.

QUORUM

Reports the quorum status in the form number_of_instances_in-sync/number_of_instances_configured.

GRPLSN

Specifies the log sequence number (LSN) that has been replicated between a quorum of instances in the HA group, in nnnnn:nnnnn:nnnnn:nnnnn format. Blank if the LSN is not known.

GRPNAME

Specifies the name of the HA group. Blank if a name has not been configured.

GRPROLE

Specifies the current role of the HA group and is one of:

• Live - A live group
• Recovery - A recovery group
• Pending live - A group waiting to become Live
• Pending recovery - A group waiting to become Recovery
• Not configured - Native HA has not been configured
• Unknown - The current role cannot be determined

These fields appear when extended information is requested for all instances by using the -x parameter:

REPLADDR

The replication address of the queue manager instance.

CONNACTV

Indicates whether the instance is connected to the active instance.

BACKLOG

Indicates the number of KB that the instance is behind the active instance.

CONNINST

Indicates whether the named instance is connected to this instance.

ALTDATE

Indicates the date when this information was last updated. Blank if it has never been updated.

ALTTIME

Indicates the time when this information was last updated. Blank if it has never been updated.

ACKLSN

Indicates the LSN that the instance has acknowledged as having been written to the recovery log, in nnnnn:nnnnn:nnnnn:nnnnn format. Blank if the LSN value is not known.

HASTATUS

Indicates the operational status of this instance and is one of the following values:

• Normal - the instance is operating normally.
• Checking - the instance is being checked to ensure the consistency of the recovery log.
• Synchronizing - the instance is being sent data to bring it in-sync.
• Rebasing - a new recovery log is being sent to the instance.
• Disk full - the instance has a full disk.
• Disconnected - cannot communicate with the instance.
• Unknown - the instance status is unknown.

SYNCTIME

Indicates the time when this instance was last in-sync with the active queue manager, in ISO 8601 format. Blank if the time is not known.

These fields appear when group information is requested by using the -g parameter. These fields are useful to view the status when the group is part of a Native HA CRR configuration:

GRPNAME

Specifies the name of the HA group. Blank if a name has not been configured.

GRPROLE

Specifies the current role of the HA group and is one of:

• Live - A live group
• Recovery - A recovery group
• Pending live - A group waiting to become Live
• Pending recovery - A group waiting to become Recovery
• Unknown - The current role cannot be determined
• Not configured - Native HA has not been configured

GRPADDR

The IP address that is used to connect to the group. This is set to Unknown if the address is not known because a connection to the group has not occurred.

GRPVER

Indicates the version of the current group leader in V.R.M.F format, where V.R.M.F is the Version, Release, Modification, and Fix Pack number. If the value is not known, this is set to ?.?.?.?

CONNGRP

Indicates whether the group is connected to the local group and is one of:

• yes - the group is connected.
• no - the group is disconnected.
• unknown - the connection status is unknown.
• suspended - the group is connected, but the group cannot replicate data until a configuration incompatibility between the groups is resolved.

This field is displayed only for groups other than the local group.

INSYNC

This is only displayed for a group with role Recovery or Pending live. It indicates whether the group could become Live with no loss of data if a failover occurred. (Note that a Recovery group might not be in-sync with the Live group very often because the log is replicated asynchronously. Use the RCOVLSN and RCOVTIME values to indicate if data loss might occur in an unplanned failover).

SYNCTIME

This is only displayed for a group with role Recovery or Pending live. It indicates the time when this group was last in-sync with the Live group, in ISO 8601 format (blank if it is not known). (Note that a Recovery group might not be in-sync with the Live group very often because the log is replicated asynchronously. Use the RCOVLSN and RCOVTIME values to indicate if data loss might occur in an unplanned failover).

BACKLOG

This is only displayed for a group with role Recovery or Pending live. It indicates the number of KB that the group is behind the live group.

GRSTATUS

Indicates the operational status of this group and is one of the following values:

• Normal - the group is operating normally.
• Checking - the group is being checked to ensure the consistency of the recovery log.
• Synchronizing - the group is being sent data to bring it in-sync.
• Rebasing - a new recovery log is being sent to the instance.
• Waiting for connection - the group is waiting for a connection from a group with the role Live or Pending Recovery.
• Partitioned - two groups believe they are the Live group (that is, a 'split brain' status).
• Unknown - the group status is unknown.

RCOVLSN

A log sequence number (LSN) that the group could recover to, in nnnnn:nnnnn:nnnnn:nnnnn format. Any data written to the recovery log after this point might be lost if an unplanned failover occurs. This value can be passed into the -s option of the dmpmqlog command.

RCOVTIME

A time that the group could recover to, in ISO 8601 format. Any data written to the recovery log after this time might be lost if an unplanned failover occurs. This value can be passed into the -t or -u options of the dmpmqlog command.

INITLSN

This is only displayed for a group with role Live or Pending recovery. It indicates the log sequence number (LSN) of the last log record recovered when the Native HA group initially became live, in the format nnnnn:nnnnn:nnnnn:nnnnn, or blank if the group is not live.

INITTIME

This is only displayed for a group with role Live or Pending recovery. It indicates the time of the last log record recovered when the Native HA group initially became live, in ISO 8601 format. Blank, if the group is not live.

LIVETIME

This is only displayed for a group with role Live or Pending recovery. It indicates the time the group first became the live group, in ISO 8601 format. Blank, if the group is not live.

ALTDATE

Indicates the date when this information was last updated.

ALTTIME

Indicates the time when this information was last updated.

• Find the pods that are part of your queue manager.
  • On Red Hat® OpenShift®:

    oc get pod --selector app.kubernetes.io/instance=nativeha-qm

  • On Amazon EKS:

    kubectl get pod --selector app.kubernetes.io/instance=nativeha-qm

• Run the dspmq in one of the pods
  • On Red Hat OpenShift:

    oc exec -t <POD> -- dspmq

    For an interactive shell, where you can run dspmq directly.

    oc rsh <POD>

  • On Amazon EKS:

    kubectl exec -t <POD> -- dspmq

    For an interactive shell, where you can run dspmq directly.

    kubectl exec -it <POD> -- sh

• To determine whether a queue manager instance is running as the active instance or as a replica:
  • On Red Hat OpenShift:

    oc exec -t <POD> -- dspmq -o status -m <QMGR_NAME>

  • On Amazon EKS:

    kubectl exec -t <POD> -- dspmq -o status -m <QMGR_NAME>

  An active instance of a queue manager named BOB would report the following status:

  QMNAME(BOB)                 STATUS(Running)

  An instance of BOB that is a recovery group leader in a Native HA CRR configuration would report the following status:

  QMNAME(BOB)                 STATUS(Recovery group leader)

  A replica instance of a queue manager named BOB would report the following status:

  QMNAME(BOB)                 STATUS(Replica)

  An inactive instance would report the following status:

  QMNAME(BOB)                 STATUS(Ended Immediately)

• To determine Native HA operational status of the instance in the specified pod:
  • On Red Hat OpenShift:

    oc exec -t <POD> -- dspmq -o nativeha -m <QMGR_NAME>

  • On Amazon EKS:

    kubectl exec -t <POD> -- dspmq -o nativeha -m <QMGR_NAME>

  The active instance of a queue manager named BOB might report the following status:

  QMNAME(BOB)         ROLE(Active) INSTANCE(inst1) INSYNC(Yes) QUORUM(3/3) GRPLSN(<0:25:365:1000>) GRPNAME() GRPROLE(Live)

  A replica instance of a queue manager named BOB might report the following status:

  QMNAME(BOB)         ROLE(Replica) INSTANCE(inst2) INSYNC(Yes) QUORUM(2/3) GRPLSN(<0:25:365:1000>) GRPNAME() GRPROLE(Live)

  An inactive instance of a queue manager named BOB might report the following status:

  QMNAME(BOB)         ROLE(Unknown) INSTANCE(inst3) INSYNC(no) QUORUM(0/3) GRPLSN() GRPNAME() GRPROLE(Live)

  An active instance of a queue manager named BOB belonging to the Live group in a Native HA CRR configuration might report the following status:

  QMNAME(BOB)         ROLE(Active) INSTANCE(inst1) INSYNC(yes) QUORUM(3/3) GRPLSN(<0:25:365:1000>) GRPNAME(alpha) GRPROLE(Live)

  The same instance when the group was waiting to become the recovery group might report the following status:

  QMNAME(BOB)         ROLE(Active) INSTANCE(inst1) INSYNC(yes) QUORUM(3/3) GRPLSN(<0:25:365:1000>) GRPNAME(alpha) GRPROLE(Pending recovery) 

  A queue manager named CAROL that is not an instance of a Native HA or Native HA CRR configuration would report the following status:

  QMNAME(CAROL)       ROLE(Not configured) INSTANCE() INSYNC() QUORUM() GRPLSN() GRPNAME() GRPROLE(Not configured)

• To determine the Native HA operational status of all the instances in the Native HA configuration:
  • On Red Hat OpenShift:

    oc exec -t <POD> -- dspmq -o nativeha -x -m <QMGR_NAME>

  • On Amazon EKS:

    kubectl exec -t <POD> -- dspmq -o nativeha -x -m <QMGR_NAME>

  For example, if you issue this command on the node running the active instance of queue manager BOB, where all instances are in-sync, you might receive the following status:

  QMNAME(BOB)         ROLE(Active) INSTANCE(inst1) INSYNC(yes) QUORUM(2/3) GRPLSN(<0:25:365:1000>) GRPROLE(Live) GRPNAME()
   INSTANCE(inst1) ROLE(Active)  REPLADDR(localhost) CONNACTV(yes) INSYNC(yes) BACKLOG(0)  CONNINST(yes) ACKLSN(<0:26:365:1000>)  HASTATUS(Normal)        SYNCTIME(2023-12-20T17:13:27.853412Z) ALTDATE(2023-12-20) ALTTIME(17.13.27)
   INSTANCE(inst3) ROLE(Replica) REPLADDR(localhost) CONNACTV(yes) INSYNC(yes) BACKLOG(0)  CONNINST(yes) ACKLSN (<0:24:365:1000>) HASTATUS(Normal)        SYNCTIME(2023-12-20T17:13:27.853412Z) ALTDATE(2023-12-20) ALTTIME(17.13.27)
   INSTANCE(inst2) ROLE(Replica) REPLADDR(localhost) CONNACTV(yes) INSYNC(yes) BACKLOG(0)  CONNINST(yes) ACKLSN (<0:25:365:1000>) HASTATUS(Synchronizing) SYNCTIME(2023-12-20T17:13:22.123321Z) ALTDATE(2023-12-20) ALTTIME(17.13.27)
  

  In another example, if you issue this command on a node running a replica instance of queue manager BOB, which is lagging behind, you might receive the following status:

  QMNAME(BOB)         ROLE(Replica) INSTANCE(inst2) INSYNC(no) QUORUM(2/3) GRPLSN(<0:25:365:1000>) GRPROLE(Live) GRPNAME()
   INSTANCE(inst1) ROLE(Active)  REPLADDR(localhost) CONNACTV(yes) INSYNC(yes) BACKLOG(0)  CONNINST(yes) ACKLSN(<0:26:365:1000>)  HASTATUS(Normal)        SYNCTIME(2023-12-20T17:13:27.853412Z) ALTDATE(2023-12-20) ALTTIME(17.13.27)
   INSTANCE(inst3) ROLE(Replica) REPLADDR(localhost) CONNACTV(yes) INSYNC(yes) BACKLOG(0)  CONNINST(yes) ACKLSN (<0:24:365:1000>) HASTATUS(Normal)        SYNCTIME(2023-12-20T17:13:27.853412Z) ALTDATE(2023-12-20) ALTTIME(17.13.27)
   INSTANCE(inst2) ROLE(Replica) REPLADDR(localhost) CONNACTV(yes) INSYNC(no) BACKLOG(435) CONNINST(yes) ACKLSN (<0:25:365:1000>) HASTATUS(Synchronizing) SYNCTIME(2023-12-20T17:13:22.123321Z) ALTDATE(2023-12-20) ALTTIME(17.13.27)
  

  If you issue this command on a node running an inactive instance of queue manager BOB, you might receive the following status:

  QMNAME(BOB)			ROLE(Unknown) INSTANCE(inst3) INSYNC(no) QUORUM(0/3) GRPLSN() GRPROLE() GRPNAME()
   INSTANCE(inst1) ROLE(Unknown) REPLADDR(9.20.123.45) CONNACTV(Unknown) INSYNC(Unknown) BACKLOG(Unknown) CONNINST(No) ACKLSN()  HASTATUS(Unknown) SYNCTIME() ALTDATE() ALTTIME() ALTDATE() ALTTIME()
   INSTANCE(inst2) ROLE(Unknown) REPLADDR(9.20.123.46) CONNACTV(Unknown) INSYNC(Unknown) BACKLOG(Unknown) CONNINST(No) ACKLSN()  HASTATUS(Unknown) SYNCTIME() ALTDATE() ALTTIME() ALTDATE() ALTTIME()
   INSTANCE(inst3) ROLE(Unknown) REPLADDR(9.20.123.47) CONNACTV(No)      INSYNC(Unknown) BACKLOG(Unknown) CONNINST(No) ACKLSN()  HASTATUS(Unknown) SYNCTIME() ALTDATE() ALTTIME() ALTDATE() ALTTIME()
  

  If you issue the command when the instances are still negotiating which is active and which are replicas, you would receive the following status:

  QMNAME(BOB)              STATUS(Negotiating)

• To determine the operational status of all the Native HA groups in a Native HA CRR configuration:
  • On Red Hat OpenShift:

    oc exec -t <POD> -- dspmq -o nativeha -g -m <QMGR_NAME>

  • On Amazon EKS:

    kubectl exec -t <POD> -- dspmq -o nativeha -g -m <QMGR_NAME>

  If you issue this command on a member of the Live group, named alpha, you might receive the following status for alpha, and the Recovery group, beta:

  GRPNAME(alpha) GRPROLE(Live) GRPADDR(9.20.135.129) GRPVER(09040200) 
  GRSTATUS(Normal) RCOVLSN(<0:0:104:65163>) RCOVTIME(2023-12-20T17:37:27.114787Z) 
  INITLSN(<0:0:101:352>) INITTIME(2023-12-19T09:13:27.512294) 
  LIVETIME(2023-12-19T06:14:27.512294) ALTDATE(2023-12-20) ALTTIME(17.13.27)

  GRPNAME(beta) GRPROLE(Recovery) GRPADDR(9.20.135.130) GRPVER(09040300) 
  CONNGRP(yes) GRSTATUS(Synchronizing) RCOVLSN(<0:0:19:16723>) 
  RCOVTIME(2023-12-20T17:36:23.463209Z) BACKLOG(5619) INSYNC(no) SYNCTIME() 
  ALTDATE(2023-12-20) ALTTIME(17.13.27)