IBM MQ errors like 'MQRC_CONNECTION_BROKEN' or 'com.ibm.msg.client.jms.DetailedJMSException' appear in log

Troubleshooting

Problem

Multiple IBM MQ errors like 'MQRC_CONNECTION_BROKEN' or 'com.ibm.msg.client.jms.DetailedJMSException' appear in log.

Symptom

Multiple occurrences of the following errors can be seen in the logs (Similar errors may appear in the Graylog).

----------------

21:30:27,931:ERRORDTL:Default Executor-thread-1611: [1692307827923]com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
   at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
   at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:458)
   at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7ProviderConnection(WMQConnectionFactory.java:8683)
   at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:8023)
   at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl._createConnection(JmsConnectionFactoryImpl.java:322)
   at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:242)
   at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6026)
   at com.ibm.mq.jms.MQQueueConnectionFactory.createQueueConnection(MQQueueConnectionFactory.java:111)
   at com.ibm.mq.jms.MQQueueConnectionFactory.createConnection(MQQueueConnectionFactory.java:200)
   at com.yantra.interop.services.jms.JMSResourceProxy.createConnection(JMSResourceProxy.java:138)
   at com.yantra.interop.services.jms.JMSProducer.getMainJMSObjectKey(JMSProducer.java:740)
   at com.yantra.interop.services.jms.JMSProducer.connectToMainJMS(JMSProducer.java:651)
   at com.yantra.interop.services.jms.JMSProducer.access$500(JMSProducer.java:63)
   at com.yantra.interop.services.jms.JMSProducer$1.retry(JMSProducer.java:530)
   at com.yantra.interop.services.jms.JMSProducer.executeWithRetry(JMSProducer.java:386)
   at com.yantra.interop.services.jms.JMSProducer.connectAndSendToCurrentServer(JMSProducer.java:525)
   at com.yantra.interop.services.jms.JMSProducer.connectAndSend(JMSProducer.java:424)
   at com.yantra.interop.services.jms.JMSProducer.send(JMSProducer.java:326)
   at com.yantra.interop.services.jms.JMSProducer.send(JMSProducer.java:279)
   at com.yantra.interop.services.jms.JMSContextMediator.dispatchMessage(JMSContextMediator.java:146)
   at com.yantra.interop.services.jms.JMSCallback.sendMessage(JMSCallback.java:64)
   at com.yantra.interop.services.flowcomponents.TransportFlowComponent.execute(TransportFlowComponent.java:116)

----------------

21:30:05,192:ERROR:Default Executor-thread-1868: [1692307805182]<?xml version="1.0" encoding="UTF-8"?>
<Errors>
<Error ErrorCode="com.ibm.msg.client.jms.DetailedJMSException"
ErrorDescription="Error description not available" ErrorRelatedMoreInfo="">
<Attribute Name="ErrorCode" Value="com.ibm.msg.client.jms.DetailedJMSException"/>
<Attribute Name="ErrorDescription" Value="Error description not available"/>
<Error ErrorCode="com.ibm.msg.client.jms.DetailedJMSException"
ErrorDescription="" ErrorRelatedMoreInfo="JMSWMQ2008: Failed to open MQ queue 'IV_DEMAND_EVENT_Q'.">
<Stack>com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ2008: Failed to open MQ queue 'IV_DEMAND_EVENT_Q'.
JMS attempted to perform an MQOPEN, but IBM MQ reported an error.
Use the linked exception to determine the cause of this error. Check that the specified queue and queue manager are defined correctly.
   at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:595)
   at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
   at com.ibm.msg.client.wmq.internal.WMQMessageProducer.checkJmqiCallSuccess(WMQMessageProducer.java:1358)
   at com.ibm.msg.client.wmq.internal.WMQMessageProducer.checkJmqiCallSuccess(WMQMessageProducer.java:1313)
   at com.ibm.msg.client.wmq.internal.WMQMessageProducer.access$800(WMQMessageProducer.java:75)
   at com.ibm.msg.client.wmq.internal.WMQMessageProducer$SpiIdentifiedProducerShadow.initialise(WMQMessageProducer.java:900)

Cause

One of the probable cause for IBM MQ failover in Order Management on Cloud (OMoC) environments is frequent infrastructure, application, security updates. IBM Sterling Order Management System has a weekly scheduled maintenance window to ensure reliability and security on all OMoC environments.

Environment

IBM Sterling Order Management System (MC, Dev, QA, Pre-prod, Prod)

Diagnosing The Problem

QA, Dev and MC environments (where applicable) would each experience a single 3-5 minute period of unavailability at some point in time between the start time of this maintenance and 24hrs later. Preproduction and Production environments will not exhibit any downtime.

Check the maintenance time from the SST event calendar. IBM MQ servers may also be restarted one or two times within a window of up to 5 hours after this event’s start time to apply updates. Check whether MQ errors are coming during maintenance window.

Resolving The Problem

All external services and IBM MQ clients can automatically reconnect to IBM MQ servers in the event of a brief restart. It is recommended to configure OMS services with an automatic retry interval by following the steps in this technote: https://www.ibm.com/support/pages/node/6601679.

Related Information

IBM Sterling Order Management System product overview [Refer section - Maintena…

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB59","label":"Sustainability Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS6PEW","label":"IBM Sterling Order Management"},"ARM Category":[{"code":"a8m0z000000cy06AAA","label":"Services and MQ"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Tips