APAR status
Closed as program error.
Error description
A WebSphere MQ Resource Adapter (WMQ-RA) is being used within a WebSphere Application Server environment to drive an Activation Specification, consuming messages from a queue where the queue manager is being communicated with using CLIENT mode transport. The Activation Specification is observed to stop consuming messages from the queue, which in turn stops the application's MDB threads from running, even though there are messages available for consumption on the queue. A thread dump taken of the JVM when it is in state shows the following pair of threads: DispatchThread -------------- com/ibm/mq/connector/inbound/ServerSessionPoolImpl.getServerSess ion at com/ibm/msg/client/jms/internal/JmsConnectionConsumerImpl$JmsMes sageReferenceHandlerImpl.endDeliverInternal at com/ibm/msg/client/jms/internal/JmsConnectionConsumerImpl$JmsMes sageReferenceHandlerImpl.handleMessageReference at com/ibm/msg/client/jms/internal/JmsConnectionImpl$JmsProviderMes sageRefHandler.handleMessageReference at com/ibm/msg/client/wmq/internal/WMQConnectionBrowser$WMQConnecti onBrowserShadow.consumer at com/ibm/mq/jmqi/remote/impl/RemoteProxyQueue.driveConsumer at com/ibm/mq/jmqi/remote/impl/RemoteProxyQueue.deliverMsgs at com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.deliverMsgsReco nnectable at com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.deliverMsgs at com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.run at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.runTas k at com/ibm/msg/client/commonservices/workqueue/SimpleWorkQueueItem. runItem at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.run at com/ibm/ws/wmqcsi/workqueue/WorkQueueManagerImpl$WorkQueueRunnab le.run at java/lang/Thread.run "WMQJCAResourceAdapter : nnnnn" ------------------------------- at java/lang/Object.wait at java/lang/Object.wait at com/ibm/mq/jmqi/remote/util/ReentrantMutex.acquire (entered lock: com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x00000007A6 EE7058, entry count: 2) at com/ibm/mq/jmqi/remote/util/ReentrantMutex.acquire (entered lock: com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x00000007A6 EE7058, entry count: 1) at com/ibm/mq/jmqi/remote/api/RemoteHconn.requestDispatchLock at com/ibm/mq/jmqi/remote/api/RemoteFAP.MQCTL at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.suspendAs yncService at com/ibm/msg/client/wmq/internal/WMQConnection.suspendAsyncServic e at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close (entered lock: java/lang/Object@0x00000007A3FD90A8, entry count: 1) at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close at com/ibm/msg/client/wmq/internal/WMQSession.stop (entered lock: java/lang/Object@0x00000007A80C3468, entry count: 1) at com/ibm/msg/client/jms/internal/JmsSessionImpl.stop at com/ibm/msg/client/jms/internal/JmsSessionImpl.close at com/ibm/msg/client/jms/internal/JmsSessionImpl.close at com/ibm/mq/jms/MQSession.close at com/ibm/mq/connector/inbound/ServerSessionImpl.close at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.sessionReleas ed (entered lock: java/util/ArrayList@0x00000007A6EE83E8, entry count: 1) at com/ibm/mq/connector/inbound/MessageEndpointDeployment.releaseSe ssion at com/ibm/mq/connector/inbound/MessageEndpointDeployment.workCompl eted at com/ibm/ejs/j2c/work/WorkProxy.run at com/ibm/ws/util/ThreadPool$Worker.run
Local fix
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of the: - WebSphere MQ V7.1 JCA Resource Adapter - WebSphere MQ V7.5 JCA Resource Adapter - IBM MQ V8 JCA Resource Adapter - IBM MQ V9 JCA Resource Adapter who are using an Activation Specification to consume messages from an MQ Queue or Topic when connected to the queue manager either in CLIENT transport (using TCP/IP) or BINDINGs transport, where there are repeated failures which occur during the internal work associated with processing the message within the MDB before the MDB is driven, for example caused by repeated network failures which disconnect the application server from the queue manager. Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: This problem is broadly similar to that documented under APAR IT15559: http://www.ibm.com/support/docview.wss?uid=swg1IT15559 in that a deadlock occurs between two threads associated with the MQ Activation Specification as a result of attempting to close a ServerSession, while simultaneously driving new work through the Activation Specification. In IT15559, the ServerSession was being closed by the Pool Scavenger Thread as a result of a ServerSession being unused in the pool for a time which exceeded the pool timeout value. The distinction between that APAR and this one is that here, a ServerSession is being closed as a result of the following JEE method call throwing an exception to indicate a problem had occurred in the pre-processing of an MDB's message: javax.resource.spi.endpoint.MessageEndpoint.beforeDelivery(Metho d) For example, if the ServerSession's associated javax.jms.Session was no longer connected to the queue manager as a result of a network disconnection event, and two-phase commit (XA) was being used to run the transaction for the MDB, the application server would be unable to start the XA transaction to enlist the queue manager and the above method would fail. In this circumstance, the code design is to dispose the ServerSession and associated JMS Session and create a replacement to drive future work under the MDB. The subsequent close() call on the ServerSession could result in deadlock with the Activation Specification's DispatchThread, if in parallel more work was being driven by another message being available on the queue.
Problem conclusion
Similar to design change associated with IT15559, the code has been updated such that Activation Specification pool lock is no longer held when closing the ServerSession, which circumvents the deadlock. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v7.5 7.5.0.8 v8.0 8.0.0.7 v9.0 CD 9.0.2 v9.0 LTS 9.0.0.1 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT18624
Reported component name
WMQ WINDOWS V7
Reported component ID
5724H7220
Reported release
710
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2017-01-03
Closed date
2017-01-30
Last modified date
2017-06-01
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WMQ WINDOWS V7
Fixed component ID
5724H7220
Applicable component levels
R710 PSY
UP
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1"}]
Document Information
Modified date:
08 March 2021