APAR status
Closed as program error.
Error description
An message-driven bean (MDB) application, which is configured to run from an MQ queue using the IBM MQ classes for JMS, stops consuming messages after a period of time. The MDB shares a "javax.jms.ServerSessionPool" object with other MDBs which are running in the same JVM, which are consuming messages from different MQ queues. A Javacore taken when the messages had stopped processing messages shows Java call stacks of the form: "Thread-54" (id: 84, state: WAITING) priority=5, interrupted=false, daemon=false at java.lang.Object.wait at java.lang.Object.wait at com.ibm.mq.jmqi.remote.util.ReentrantMutex.acquire at com.ibm.mq.jmqi.remote.util.ReentrantMutex.acquire at com.ibm.mq.jmqi.remote.api.RemoteHconn.requestDispatchLock at com.ibm.mq.jmqi.remote.api.RemoteFAP.MQCTL at com.ibm.mq.jmqi.monitoring.JmqiInterceptAdapter.MQCTL at com.ibm.msg.client.wmq.internal.WMQConsumerOwnerShadow.suspendAs yncService at com.ibm.msg.client.wmq.internal.WMQConnection.suspendAsyncServic e at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close at com.ibm.msg.client.wmq.internal.WMQSession.loadMessageReference at com.ibm.msg.client.jms.internal.JmsSessionImpl.consume at com.ibm.msg.client.jms.internal.JmsSessionImpl.run at com.ibm.mq.jms.MQSession.run at myApplication.run at java.lang.Thread.run "DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[:/66477451][connectio nId=414D51436D7166766A6D73312020202037B6B0593A0C3822]]" (id: 88, state: WAITING) priority=5, interrupted=false, daemon=true at java.lang.Object.wait at java.lang.Object.wait at my.ServerSessionPool.implementation.getServerSession at com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes sageReferenceHandlerImpl.endDeliverInternal at com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes sageReferenceHandlerImpl.endDeliver at com.ibm.msg.client.jms.internal.JmsConnectionImpl$JmsProviderMes sageRefHandler.endDeliver at com.ibm.msg.client.wmq.internal.WMQConnectionBrowser$WMQConnecti onBrowserShadow.consumer at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.callConsumer at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.run at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTas k at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem. runItem at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.run WorkQueueItem at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManage rImplementation$ThreadPoolWorker.run
Local fix
Do not share a: javax.jms.ServerSessionPool between MDBs. Create a new instance for each MDB which is running within the JVM.
Problem summary
**************************************************************** USERS AFFECTED: This issue affects message-driven bean (MDB) applications deployed into application server environments that create multiple JMS "javax.jms.ConnectionConsumer" objects to consume messages from different destinations, where the JMS ConnectionConsumers share the same "javax.jms.ServerSessionPool" object instance. Activation Specification implementations which are provided by the IBM MQ JCA Resource Adapter are not affected by this problem, as this Resource Adapter does not share a "javax.jms.ServerSessionPool" between different Activation Specifications. This means that MQ Activation Specifications running within IBM's WebSphere Application Server are not affected by this problem. However it can affect a Listener Port running within the WebSphere Application Server, if a ListenerPort for one MDB is stopped, and another one is started to consume messages from a different Destination on the same queue manager, due to the way in which the WebSphere Application Server pools JMS Sessions. Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: A deadlock may occur in an application using a message-driven bean (MDB) which uses a "javax.jms.ServerSessionPool" implementation which is shared across multiple MDBs which are configured to consume messages from different MQ destinations. Prior to APAR IT10205: http://www.ibm.com/support/docview.wss?uid=swg1IT10205 in order to encounter the deadlock, you must have a ServerSessionPool maximum size of a single ServerSession. A new lock introduced under APAR IT10205 results in you being able to encounter this problem with any ServerSessionPool size. This APAR was introduced in MQ versions: MQ 8.0.0.5 MQ 9.0.0.0 The problem occurred when the MQ "DispatchThread" (which is a named thread associated with the MDB that browses for messages on the MQ Destination, configures a ServerSession with information about how to identify the browsed message, and requests that it initiates its MDB processing) attempted to get a ServerSession from the implemented "javax.jms.ServerSessionPool". However the DispatchThread was unable to obtain a free ServerSession instance from the pool as all ServerSessions were in use. In this scenario, the DispatchThread would block at that point on the "javax.jms.ServerSessionPool" implementation, waiting for a ServerSession to be returned to the pool following the completion of an running MDB onMessage() instance. This is seen in a Javacore with a stack of the form: "DispatchThread" (id: 88, state: WAITING) priority=5, interrupted=false, daemon=true at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at my.ServerSessionPoolImplementation.getServerSession at com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes sageReferenceHandlerImpl.endDeliverInternal at com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes sageReferenceHandlerImpl.endDeliver at com.ibm.msg.client.jms.internal.JmsConnectionImpl$JmsProviderMes sageRefHandler.endDeliver at com.ibm.msg.client.wmq.internal.WMQConnectionBrowser$WMQConnecti onBrowserShadow.consumer at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.callConsumer at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.run ... ... ... However, the running MDB instance thread was currently blocked waiting for a lock which the above DispatchThread holds. This thread had been asked to start its MDB processing cycle, which involves destructively consuming the message from the MQ Destination. When consuming messages to deliver to MDB instances in this way (browse-then-get), which is referred to as the Application Server Facilities mode of operation, IBM MQ's implementation of a JMS Session that is associated with a ServerSession uses a cached "javax.jms.MessageConsumer". This is in order to optimise the consumption of messages from a particular queue by preventing the need to perform a new MQOPEN request each time the ServerSession runs. Because a "javax.jms.ServerSessionPool" implementation may contain ServerSessions which are being used to consume messages from multiple destinations, the MQ classes for JMS must verify that the message it has been told to go and consume to drive the MDB's "onMessage(javax.jms.Message)" method, is on the same Destination as the cached MessageConsumer is configured to consume messages from. If the MessageConsumer does not match the Destination, the ServerSession closes the existing MessageConsumer, and creates a new one for the new Destination. However in order to do this, the thread executing the ServerSession processing required a lock owned by the "DispatchThread". This result was that this thread gets blocked with a Java stack of the form: "Thread-1" (id: 84, state: WAITING) priority=5, interrupted=false, daemon=false at java.lang.Object.wait at java.lang.Object.wait at com.ibm.mq.jmqi.remote.util.ReentrantMutex.acquire at com.ibm.mq.jmqi.remote.util.ReentrantMutex.acquire at com.ibm.mq.jmqi.remote.api.RemoteHconn.requestDispatchLock at com.ibm.mq.jmqi.remote.api.RemoteFAP.MQCTL at com.ibm.mq.jmqi.monitoring.JmqiInterceptAdapter.MQCTL at com.ibm.msg.client.wmq.internal.WMQConsumerOwnerShadow.suspendAs yncService at com.ibm.msg.client.wmq.internal.WMQConnection.suspendAsyncServic e at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close at com.ibm.msg.client.wmq.internal.WMQSession.loadMessageReference at com.ibm.msg.client.jms.internal.JmsSessionImpl.consume at com.ibm.msg.client.jms.internal.JmsSessionImpl.run at com.ibm.mq.jms.MQSession.run at myApplication.run at java.lang.Thread.run This thread had also taken a new lock which was introduced under APAR IT10205, which prevents other ServerSession threads who are also trying to close their respective MessageConsumers to replace with one for associated with the correct MQ Destination. These threads have the form: at sun.misc.Unsafe.park at java.util.concurrent.locks.LockSupport.park at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndChe ckInterrupt at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQue ued at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQue ued at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock at java.util.concurrent.locks.ReentrantLock.lock at com.ibm.msg.client.wmq.internal.WMQConsumerOwnerShadow.suspendAs yncService at com.ibm.msg.client.wmq.internal.WMQConnection.suspendAsyncServic e at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close at com.ibm.msg.client.wmq.internal.WMQSession.loadMessageReference at com.ibm.msg.client.jms.internal.JmsSessionImpl.consume at com.ibm.msg.client.jms.internal.JmsSessionImpl.run at com.ibm.mq.jms.MQSession.run at myApplication.run at java.lang.Thread.run and would be observed if the ServerSessionPool size is greater than 1 when the problem is encountered. The net result of this was that the "DispatchThread" is stuck waiting for a ServerSession to be returned to the pool. However all of the ServerSessions were stuck on a thread which is attempting to get a lock which the "DispatchThread" was holding. The result is that the MDB is deadlocked.
Problem conclusion
The IBM MQ classes for JMS has been updated such that when closing the "javax.jms.MessageConsumer" associated with the ServerSession, the lock held by the "DispatchThread" is no longer requested, which prevents the deadlock from happening. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v8.0 8.0.0.9 v9.0 LTS 9.0.0.4 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT22327
Reported component name
IBM MQ BASE M/P
Reported component ID
5724H7261
Reported release
900
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2017-09-08
Closed date
2018-01-29
Last modified date
2018-05-09
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
IBM MQ BASE M/P
Fixed component ID
5724H7261
Applicable component levels
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"9.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
09 May 2018