APAR status
Closed as program error.
Error description
When using the WMQ 7.1 Resource Adapter (supplied with WebSphere Application Server v8.5.5.9), listener ports connecting using BINDINGS transport hang when being shut down. The WebSphere Application Server SystemOut.log shows: [5/8/16 1:25:57:383 CDT] 000000a2 ThreadMonitor W WSVR0605W: Thread "MessageListenerThreadPool : 3" (000000f5) has been active for 622751 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung. at com.ibm.msg.client.wmq.internal.WMQSession .loadMessageReference (WMQSession.java:1207) A Javacore taken at the time of the hang shows the following four threads of interest: **************************************************************** ******* Thread 1: ------------ "Jmqi AsyncConsume Thread. tid: <number> Waiting on: java/util/Vector@0x00000000D1170F98 Owned by: <unowned> Java callstack: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:172(Compiled Code)) at com/ibm/ejs/jms/listener/ServerSessionPool. getServerSession(ServerSessionPool.java:452(Compiled Code)) (entered lock: java/util/Vector@0x00000000D1170F98, entry count: 1) at com/ibm/msg/client/jms/internal/ JmsConnectionConsumerImpl$JmsMessageReferenceHandlerImpl. endDeliverInternal (JmsConnectionConsumerImpl.java:338(Compiled Code)) at com/ibm/msg/client/jms/internal/ JmsConnectionConsumerImpl$JmsMessageReferenceHandlerImpl. handleMessageReference (JmsConnectionConsumerImpl.java:456(Compiled Code)) at com/ibm/msg/client/jms/internal/ JmsConnectionImpl$JmsProviderMessageRefHandler. handleMessageReference (JmsConnectionImpl.java:1593(Compiled Code)) at com/ibm/msg/client/wmq/internal/ WMQConnectionBrowser$WMQConnectionBrowserShadow. consumer(WMQConnectionBrowser.java:872(Compiled Code)) at com/ibm/mq/jmqi/local/internal/LocalProxyConsumer. jmqiConsumerMethod(LocalProxyConsumer.java:222(Compiled Code)) **************************************************************** ******* Thread 2: ---------- "Thread-<number>" Waiting on: com/ibm/mq/jmqi/JmqiWorkerThread@0x00000000D1170F38 Owned by: "JmqiWorkerThread" Java callstack: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:172(Compiled Code)) at com/ibm/mq/jmqi/JmqiWorkerThread. syncExec(JmqiWorkerThread.java:179(Compiled Code)) (entered lock: com/ibm/mq/jmqi/JmqiWorkerThread@0x00000000D1170F38, entry count: 1) (entered lock: java/lang/Object@0x00000000D117C3D8, entry count: 1) at com/ibm/mq/jmqi/local/LocalHconn.syncExec (LocalHconn.java:501(Compiled Code)) at com/ibm/mq/jmqi/local/LocalMQ.MQCTL(LocalMQ.java:2933) at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow. suspendAsyncService(WMQConsumerOwnerShadow.java:448(Compiled Code)) at com/ibm/msg/client/wmq/internal/WMQConnection. suspendAsyncService(WMQConnection.java:2235) at com/ibm/msg/client/wmq/internal/WMQConsumerShadow. close(WMQConsumerShadow.java:325) (entered lock: java/lang/Object@0x00000000D15088A0, entry count: 1) at com/ibm/msg/client/wmq/internal/WMQConsumerShadow. close(WMQConsumerShadow.java:279) at com/ibm/msg/client/wmq/internal/ WMQSession.stop(WMQSession.java:1544(Compiled Code)) (entered lock: java/lang/Object@0x00000000D15082E0, entry count: 1) at com/ibm/msg/client/jms/internal/JmsSessionImpl. stop(JmsSessionImpl.java:2227(Compiled Code)) at com/ibm/msg/client/jms/internal/JmsSessionImpl. stop(JmsSessionImpl.java:2212(Compiled Code)) at com/ibm/msg/client/jms/internal/JmsConnectionImpl. stop(JmsConnectionImpl.java:928(Compiled Code)) (entered lock: com/ibm/msg/client/jms/internal/State@0x00000000D11F4198, entry count: 1) at com/ibm/mq/jms/MQConnection.stop(MQConnection.java:459(Compiled Code)) at com/ibm/ejs/jms/JMSConnectionHandle.stop(JMSConnectionHandle.jav a:774) (entered lock: java/lang/String@0x00000000D11737C8, entry count: 1) at com/ibm/ejs/jms/listener/MDBListenerImpl$2. run(MDBListenerImpl.java:1182) **************************************************************** ******* Thread 3: ----------- "JmqiWorkerThread" Java callstack: at com/ibm/mq/jmqi/local/internal/base/Native.MQCTL(Native Method) at com/ibm/mq/jmqi/local/LocalMQ$10.run(LocalMQ.java:2948) at com/ibm/mq/jmqi/JmqiWorkerThread.run(JmqiWorkerThread.java:245) (entered lock: com/ibm/mq/jmqi/JmqiWorkerThread@0x00000000D1170F38, entry count: 1) at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem. runTask(WorkQueueItem.java:209) at com/ibm/msg/client/commonservices/workqueue/SimpleWorkQueueItem. runItem(SimpleWorkQueueItem.java:100) at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem. run(WorkQueueItem.java:224) at com/ibm/ws/wmqcsi/workqueue/WorkQueueManagerImpl$WorkQueueRunnab le. run(WorkQueueManagerImpl.java:550) at java/lang/Thread.run(Thread.java:809) **************************************************************** ******* Thread 4 ----------- "MessageListenerThreadPool : <number" Blocked on: java/lang/Object@0x00000000D15082E0 Owned by: "<number>" Java callstack: at com/ibm/msg/client/wmq/internal/WMQSession. loadMessageReference(WMQSession.java:1207(Compiled Code)) at com/ibm/msg/client/jms/internal/JmsSessionImpl. consume(JmsSessionImpl.java:3268(Compiled Code)) at com/ibm/msg/client/jms/internal/JmsSessionImpl. run(JmsSessionImpl.java:2903(Compiled Code)) at com/ibm/msg/client/jms/internal/JmsXAQueueSessionImpl$1. run(JmsXAQueueSessionImpl.java:1007(Compiled Code)) at com/ibm/mq/jms/MQSession.run(MQSession.java:958(Compiled Code)) at com/ibm/ejs/jms/JMSSessionHandle. run(JMSSessionHandle.java:1056(Compiled Code)) at com/ibm/ejs/jms/listener/ServerSession. connectionConsumerOnMessage (ServerSession.java:1090(Compiled Code)) at com/ibm/ejs/jms/listener/ServerSession. onMessage(ServerSession.java:760(Compiled Code)) at com/ibm/ejs/jms/listener/ServerSession. dispatch(ServerSession.java:726(Compiled Code)) at sun/reflect/GeneratedMethodAccessor73. invoke(Bytecode PC:32(Compiled Code)) at sun/reflect/DelegatingMethodAccessorImpl. invoke(DelegatingMethodAccessorImpl.java:56(Compiled Code)) at java/lang/reflect/Method.invoke(Method.java:620(Compiled Code)) at com/ibm/ejs/jms/listener/ServerSessionDispatcher. dispatch(ServerSessionDispatcher.java:47(Compiled Code)) at com/ibm/ejs/container/MDBWrapper. onMessage(MDBWrapper.java:100(Compiled Code)) at com/ibm/ejs/container/MDBWrapper. onMessage(MDBWrapper.java:132(Compiled Code)) at com/ibm/ejs/jms/listener/ ServerSession.run(ServerSession.java:581(Compiled Code)) at com/ibm/ws/util/ ThreadPool$Worker.run(ThreadPool.java:1881(Compiled Code)) **************************************************************** *******
Local fix
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of: - The WebSphere Application Server V8.5.5.9 WebSphere MQ messaging provider who have message-driven bean applications that have been deployed against a a Listener Port which has: - Been configured to connect to a queue manager using the BINDINGS transport. - The Maximum sessions property set to the value 1. Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: When a Listener Port that has been configured to connect to a queue manager using the BINDINGS transport starts up, it will register a callback with the queue manager on the queue or topic it has been configured to monitor. When a message arrives on the JMS Destination, the queue manager calls back to the main Listener Port thread, passing it a message token representing the message. The main Listener Port thread will then: - Get a Server Session from it's Server Session Pool. - Construct an internal message reference object, that represents the WebSphere MQ message that has been detected. - Load the message reference into the Server Session. - Run the Server Session on a new thread. The Server Session uses the information in the message reference to get the message, and pass it to an instance of a message-driven bean application for processing. When the message has been processed, and the message-driven bean's onMessage() method has completed, the Server Session returns itself to the Server Session Pool where it is available for reuse. When the hang reported by this APAR occurred, the following sequence of events had taken place: Thread 1: ----------- - The queue manager finds a message on the queue being monitored by the Listener Port, and calls back to the main Listener Port thread passing it a message token for the message. - The Listener Port thread constructs a message reference containing that message token, gets a Server Session from it's Server Session Pool and then starts that Server Session. - The main Listener Port thread then waits for the queue manager to call it back again. - A few milliseconds later, the queue manager finds a second message on the queue, and calls back to the main Listener Port thread again. - The Listener Port thread constructs a message reference for this message, and tries to get a Server Session from the Server Session Pool. However, only Server Session in the Server Session Pool is in use, so Thread 1 becomes blocked. Thread 2: --------- - At this point, a request is made to stop the application server. While processing this request, the Listener Port starts shutting down. - As part of the processing of this method, the thread takes an internal lock. - The thread then issues a WebSphere MQ MQCTL API call, passing in the option MQOP_SUSPEND, to suspend the call back that has been registered with the queue manager for the Listener Port. - Because the Listener Port is connecting to the queue manager using the BINDINGS transport, a new worker thread is started to perform the MQCTL call. - This thread then waits for the worker thread to complete. Thread 3: ---------- - The worker thread starts, and calls into some native queue manager code to perform the actual MQCTL processing. - This thread has to wait for any call backs that are currently in progress to complete before the MQCTL API call can return. - There is one outstanding call back, which is currently running on Thread 1. As a result, Thread 3 has to wait for Thread 1 to finish before it can continue. Thread 4: ---------- - While Thread 2 was performing it's processing, the JMS Server Session started by Thread 1 finally starts. - This thread starts processing the message reference, and then becomes blocked waiting for the lock held by Thread 2. At this point, a deadlock occurs: - Thread 1 is waiting for the Server Session being used by Thread 4 to be put back in the Server Session Pool. - Thread 4 is waiting for an internal lock held by Thread 2. - Thread 2 is waiting for the worker thread Thread 3 to finish processing the MQCTL API call. - Thread 3 is waiting for the call back running on Thread 1 to complete.
Problem conclusion
The WebSphere MQ Resource Adapter (the component of WebSphere Application Server that handles all of the communication with WebSphere MQ) has been updated to ensure that, when a Listener Port that is connecting to a queue manager using the BINDINGS transport is stopped, any message references which have not yet been processed will be discarded. This allows the Listener Port to shut down as expected, and prevents the deadlock mentioned in this APAR from occurring. The messages mentioned within the message references will remain on the destination that the Listener Port was monitoring, and will be available to be processed when the Listener Port is restarted. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v7.1 7.1.0.9 v7.5 7.5.0.8 v8.0 8.0.0.6 v9.0 CD 9.0.2 v9.0 LTS 9.0.0.1 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IV85254
Reported component name
WMQ LIN X86 V7
Reported component ID
5724H7224
Reported release
710
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2016-05-27
Closed date
2016-11-25
Last modified date
2017-08-02
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WMQ LIN X86 V7
Fixed component ID
5724H7224
Applicable component levels
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1"}]
Document Information
Modified date:
08 March 2021