APAR status
Closed as program error.
Error description
A deadlock can occur in the WebSphere MQ resource adapter while closing the ServerSessionPool associated with an activation specification. A Javacore (thread dump) of the application server JVM shows the following two threads in a logical deadlock: "WebSphere non-WLM Dispatch Thread t=00abae88" J9VMThread:0x000000003BA58600, j9thread_t:0x000000480908B488, java/lang/Thread:0x000000008E5DE4C0, state:CW, prio=5 (native thread ID:0x3B321000, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000101) Waiting on: com/ibm/mq/connector/inbound/ServerSessionImpl$IsInUseLock@0x000 000009304F648 Owned by: <unowned> Java callstack: at java/lang/Object.wait at java/lang/Object.wait at com/ibm/mq/connector/inbound/ServerSessionImpl.close (entered lock: com/ibm/mq/connector/inbound/ServerSessionImpl$IsInUseLock@0x000 000009304F648, entry count: 1) at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.closeInternal at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.close at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stopDeliv eryASF at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stop at com/ibm/mq/connector/ResourceAdapterImpl.endpointDeactivation (entered lock: java/util/HashMap@0x0000000083E0D2C0, entry count: 1) at com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateUnderRAClass LoaderContext at com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateEndPoint at com/ibm/ejs/j2c/mbeans/MessageEndpointMBeanImpl.pause (entered lock: com/ibm/ejs/j2c/mbeans/MessageEndpointMBeanImpl@0x000000008B5454 88, entry count: 1) at sun/reflect/NativeMethodAccessorImpl.invoke0 ... "WebSphere WLM Dispatch Thread t=00ac20b8" J9VMThread:0x000000003B9BA300, j9thread_t:0x000000480916C4D0, java/lang/Thread:0x000000008E5BBAD0, state:CW, prio=5 (java/lang/Thread getId:0x9E, isDaemon:true) (native thread ID:0x3A9C4400, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000101) Waiting on: com/ibm/mq/connector/inbound/ServerSessionPoolImpl$CloseState@0x 000000009304F130 Owned by: <unowned> Java callstack: at java/lang/Object.wait at java/lang/Object.wait at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.close (entered lock: com/ibm/mq/connector/inbound/ServerSessionPoolImpl$CloseState@0x 000000009304F130, entry count: 2) at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stopDeliv eryASF at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stop at com/ibm/mq/connector/inbound/MessageEndpointDeployment.onExcepti on (entered lock: com/ibm/ejs/container/MessageEndpointFactoryImpl@0x000000008AF39 900, entry count: 1) at com/ibm/mq/connector/inbound/AbstractWorkImpl.run at com/ibm/mq/connector/inbound/ServerSessionImpl.start at com/ibm/ws/wmqra/zdispatcher/WASServantConnectionConsumer.delive rMessagesFromReference at com/ibm/ws/wmqra/zdispatcher/WASServantConnectionConsumerReposit ory.dispatch at com/ibm/ws/wmqra/zdispatcher/WASServantDispatchTarget.dispatch at com/ibm/ws/sib/z/dispatcher/WorkObjectImpl.fireWork at com/ibm/ws390/sib/dispatcher/WorkObjectProxyImpl.dispatchWorkObj ectImpl ...
Local fix
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of the: - WebSphere MQ V7.1 resource adapter - WebSphere MQ V7.5 resource adapter - IBM MQ V8 resource adapter - IBM MQ V9 resource adapter - WebSphere Application Server V8.5 MQ messaging provider - WebSphere Application Server V9 MQ messaging provider who use activation specifications to deliver messages asynchronously to message-driven bean (MDB) applications. Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: A deadlock within the MQ resource adapter occurred in the scenario when: 1) an XA transaction error was thrown (for example, because of a broken TCP/IP connection between the application server and queue manager) while a ServerSession thread was attempting to deliver a message to a message-driven bean (MDB) application (see the thread named "WebSphere WLM Dispatch Thread t=00ac20b8" in the Problem Description), and 2) another thread was already attempting to stop the message endpoint and close the ServerSessionPool associated with an activation specification (see the thread named "WebSphere non-WLM Dispatch Thread t=00abae88" in the Problem Description). This was because the thread stopping the message endpoint was attempting to close off ServerSessions in the activation specification's ServerSessionPool. While doing this, it found a ServerSession that was being used to try and deliver a message to an MDB, so it blocked waiting for that ServerSession to finish and be returned to the ServerSessionPool before continuing to close it. However, once the thread that was running the ServerSession experienced a XA error during message delivery it also attempted to perform some clean-up, before starting reconnection retry processing. This included closing the ServerSessionPool associated with the activation specification. The thread running the ServerSession determined that another thread was already processing a close of the ServerSessionPool and so blocked, waiting for that operation to complete. However, this thread could not complete the pool closure because the ServerSession that was in use had not been returned to the pool. As such, neither threads could progress.
Problem conclusion
The MQ resource adapter has been updated such that when a thread is closing an activation specification's ServerSessionPool it does not wait for ServerSessions that are processing a message delivery failure to be returned to the pool before they are closed. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v7.1 7.1.0.9 v7.5 7.5.0.9 v8.0 8.0.0.8 v9.0 LTS 9.0.0.2 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT19154
Reported component name
WMQ WINDOWS V7
Reported component ID
5724H7220
Reported release
710
Status
CLOSED PER
PE
NoPE
HIPER
YesHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2017-02-07
Closed date
2017-06-30
Last modified date
2017-06-30
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WMQ WINDOWS V7
Fixed component ID
5724H7220
Applicable component levels
R710 PSY
UP
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1"}]
Document Information
Modified date:
08 March 2021