IT19154: Deadlock in the MQ resource adapter within WebSphere AppServer closing the ServerSessionPool after a broken connection.

APAR status

• Closed as program error.

Error description

• A deadlock can occur in the WebSphere MQ resource adapter while
  closing the ServerSessionPool associated with an activation
  specification.  A Javacore (thread dump) of the application
  server JVM shows the following two threads in a logical
  deadlock:
  
  "WebSphere non-WLM Dispatch Thread t=00abae88"
  J9VMThread:0x000000003BA58600, j9thread_t:0x000000480908B488,
  java/lang/Thread:0x000000008E5DE4C0, state:CW, prio=5
         (native thread ID:0x3B321000, native priority:0x5, native
  policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000101)
  Waiting on:
  com/ibm/mq/connector/inbound/ServerSessionImpl$IsInUseLock@0x000
  000009304F648 Owned by: <unowned>
  Java callstack:
    at java/lang/Object.wait
    at java/lang/Object.wait
    at com/ibm/mq/connector/inbound/ServerSessionImpl.close
       (entered lock:
  com/ibm/mq/connector/inbound/ServerSessionImpl$IsInUseLock@0x000
  000009304F648, entry count: 1)
    at
  com/ibm/mq/connector/inbound/ServerSessionPoolImpl.closeInternal
    at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.close
    at
  com/ibm/mq/connector/inbound/MessageEndpointDeployment.stopDeliv
  eryASF
    at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stop
    at
  com/ibm/mq/connector/ResourceAdapterImpl.endpointDeactivation
       (entered lock: java/util/HashMap@0x0000000083E0D2C0, entry
  count: 1)
    at
  com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateUnderRAClass
  LoaderContext
    at
  com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateEndPoint
    at com/ibm/ejs/j2c/mbeans/MessageEndpointMBeanImpl.pause
       (entered lock:
  com/ibm/ejs/j2c/mbeans/MessageEndpointMBeanImpl@0x000000008B5454
  88, entry count: 1)
    at sun/reflect/NativeMethodAccessorImpl.invoke0
    ...
  
  
  "WebSphere WLM Dispatch Thread t=00ac20b8"
  J9VMThread:0x000000003B9BA300, j9thread_t:0x000000480916C4D0,
  java/lang/Thread:0x000000008E5BBAD0, state:CW, prio=5
         (java/lang/Thread getId:0x9E, isDaemon:true)
         (native thread ID:0x3A9C4400, native priority:0x5, native
  policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000101)
  Waiting on:
  com/ibm/mq/connector/inbound/ServerSessionPoolImpl$CloseState@0x
  000000009304F130 Owned by: <unowned>
  Java callstack:
    at java/lang/Object.wait
    at java/lang/Object.wait
    at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.close
       (entered lock:
  com/ibm/mq/connector/inbound/ServerSessionPoolImpl$CloseState@0x
  000000009304F130, entry count: 2)
    at
  com/ibm/mq/connector/inbound/MessageEndpointDeployment.stopDeliv
  eryASF
    at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stop
    at
  com/ibm/mq/connector/inbound/MessageEndpointDeployment.onExcepti
  on
       (entered lock:
  com/ibm/ejs/container/MessageEndpointFactoryImpl@0x000000008AF39
  900, entry count: 1)
    at com/ibm/mq/connector/inbound/AbstractWorkImpl.run
    at com/ibm/mq/connector/inbound/ServerSessionImpl.start
    at
  com/ibm/ws/wmqra/zdispatcher/WASServantConnectionConsumer.delive
  rMessagesFromReference
    at
  com/ibm/ws/wmqra/zdispatcher/WASServantConnectionConsumerReposit
  ory.dispatch
    at
  com/ibm/ws/wmqra/zdispatcher/WASServantDispatchTarget.dispatch
    at com/ibm/ws/sib/z/dispatcher/WorkObjectImpl.fireWork
    at
  com/ibm/ws390/sib/dispatcher/WorkObjectProxyImpl.dispatchWorkObj
  ectImpl
    ...
  

Local fix

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of the:
  
    - WebSphere MQ V7.1 resource adapter
    - WebSphere MQ V7.5 resource adapter
    - IBM MQ V8 resource adapter
    - IBM MQ V9 resource adapter
  
    - WebSphere Application Server V8.5 MQ messaging provider
    - WebSphere Application Server V9 MQ messaging provider
  
  who use activation specifications to deliver messages
  asynchronously to message-driven bean (MDB) applications.
  
  
  Platforms affected:
  MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  A deadlock within the MQ resource adapter occurred in the
  scenario when:
  
    1) an XA transaction error was thrown (for example, because of
  a broken TCP/IP connection between the application server and
  queue manager) while a ServerSession thread was attempting to
  deliver a message to a message-driven bean (MDB) application
  (see the thread named "WebSphere WLM Dispatch Thread t=00ac20b8"
  in the Problem Description),
  
  and
  
    2) another thread was already attempting to stop the message
  endpoint and close the ServerSessionPool associated with an
  activation specification (see the thread named "WebSphere
  non-WLM Dispatch Thread t=00abae88" in the Problem Description).
  
  
  This was because the thread stopping the message endpoint was
  attempting to close off ServerSessions in the activation
  specification's ServerSessionPool. While doing this, it found a
  ServerSession that was being used to try and deliver a message
  to an MDB, so it blocked waiting for that ServerSession to
  finish and be returned to the ServerSessionPool before
  continuing to close it.  However, once the thread that was
  running the ServerSession experienced a XA error during message
  delivery it also attempted to perform some clean-up, before
  starting reconnection retry processing.  This included closing
  the ServerSessionPool associated with the activation
  specification.
  
  The thread running the ServerSession determined that another
  thread was already processing a close of the ServerSessionPool
  and so blocked, waiting for that operation to complete.
  However, this thread could not complete the pool closure because
  the ServerSession that was in use had not been returned to the
  pool.  As such, neither threads could progress.
  

Problem conclusion

• The MQ resource adapter has been updated such that when a thread
  is closing an activation specification's ServerSessionPool it
  does not wait for ServerSessions that are processing a message
  delivery failure to be returned to the pool before they are
  closed.
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v7.1       7.1.0.9
  v7.5       7.5.0.9
  v8.0       8.0.0.8
  v9.0 LTS   9.0.0.2
  
  The latest available maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WMQ WINDOWS V7

• Fixed component ID

  5724H7220

Applicable component levels

• R710 PSY

     UP