APAR status
Closed as program error.
Error description
An MQ classes for JMS application was upgraded from MQ v6.0 to MQ v9.3.1.0 CD. After the upgrade, the application threads were seen to be deadlocked within the MQ classes for JMS API. Restarting the application JVM resumed MQ functionality, but after some time the deadlock was observed again. A Java thread dump taken when the JVM was in this state revealed two threads of the form: "RcvThread: com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection@985027307[qmid=m yQMGR....]" java.lang.Thread.State: BLOCKED at com.ibm.mq.jmqi.remote.impl.RemoteConnection.getSessionByConvId - waiting to lock <0x000000003f1aab0b> (a com.ibm.mq.jmqi.remote.impl.RemoteConnection$SessionsMutex) owned by "myApplicationThradName" at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.run at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTas k at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem. runItem at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.run WorkQueueItem at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManage rImplementation$ThreadPoolWorker.run "myApplicationThreadName" #187 prio=5 os_prio=0 tid=0x00000000000000bb nid=0x00bb waiting on condition [0x000000000000004c] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000000e3c5344> (a java.util.concurrent.locks.ReentrantLock$FairSync) owned by "RcvThread: com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection@985027307[qmid=m yQMGR....]" at java.util.concurrent.locks.LockSupport.park at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndChe ckInterrupt at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQue ued at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire at java.util.concurrent.locks.ReentrantLock$FairSync.lock at java.util.concurrent.locks.ReentrantLock.lock at com.ibm.msg.client.commonservices.locking.TraceableLock.lock at com.ibm.msg.client.commonservices.locking.TraceableReentrantLock .lock at com.ibm.mq.jmqi.remote.impl.RemoteConnection.requestSendLock at com.ibm.mq.jmqi.remote.impl.RemoteConnection.sendTSH at com.ibm.mq.jmqi.remote.impl.RemoteConnection.sendTSH at com.ibm.mq.jmqi.remote.impl.RemoteConnection.confirmSessionEndTo QMgr
Local fix
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of the MQ classes for Java/JMS who are connected to the queue manager using a CLIENT transport (TCP/IP) connection, over a SSL/TLS socket, with the queue manager configured to re-negotiate the SSL/TLS secret key after an amount of data has been transmitted over the socket. This is configured on the queue manager with the queue manager attribute: SSLRKEYC Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: The sequence of events which leads to this deadlock are as follows: (1) The queue manager sends a request to the MQ classes for Java/JMS client API asking for a SSL/TLS key reset to take place. The MQ classes for Java/JMS TCP/IP listening thread,"RcvThread", processes this request, and takes a lock to prevent any other threads from sending data over the socket while the key reset is in progress. (2) In parallel, an application thread is closing a JMS Session or Connection, or MQ classes for Java MQQueueManager object, resulting in an MQDISC flow being sent to the queue manager, which then responded to the request and replied to the client API. This application thread then takes a lock to allow it to update the conversation sharing status of the socket, and wants to send some data back to the queue manager to indicate that the current conversation is no longer going to be able to receive any further data over the socket. It cannot send this because the "RcvThread" has the TCP/IP socket send lock, and so is blocked from further action. (3) Shortly afterwards, the RcvThread receives another packet of data from the queue manager. In order to know which application thread this packet should be routed to, it requires access to the conversation sharing status, which is locked by the other application thread in step (2) above, and so this thread is then also blocked. This sequence of events lead to the deadlock situation, and a JVM restart was required to resume connectivity with the queue manager. The two threads deadlocked in this way would have a thread stacks of the form (example taken from the MQ classes for JMS): "RcvThread: com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection@985027307[qmid=m yQMGR....]" #76 prio=5 os_prio=0 tid=0x000000000000004c nid=0x004c waiting on condition [0x00000000000000bb] java.lang.Thread.State: BLOCKED at com.ibm.mq.jmqi.remote.impl.RemoteConnection.getSessionByConvId( RemoteConnection.java:4459) - waiting to lock <0x000000003f1aab0b> (a com.ibm.mq.jmqi.remote.impl.RemoteConnection$SessionsMutex) owned by "myApplicationThradName" at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.run(RemoteRcvThread. java:182) at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTas k(WorkQueueItem.java:319) at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem. runItem(SimpleWorkQueueItem.java:99) at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(Wo rkQueueItem.java:343) at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.run WorkQueueItem(WorkQueueManager.java:312) at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManage rImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementat ion.java:1240) "myApplicationThreadName" #187 prio=5 os_prio=0 tid=0x00000000000000bb nid=0x00bb waiting on condition [0x000000000000004c] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000000e3c5344> (a java.util.concurrent.locks.ReentrantLock$FairSync) owned by "RcvThread: com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection@985027307[qmid=m yQMGR....]" at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175 ) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndChe ckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQue ued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Ab stractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$FairSync.lock(Reentrant Lock.java:224) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java :285) at com.ibm.msg.client.commonservices.locking.TraceableLock.lock(Tra ceableLock.java:75) at com.ibm.msg.client.commonservices.locking.TraceableReentrantLock .lock(TraceableReentrantLock.java:73) at com.ibm.mq.jmqi.remote.impl.RemoteConnection.requestSendLock(Rem oteConnection.java:3632) at com.ibm.mq.jmqi.remote.impl.RemoteConnection.sendTSH(RemoteConne ction.java:2632) at com.ibm.mq.jmqi.remote.impl.RemoteConnection.sendTSH(RemoteConne ction.java:2599) at com.ibm.mq.jmqi.remote.impl.RemoteConnection.confirmSessionEndTo QMgr(RemoteConnection.java:968) at com.ibm.mq.jmqi.remote.impl.RemoteConnection.removeSession(Remot eConnection.java:911) - locked <0x000000003f1aab0b> (a com.ibm.mq.jmqi.remote.impl.RemoteConnection$SessionsMutex) at com.ibm.mq.jmqi.remote.impl.RemoteSession.disconnect(RemoteSessi on.java:1745) - locked <0x00000000488c40ae> (a com.ibm.mq.jmqi.remote.impl.RemoteSession$RequestMessagesMutex) at com.ibm.mq.jmqi.remote.impl.RemoteConnection.completeClose(Remot eConnection.java:4741) at com.ibm.mq.jmqi.remote.api.RemoteFAP.MQDISC(RemoteFAP.java:1726) at com.ibm.mq.jmqi.remote.api.RemoteFAP.MQDISC(RemoteFAP.java:1611) at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.MQDISC(InterceptedJmqiIm pl.java:410) at com.ibm.mq.ese.jmqi.ESEJMQI.MQDISC(ESEJMQI.java:396) at com.ibm.msg.client.wmq.internal.WMQSession.disconnect(WMQSession .java:817) at com.ibm.msg.client.wmq.internal.WMQSession.close(WMQSession.java :783) at com.ibm.msg.client.jms.internal.JmsSessionImpl.close(JmsSessionI mpl.java:632) at com.ibm.msg.client.jms.internal.JmsSessionImpl.close(JmsSessionI mpl.java:355) at com.ibm.mq.jms.MQSession.close(MQSession.java:280) at myApplication
Problem conclusion
The locking model used by the MQ classes for Java/JMS when communicating with the queue manager over TCP/IP, has been adjusted to prevent the deadlock from occurring. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v9.x CD 9.3.3 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT43702
Reported component name
MQ BASE V9.3
Reported component ID
5724H7291
Reported release
931
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2023-05-05
Closed date
2023-06-21
Last modified date
2023-06-21
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
MQ BASE V9.3
Fixed component ID
5724H7291
Applicable component levels
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"931","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
22 June 2023