APAR status
Closed as program error.
Error description
A MQ classes for Java application attempts to establish a new connection to the queue manager, by calling the constructor method: com.ibm.mq.MQQueueManager( String queueManagerName, Hashtable properties ); while other application threads are already connected to the queue manager. The constructor call never returns. Inspection of the JVM threads revealed that the JVM had detected that a deadlock has occurred, between the thread stacks: RcvThread : Deadlocked on Lock[java.util.concurrent.locks.ReentrantLock$NonfairSync@7febca c8] held by thread [Thread_B] sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndChe ckInterrupt java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQue ued java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire java.util.concurrent.locks.ReentrantLock$NonfairSync.lock java.util.concurrent.locks.ReentrantLock.lock com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.remove Connection com.ibm.mq.jmqi.remote.impl.RemoteConnection.removeSelf com.ibm.mq.jmqi.remote.impl.RemoteConnection.asyncConnectionBrok en com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.run Thread_B : Deadlocked on Lock[java.util.concurrent.locks.ReentrantLock$FairSync@70e46aef] held by thread [RcvThread] java.util.concurrent.locks.ReentrantLock.lock com.ibm.msg.client.commonservices.locking.TraceableLock.lock com.ibm.msg.client.commonservices.locking.TraceableReentrantLock .lock com.ibm.mq.jmqi.remote.impl.RemoteConnection.requestSendLock com.ibm.mq.jmqi.remote.impl.RemoteConnection.sendTSH com.ibm.mq.jmqi.remote.impl.RemoteConnection.sendTSH com.ibm.mq.jmqi.remote.impl.RemoteSession.startConversation com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.assign SessionFromConnection com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSes sionFromEligibleConnection com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSes sion com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.access$100 com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect ... ... ...
Local fix
Problem summary
**************************************************************** USERS AFFECTED: Users of the MQ classes for Java/JMS connecting to the queue manager in CLIENT transport with a TLS/SSL secured channel, where the SSL/TLS key reset option has been enabled, such as by making use of the queue manager property: SSLRKEYC Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: The sequence of events which led to this deadlock was as follows: The queue manager sent a request to the MQ classes for Java/JMS client API over an existing TCP/IP connection, asking for a SSL/TLS key reset operation to take place. The MQ classes for Java/JMS then paused traffic over the socket until this key reset operation completed, by taking a lock to prevent other threads from sending data over the socket. Another application thread is started to connect to the queue manager, where it attempted to connect over the same TCP/IP socket as the one which is doing the SSL/TLS key reset. It had taken a lock associated with the pool of conversations on the socket, while it attempted to connect. However it is blocked by the key reset operation, and pauses waiting for that to complete. The MQ classes for Java/JMS TCP/IP listening thread, called the "RcvThread", continued with the key reset operation, but encountered an exception with the socket - such as when the network had disconnected the socket unexpectedly. The RcvThread attempted to recover from this condition, which required the lock to the conversation pool held by the connecting thread, and so is blocked. This resulted in a deadlock. Inspection of the JVM threads would reveal two thread stacks of the following form: RcvThread : Deadlocked on Lock[java.util.concurrent.locks.ReentrantLock$NonfairSync@7febca c8] held by thread [Thread_B] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175 ) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndChe ckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQue ued(AbstractQueuedSynchronizer.java:870) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Ab stractQueuedSynchronizer.java:1199) java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(Reentr antLock.java:209) com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.remove Connection(RemoteConnectionSpecification.java:742) com.ibm.mq.jmqi.remote.impl.RemoteConnection.removeSelf(RemoteCo nnection.java:4007) com.ibm.mq.jmqi.remote.impl.RemoteConnection.asyncConnectionBrok en(RemoteConnection.java:3951) com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.run(RemoteRcvThread. java:644) Thread_B: Waiting for the lock that the RcvThread has: java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java :285) com.ibm.msg.client.commonservices.locking.TraceableLock.lock(Tra ceableLock.java:75) com.ibm.msg.client.commonservices.locking.TraceableReentrantLock .lock(TraceableReentrantLock.java:73) com.ibm.mq.jmqi.remote.impl.RemoteConnection.requestSendLock(Rem oteConnection.java:3632) com.ibm.mq.jmqi.remote.impl.RemoteConnection.sendTSH(RemoteConne ction.java:2632) com.ibm.mq.jmqi.remote.impl.RemoteConnection.sendTSH(RemoteConne ction.java:2599) com.ibm.mq.jmqi.remote.impl.RemoteSession.startConversation(Remo teSession.java:1174) com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.assign SessionFromConnection(RemoteConnectionSpecification.java:532) com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSes sionFromEligibleConnection(RemoteConnectionSpecification.java:43 7) com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSes sion(RemoteConnectionSpecification.java:176) com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(Remo teConnectionPool.java:127) com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(Remot eFAP.java:13399) com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.access$100(Remote FAP.java:13199) com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java: 1449) com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java: 1390) ... ... ... The JVM may also report other threads involved in the deadlock, but these are the primary two causing the issue.
Problem conclusion
The code has been updated to adjust the locking model to avoid the deadlock. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v9.x CD 9.3.5 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT44500
Reported component name
MQ BASE V9.3
Reported component ID
5724H7291
Reported release
931
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2023-09-11
Closed date
2023-10-20
Last modified date
2023-10-20
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
MQ BASE V9.3
Fixed component ID
5724H7291
Applicable component levels
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"931","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
21 October 2023