APAR status
Closed as program error.
Error description
After upgrading to a version of WebSphere MQ or IBM MQ that contains the fix for APAR IV57472: http://www-01.ibm.com/support/docview.wss?uid=swg1IV57472 a deadlock can occur when using the MQ JMS Automatic Client Reconnect function when closing a JMS Session. A Javacore (thread dump) taken of the Java Runtime Environment when this occurs shows two hung threads with the following Java call stacks: Thread 1234: (state = BLOCKED) - java.lang.Object.wait(long) - java.lang.Object.wait() - com.ibm.mq.jmqi.remote.api.RemoteHconn.checkUsable() line=2148 - com.ibm.mq.jmqi.remote.api.RemoteHconn.leaveNotifyCall(int) line=915 - com.ibm.mq.jmqi.remote.api.RemoteFAP.MQCLOSE(com.ibm.mq.jmqi.han dles.Hconn, com.ibm.mq.jmqi.handles.Phobj, int, com.ibm.mq.jmqi.handles.Pint, com.ibm.mq.jmqi.handles.Pint) line=7354 - com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.MQCLOSE(com.ibm.mq.jmqi. handles.Hconn, com.ibm.mq.jmqi.handles.Phobj, int, com.ibm.mq.jmqi.handles.Pint, com.ibm.mq.jmqi.handles.Pint) line=433 - com.ibm.mq.ese.jmqi.ESEJMQI.MQCLOSE(com.ibm.mq.jmqi.handles.Hcon n, com.ibm.mq.jmqi.handles.Phobj, int, com.ibm.mq.jmqi.handles.Pint, com.ibm.mq.jmqi.handles.Pint) line=189 - com.ibm.msg.client.wmq.internal.WMQSyncConsumerShadow.closeInter nal(boolean) line=418 - com.ibm.msg.client.wmq.internal.WMQConsumerShadow.closeInternal( java.util.concurrent.locks.ReentrantLock, boolean) line=266 - com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close(java.uti l.concurrent.locks.ReentrantLock) line=365 - com.ibm.msg.client.wmq.internal.WMQMessageConsumer.close(boolean , java.util.concurrent.locks.ReentrantLock) line=378 - com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.close(boo lean) line=266 - com.ibm.msg.client.jms.internal.JmsSessionImpl.close(boolean) line=390 - com.ibm.msg.client.jms.internal.JmsSessionImpl.close() line=303 Thread 9876: (state = BLOCKED) - java.lang.Object.wait(long) - java.lang.Object.wait() - com.ibm.mq.jmqi.remote.util.ReentrantMutex.acquire(int, int) line=137 - com.ibm.mq.jmqi.remote.util.ReentrantMutex.acquire(int) line=74 - com.ibm.mq.jmqi.remote.api.RemoteHconn.enterCall(com.ibm.mq.jmqi .remote.util.ReentrantMutex, boolean, boolean) line=1854 - com.ibm.mq.jmqi.remote.api.RemoteHconn.enterNotifyCall(boolean) line=1836 - com.ibm.mq.jmqi.remote.impl.RemoteReconnectThread.reconnect(com. ibm.mq.jmqi.remote.api.RemoteHconn) line=241 - com.ibm.mq.jmqi.remote.impl.RemoteReconnectThread.run() line=133 - com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTas k() line=214
Local fix
Disable the use of the MQ JMS Automatic Client Reconnect function and have the classes for JMS application implement reconnection logic.
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of the: - WebSphere MQ v7.1 - WebSphere MQ v7.5 - IBM MQ v8.0.0.1 (or later) classes for JMS that make use of the MQ JMS Automatic Client Reconnect function and have the fix for APAR IV57472. This includes: - WebSphere MQ v7.1.0.6 and v7.1.0.7 - WebSphere MQ v7.5.0.5 and v7.5.0.6 - IBM MQ v8.0.0.1 and later Fix Packs Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: APAR IV57472: http://www-01.ibm.com/support/docview.wss?uid=swg1IV57472 addressed a race condition between two threads, one closing a JMS Session and the other closing a JMS MessageConsumer, that could result in the queue manager logging a AMQ9504 error message. Further details are contained within closing words for APAR IV57472. The fix for this issue was to ensure a second lock was held, internally known as the "Notify Lock", when a thread was either sending an MQCLOSE or MQDISC flow to the queue manager, as well as holding the "Call Lock" which synchronises API calls for a connection handle. However, this fix introduced a regression into the MQ classes for JMS Automatic Client Reconnect logic that could have resulted in a deadlock between an application thread closing a JMS Session (and therefore making an MQCLOSE API call) and the Automatic Client Reconnection thread. This could have occurred in the case where the connection, on which the MQCLOSE API call was to be made, was found to be broken and client reconnect processing was invoked. In this scenario, the thread performing the MQCLOSE API call waited for the reconnection processing to complete but continued to hold the Notify Lock. In order for the reconnection thread to reestablish broken connections and restore all connection handles on that connection, it requires the "Notify Lock". As such, the Automatic Client Reconnection thread waited for the Notify Lock before attempting to reconnect any broken connection handles while the thread performing the MQCLOSE API call was waiting for the reconnection to complete whilst holding the Notify Lock.
Problem conclusion
The MQ classes for JMS Automatic Client Reconnect code has been updated to ensure threads that are performing MQCLOSE or MQDISC API calls release the Notify Lock (which can be held after APAR IV57472) while Automatic Client Reconnect processing is active. Once connection handles have been reestablished by the Automatic Client Reconnection thread, the application thread performing the MQCLOSE or MQDISC API calls reacquires any locks (including the Notify Lock) that are required to complete the API call. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v7.1 7.1.0.8 v7.5 7.5.0.8 v8.0 8.0.0.6 v9.0 CD 9.0.1 v9.0 LTS 9.0.0.1 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IV83747
Reported component name
WMQ LIN X86 V7
Reported component ID
5724H7224
Reported release
710
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2016-04-14
Closed date
2016-06-20
Last modified date
2017-06-01
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WMQ LIN X86 V7
Fixed component ID
5724H7224
Applicable component levels
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1"}]
Document Information
Modified date:
08 March 2021