APAR status
Closed as program error.
Error description
A hang can occur when trying to stop an application containing an MDB bound to an MQ Activation Specification. The deactivation of the endpoint is blocked in the throttle in the Control Region Adjunct process. A javacore of the control region adjunct process shows a callstack of the form below: 3XMTHREADINFO "WebSphere non-WLM Dispatch Thread t=009ac248" J9VMThread:0x0000000039B3F100, omrthread_t:0x000000500A328560, java/lang/Thread:0x0000000086F90998, state:CW, prio=5 3XMJAVALTHREAD (java/lang/Thread getId:0x87, isDaemon:true) 3XMTHREADINFO1 (native thread ID:0x37EF6000, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000101) 3XMCPUTIME CPU usage total: 21.335000000 secs, current category="Application" 3XMTHREADBLOCK Waiting on: com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x0000000083 BDF228 Owned by: <unowned> 3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0) 3XMTHREADINFO3 Java callstack: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:189(Compiled Code)) at com/ibm/mq/jmqi/remote/util/ReentrantMutex.acquire(ReentrantMute x.java:167) (entered lock: com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x0000000083 BDF228, entry count: 2) at com/ibm/mq/jmqi/remote/util/ReentrantMutex.acquire(ReentrantMute x.java:73) (entered lock: com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x0000000083 BDF228, entry count: 1) at com/ibm/mq/jmqi/remote/api/RemoteHconn.requestDispatchLock(Remot eHconn.java:1198) at com/ibm/mq/jmqi/remote/api/RemoteFAP.MQCTL(RemoteFAP.java:2548) at com/ibm/mq/jmqi/monitoring/JmqiInterceptAdapter.MQCTL(JmqiInterc eptAdapter.java:333) at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.suspendAs yncService(WMQConsumerOwnerShadow.java:523) at com/ibm/msg/client/wmq/internal/WMQConnection.suspendAsyncServic e(WMQConnection.java:2308) at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close(WMQConsu merShadow.java:366) at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close(WMQConsu merShadow.java:319) at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close(WMQConsu merShadow.java:308) at com/ibm/msg/client/wmq/internal/WMQConnectionBrowser$WMQConnecti onBrowserShadow.close(WMQConnectionBrowser.java:488) at com/ibm/msg/client/wmq/internal/WMQConnectionBrowser.close(WMQCo nnectionBrowser.java:1518) at com/ibm/msg/client/jms/internal/JmsConnectionBrowserImpl.close(J msConnectionBrowserImpl.java:250) at com/ibm/msg/client/jms/internal/JmsConnectionBrowserImpl.close(J msConnectionBrowserImpl.java:204) at com/ibm/mq/jms/MQConnectionBrowser.close(MQConnectionBrowser.jav a:80) at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stopDeliv eryASF(MessageEndpointDeployment.java:630) at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stop(Mess ageEndpointDeployment.java:537) at com/ibm/mq/connector/ResourceAdapterImpl.endpointDeactivation(Re sourceAdapterImpl.java:680) (entered lock: java/util/HashMap@0x0000000081F4F3B0, entry count: 1) at com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateUnderRAClass LoaderContext(ActivationSpecWrapperImpl.java:513) at com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateEndPoint(Act ivationSpecWrapperImpl.java:433) at com/ibm/ejs/j2c/RAWrapperImpl.deactivateEndpoint(RAWrapperImpl.j ava:1486) (entered lock: com/ibm/ejs/j2c/RAWrapperImpl@0x0000000082206DE8, entry count: 2) at com/ibm/ejs/j2c/RALifeCycleManagerImpl.deactivateEndPoint(RALife CycleManagerImpl.java:1918) (entered lock: com/ibm/ejs/j2c/RAWrapperImpl@0x0000000082206DE8, entry count: 1) at com/ibm/ejs/container/MessageEndpointFactoryImpl.deactivateEndpo int(MessageEndpointFactoryImpl.java:237) ... The callstacks of one or more other threads show the thread stuck in the throttle, for example: 3XMTHREADINFO "DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[:/f76f99c1][connectio nId=C3E2D8C3D4D8E9C54040404040404040D7563E14B92C0001]]" J9VMThread:0x00000000346AF000, omrthread_t:0x000000500A1B7B40, java/lang/Thread:0x0000000087E1B968, state:CW, prio=5 3XMJAVALTHREAD (java/lang/Thread getId:0x5E, isDaemon:true) 3XMTHREADINFO1 (native thread ID:0x327A1800, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000101) 3XMCPUTIME CPU usage total: 0.026000000 secs, current category="Application" 3XMTHREADBLOCK Waiting on: com/ibm/ws/wmqra/zdispatcher/EndpointInflightMessageReferences@0 x0000000083BDE6E0 Owned by: <unowned> 3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0) 3XMTHREADINFO3 Java callstack: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:189(Compiled Code)) at com/ibm/ws/wmqra/zdispatcher/EndpointInflightMessageReferences.p reDispatch(EndpointInflightMessageReferences.java:297) (entered lock: com/ibm/ws/wmqra/zdispatcher/EndpointInflightMessageReferences@0 x0000000083BDE6E0, entry count: 1) at com/ibm/ws/wmqra/zdispatcher/InflightMessageReferenceManager.pre Dispatch(InflightMessageReferenceManager.java:228) at com/ibm/ws/wmqra/zdispatcher/WASAdjunctMessageReferenceHandler.h andleMessageReference(WASAdjunctMessageReferenceHandler.java:185 ) (entered lock: com/ibm/ws/wmqra/zdispatcher/WASAdjunctMessageReferenceHandler@0 x0000000083BDE738, entry count: 1) at com/ibm/mq/jms/MQMessageReferenceHandler.handleMessageReference( MQMessageReferenceHandler.java:82) at com/ibm/msg/client/jms/internal/JmsConnectionImpl$JmsProviderMes sageRefHandler.handleMessageReference(JmsConnectionImpl.java:195 8) at com/ibm/msg/client/wmq/internal/WMQConnectionBrowser$WMQConnecti onBrowserShadow.consumer(WMQConnectionBrowser.java:947) at com/ibm/mq/jmqi/remote/impl/RemoteProxyQueue.driveConsumer(Remot eProxyQueue.java:4367) at com/ibm/mq/jmqi/remote/impl/RemoteProxyQueue.deliverMsgs(RemoteP roxyQueue.java:4961) at com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.deliverMsgsReco nnectable(RemoteDispatchThread.java:532) at com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.deliverMsgs(Rem oteDispatchThread.java:505) at com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.run(RemoteDispa tchThread.java:275) at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.runTas k(WorkQueueItem.java:314) at com/ibm/msg/client/commonservices/workqueue/SimpleWorkQueueItem. runItem(SimpleWorkQueueItem.java:99) at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.run(Wo rkQueueItem.java:336) at com/ibm/ws/wmqcsi/workqueue/WorkQueueManagerImpl$WorkQueueRunnab le.run(WorkQueueManagerImpl.java:550) at java/lang/Thread.run(Thread.java:818) The problem was not reproducable when the alternate consumer counting mechanism for the throttle, introduced in APAR PI61894, was in use (specified by setting the com.ibm.ws.wmqra.ThrottleOnActiveConsumerCount system property to true).
Local fix
WebSphere Variable com.ibm.ws.wmqra.ThrottleOnActiveConsumerCount set to a value of 'true', as described in PI61894 could avoid this.
Problem summary
**************************************************************** * USERS AFFECTED: WebSphere Application Server users of * * WebSphere MQ Activation Specifications * **************************************************************** * PROBLEM DESCRIPTION: The process to stop an application * * may hang if it contains MDBs bound to * * the runtime using WMQ Activation * * Specifications. * **************************************************************** * RECOMMENDATION: * **************************************************************** Applications containing MDBs bound to the runtime via WMQ Activation Specifications listen in the control region adjunct process which contains a throttle. When such an application is stopped the WMQ resource adapter will wait for its existing processing to stop, this includes threads that are calling the MessageRefenceHandler in the control region adjunct that listens and throttles the inbound work. A timing related bug meant that the threads stuck in the throttle could remain permanently stuck when stopping the listener, in turn hanging the stop of the listener itself.
Problem conclusion
A mechanism was added to the throttle to track the activation state of the Activation Specification Endpoint so ensure that the throttle was released correctly during endpoint deactivation. The fix for this APAR is targeted for inclusion in fix pack 9.0.5.4. For more information, see 'Recommended Updates for WebSphere Application Server': https://www.ibm.com/support/pages/node/715553
Temporary fix
The problem was not reproducable when the alternate consumer counting mechanism for the throttle, introduced in APAR PI61894, was in use (specified by setting the com.ibm.ws.wmqra.ThrottleOnActiveConsumerCount system property to true).
Comments
APAR Information
APAR number
PH21305
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
900
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2020-01-22
Closed date
2020-06-15
Last modified date
2020-06-15
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500
Applicable component levels
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"900","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
16 June 2020