APAR status
Closed as program error.
Error description
When using an Activation Specification defined against the IBM MQ resource adapter deployed into a JEE application server, concurrent message delivery to a message-driven bean application declines until no further messages are delivered. If an attempt is subsequently made to stop the application or shutdown the application server, the shutdown processing hangs. A javacore from the application server when the hang occurs shows a thread, which is attempting to deactivate the endpoint associated with the application, stuck in a conditional wait state trying to close a ServerSession. For example, if the issue occurs while the IBM MQ resource adapter has been deployed into a WebSphere Libery server, then a Javacore taken when the application server is trying to shutdown would show the following thread in a conditional wait state: "Default Executor-thread-111" J9VMThread:0x00000000013E7700, omrthread_t:0x00007F0B10015938, java/lang/Thread:0x00000007F9A02340, state:CW, prio=5 Waiting on: com/ibm/mq/connector/inbound/ServerSessionImpl$IsInUseLock@0x000 000070FCB3BB0 Owned by: <unowned> Heap bytes allocated since last GC cycle=0 (0x0) Java callstack: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:189(Compiled Code)) at com/ibm/mq/connector/inbound/ServerSessionImpl.close(ServerSessi onImpl.java:303) at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.closeInternal (ServerSessionPoolImpl.java:777) at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.close(ServerS essionPoolImpl.java:711) at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stopDeliv eryASF(MessageEndpointDeployment.java:655) at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stop(Mess ageEndpointDeployment.java:538) at com/ibm/mq/connector/ResourceAdapterImpl.endpointDeactivation(Re sourceAdapterImpl.java:682) at com/ibm/ws/jca/service/EndpointActivationService.endpointDeactiv ation(EndpointActivationService.java:653) at com/ibm/ws/jca/service/JCAQuiesceListener.serverStopping(JCAQuie sceListener.java:69) at com/ibm/ws/runtime/update/internal/RuntimeUpdateManagerImpl$3.ru n(RuntimeUpdateManagerImpl.java:362) at com/ibm/ws/threading/internal/ExecutorServiceImpl$RunnableWrappe r.run(ExecutorServiceImpl.java:239(Compiled Code)) at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExec utor.java:1160(Compiled Code)) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExe cutor.java:635) at java/lang/Thread.run(Thread.java:812)
Local fix
Problem summary
**************************************************************** USERS AFFECTED: This issue affects message-driven bean applications that use the IBM MQ resource adapter in a JEE environment. Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: In IBM MQ V9.1 the IBM MQ resource adapter used a background task to retrieve a: javax.jms.ServerSession from a: javax.jms.ServerSessionPool This background task was executed in a separate thread from the "browsing thread", which was notified of an available message for the message-driven bean (MDB) application. If the background task took longer than 120 seconds to return with a ServerSession from the ServerSessionPool then the browsing thread would give up waiting, discard the message reference it currently held and return to the queue manager to request another message reference. Note that the queue manager may still return the same reference if the message to which it belongs is still available on the MDB's input queue. After the browsing thread gave up waiting, the background task that was attempting to retrieve a ServerSession was not cancelled. When it later completed, it would have obtained a ServerSession from the ServerSessionPool but there would have been no browsing thread to hand it to. The result, was that the ServerSession would never be released back to the ServerSessionPool for reuse. If this happen repeatedly, then the number of ServerSession available in the pool would be reduced over time. This would have meant that fewer MDB instances could run in parallel until no more messages could be delivered at all, which would be the case when all the ServerSessions had been removed and never returned to the ServerSessionPool. If this occurred when using a Activation Specification to deliver messages to an MDB, then the application server would hang on shutdown. When the messaging endpoint, created against the IBM MQ resource adapter, was requested to shutdown, an attempt was made to close the Activation Specification's ServerSessionPool. The shutdown thread would wait until all ServerSession objects had been released back to the ServerSessionPool, so as not to close a ServerSession while it was still in use by an MDB instance. Because the ServerSessions would never be returned to the ServerSessionPool, the shutdown thread would wait indefinitely.
Problem conclusion
The IBM MQ resource adapter has been updated such that the browsing thread, which handles references to messages available for delivery to message-driven bean applications, requests a javax.jms.ServerSession from a javax.jms.ServerSessionPool directly without using a background task and separate thread. This is the behaviour that was present in the IBM MQ resource adapter prior to version 9.0.1.0. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v9.1 CD 9.1.4 v9.1 LTS 9.1.0.4 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT30234
Reported component name
IBM MQ BASE MP
Reported component ID
5724H7271
Reported release
910
Status
CLOSED PER
PE
NoPE
HIPER
YesHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2019-09-10
Closed date
2019-09-30
Last modified date
2020-04-14
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
IBM MQ BASE MP
Fixed component ID
5724H7271
Applicable component levels
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
15 April 2020