IT43183: IBM MQ resource adapter leaks LocalServer objects if an MDB's ejbCreate() method throws an exception

APAR status

• Closed as program error.

Error description

• A message-driven bean (MDB) application has been deployed into
  WebSphere Application Server 9.0.x (which embeds IBM MQ 9.x
  resource adapter) and has been configured to use a Listener Port
  to monitor a queue on a remote IBM MQ queue manager for
  messages. The MDB has its own implementation of the ejbCreate()
  method, and that implementation contains a bug which results in
  it throwing an exception whenever it is called. This causes the
  application server to write messages similar to the example
  shown below to its SystemOut.log file every time the Listener
  Port detects a message and tries to deliver it to the MDB:
  
  CNTR0019E: EJB threw an unexpected (non-declared) exception
  during invocation of method "onMessage". Exception data:
  com.ibm.ejs.container.CreateFailureException: ; nested exception
  is:
  	java.lang.NullPointerException
  	at
  com.ibm.ejs.container.MessageDrivenBeanO.initialize(MessageDrive
  nBeanO.java:203)
  	at
  com.ibm.ejs.container.BeanOFactory.create(BeanOFactory.java:105)
  	at
  com.ibm.ejs.container.EJSHome.createBeanO(EJSHome.java:976)
  	at
  com.ibm.ejs.container.EJSHome.createBeanO(EJSHome.java:1078)
  	at
  com.ibm.ejs.container.activator.UncachedActivationStrategy.atAct
  ivate(UncachedActivationStrategy.java:63)
  	at
  com.ibm.ejs.container.activator.Activator.preInvokeActivateBean(
  Activator.java:264)
  	at
  com.ibm.ejs.container.EJSContainer.preInvokeActivate(EJSContaine
  r.java:3242)
  	at
  com.ibm.ejs.container.EJSContainer.EjbPreInvoke(EJSContainer.jav
  a:2579)
  	at
  com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:89)
  	at
  com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:132)
  	at
  com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:60
  1)
  	at
  com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)
  Caused by: java.lang.NullPointerException
  	at project.TestMDB.ejbCreate(TestMDB.java:29)
  	at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown
  Source)
  	at
  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod
  AccessorImpl.java:55)
  	at java.lang.reflect.Method.invoke(Method.java:508)
  	at
  com.ibm.ejs.container.MessageDrivenBeanO.initialize(MessageDrive
  nBeanO.java:181)
  	... 11 more
  
  
  After the application server has been running and reporting
  these errors for a while, it stops unexpectedly due to a
  java.lang.OutOfMemoryError. A heap dump generated at the time of
  the issue shows that the majority of the Java heap is taken up
  with an object of type:
  
  com.ibm.msg.client.commonservices.trace.Trace
  
  which contains an ArrayList consisting of a very large number
  of:
  
  com.ibm.mq.jmqi.local.LocalServer
  
  objects.
  

Local fix

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of:
  
  - The IBM MQ 9.1 resource adapter.
  - The IBM MQ 9.2 resource adapter.
  - The IBM MQ 9.3 resource adapter.
  - The WebSphere Application Server 9.0 IBM MQ messaging provider
  
  who have message-driven bean (MDB) applications that connect to
  a queue manager using the CLIENT transport.
  
  
  Platforms affected:
  MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  The IBM MQ resource adapter maintains an internal cache of
  objects that represent different MQ Interface (MQI) instances.
  There is typically one object for each different transport type.
  For example, the cache normally contains one instance of a
  LocalServer object, which can be used by all of the threads
  within the resource adapter that connect to a queue manager
  using the BINDINGS transport.
  
  These objects are created and added to the MQI cache the first
  time they are needed. They then remain in the cache until the
  application server that the resource adapter is running in is
  stopped.
  
  In addition to this, the IBM MQ resource adapter's trace
  component maintains an ArrayList containing references to these
  objects so that details of them can be included in diagnostic
  output such as FDCs.
  
  
  Now, when the issue reported in this APAR occurred:
  
  - The IBM MQ resource adapter was deployed into an application
  server
  - And a message-driven bean (MDB) application has been installed
  and configured to monitor a JMS destination hosted on a remote
  queue manager for messages (this meant that the MDB was
  connecting to the queue manager using the CLIENT transport).
  - And the MDB provided an implementation of the ejbCreate()
  method which threw an exception.
  
  When the IBM MQ resource adapter detected a message and tried to
  deliver it to the MDB, the following sequence of events took
  place:
  
  - The resource adapter obtained a Server Session from the
  associated Server Session Pool, loaded it up with details of the
  message that had been detected and started it running on a new
  thread.
  - The Server Session ran and called the MDB's ejbCreate() method
  to create a new instance of the bean.
  - This failed, and so the Server Session returned itself to the
  Server Session Pool.
  
  - Next, the resource adapter detected another message and tried
  to deliver this to the MDB.
  - It got the same Server Session out of the Server Session Pool
  and tried to load details of the new message into it.
  - However, the Server Session still contained references for the
  previous message. This condition was unexpected, so the resource
  adapter decided to generate an FDC.
  - As part of this processing, it needed to check if the IBM MQ
  Java native libraries were installed on the system so that their
  version information could be included.
  - Because the native libraries are associated with the BINDINGS
  transport, the resource adapter queried the internal MQI cache
  to see if there was an instance of the LocalServer class it
  could use to get the information.
  - The cache didn't contain an instance of this object, so the
  resource adapter created a new one.
  - Next, the resource adapter added information about the
  LocalServer to the ArrayList associated with its trace
  component.
  - It then started to initialize the object, which required it to
  load the native libraries - this failed because IBM MQ was not
  installed on the same system as the application server.
  - As this was expected, the LocalServer object was not added to
  the MQI cache. However, it was still left in the trace
  component's ArrayList.
  - The MQ resource adapter then generated the FDC and continued
  processing.
  - The Server Session ran and called the MDB's ejbCreate()
  method.
  - The method failed, and the Server Session returned itself to
  the ServerSession Pool.
  
  These steps occurred whenever the resource adapter tried to
  deliver a message to the MDB. This meant that, over time, the
  size of the ArrayList grew until eventually the application
  server stopped due to a java.lang.OutOfMemoryError.
  

Problem conclusion

• To resolve this issue, the IBM MQ resource adapter has been
  updated so that LocalServer objects are only added to the trace
  component's ArrayList if they have been successfully
  initialized. This ensures that the ArrayList only contains
  LocalServer objects that are stored in the MQI cache, rather
  than objects that could not be initialized.
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v9.1 LTS   9.1.0.15
  v9.2 LTS   9.2.0.15
  v9.3 LTS   9.3.0.5
  v9.x CD    9.3.3
  
  The latest available maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  IBM MQ BASE MP

• Fixed component ID

  5724H7271

Applicable component levels