IT42294: Broken JMS connections remain in JEE connection pools if an enterprise application registers its own ExceptionListener

APAR status

• Closed as program error.

Error description

• The IBM MQ resource adapter has been deployed inside a JEE
  application server, and an MQ JMS Connection Factory has been
  defined that uses a client channel definition table (CCDT) which
  contains entries to connect to two different queue managers:
  
  - QM1
  - QM2
  
  in a queue manager group. The connection pool associated with
  the Connection Factory has a maximum size of 1.
  
  An enterprise application is installed into the application
  server and performs the following steps:
  
  - Try
    - Look up the MQ Connection Factory from the application
  server's JNDI repository.
    - Create a JMS Connection from the JMS Connection Factory.
    - Set an Exception Listener on the JMS Connection.
    - Create a non-transacted JMS Session from the JMS Connection.
    - Create a Message Producer for a Queue Destination from the
  JMS Session.
    - Create a Text Message from the JMS Session.
    - Use the Message Producer to send the Text Message to the
  Queue Destination.
  - Catch Exception:
    - Write details of the exception to the standard output stream
  (System.out).
  - Finally:
    - If a JMS Connection was created, then:
      - Remove the Exception Listener from the JMS Connection, by
  setting it to null.
      - Close the JMS Connection.
    - End if
  - End try
  
  The first time the application runs, the MQ resource adapter
  uses the information in the CCDT to create a JMS Connection to
  QM1, which is then returned to the application for use. When the
  application has finished processing, this JMS Connection is
  stored in the Connection Factory's connection pool, where it is
  available for reuse.
  
  During a subsequent run of the application, QM1 is stopped. When
  this happens, the application's Exception Listener is invoked
  with a JMSException containing MQ reason code 2009
  (MQRC_CONNECTION_BROKEN). The application then closes the JMS
  Connection.
  
  From this point onwards, every time the application is run its
  ExceptionListener is driven with a JMSException containing
  reason code 2009 (MQRC_CONNECTION_BROKEN) whenever it tries to
  create a JMS Session. The information in the error message
  indicates that the MQ resource adapter is trying to communicate
  with QM1, which is unexpected. Because the Connection Factory
  has been configured with a CCDT that contains entries for two
  queue managers in a queue manager group, the resource adapter
  should be communicating with QM2 (which is up) rather than QM1
  (which is down).
  

Local fix

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of:
  
  - The IBM MQ 9.3 JMS 2.0 resource adapter.
  - The IBM MQ 9.3 JMS 3.0 resource adapter for Jakarta Messaging.
  
  who have enterprise applications that:
  
  - Use a Connection Factory that has been configured to use
  either a Connection Name List or Client Channel Definition Table
  (CCDT) to connect to queue managers.
  - And call the method
  Connection.setExceptionListener(ExceptionListener) to register
  an ExceptionListener on JMS Managed Connections obtained from
  the Connection Factory's connection pool.
  - And:
      - Either run outside of the EJB Container.
      - Or run inside the EJB Container, and have set the Java
  system property:
  
  
  -Dcom.ibm.mq.connector.performJavaEEContainerChecks=false
  
         on the application server hosting the application.
  
  
  Platforms affected:
  MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  When performing outbound messaging within a JEE application
  server, enterprise applications can make use of connection
  pooling. The application server can be configured to provides a
  pool of JMS Managed Connections, one for each Connection Factory
  that has been defined. Whenever an application uses the
  Connection Factory to create a connection, the application
  server's Connection Manager will check the appropriate
  connection pool and perform one of the following actions:
  
  - If there is a free Managed Connection in the pool, the
  Connection Manager will mark it as being "in use", create a
  connection handle for it and pass that handle back to the
  application to use.
  
  - If all of the Managed Connections in the pool are currently
  "in use", and the number of Managed Connections is less than the
  maximum size of the pool, then the Connection Manager will:
      - Create a new Managed Connection.
      - Store it in the connection pool.
      - Mark it as being "in use".
      - Create a connection handle for it.
      - And pass the connection handle back to the application.
  
  - If all of the Managed Connections in the pool are "in use",
  and the number of Managed Connections is equal to the maximum
  size of the pool (ie. the pool is full), then the Connection
  Manager will block waiting for "in-use" connection to become
  available.
  
  After an application has finished with a connection and called
  Connection.close(), the Connection Manager will:
  
  - Close the connection handle that the application was using.
  - Check if the Managed Connection has been marked as "Stale".
  - If it has, then:
      - Destroy the Managed Connection, and remove it from the
  connection pool.
  - Else:
    - Change the state of the Managed Connection from "in-use" to
  "free". This allows it to be resued later.
  - End if
  
  
  The IBM MQ JMS 2.0 resource adapter and JMS 3.0 resource adapter
  for Jakarta Messaging provide their own implementations of JMS
  Managed Connections and connection handles:
  
  "com.ibm.mq.connector.outbound.ManagedConnectionImpl":
  These objects represent Managed Connections. These objects
  contain an actual JMS Connection, along with some state
  information such as whether the Managed Connection is involved
  in a transaction. The Managed Connection also registers itself
  as an ExceptionListener on the underlying JMS Connection.
  
  "com.ibm.mq.connector.outbound.ConnectionWrapper":
  These objects are the connection handles. These contain
  references to the Managed Connection, and the underlying JMS
  Connection stored in the Managed Connection.
  
  
  Here is a diagram that shows this relationship:
  
  ConnectionWrapper:
      |->contains ManagedConnectionImpl
      |      |-> contains JMS Connection
      |              |-> ExceptionListener: ManagedConnectionImpl
      |
      |-> contains JMS Connection
      |      |-> ExceptionListener: ManagedConnectionImpl
  
  
  If an connection error occurs on the JMS Connection the MQ
  resource adapter drives that Connection's  ExceptionListener,
  which is the ManagedConnectionImpl. The
  ManagedConnectionImpl.onException() method then fires a
  CONNECTION_ERROR_OCCURRED event to the Connection Manager,
  informing it can no longer be used. When the Connection Manager
  receives this event, it marks the connection as "Stale".
  
  As mentioned above, this means that when the application closes
  its connection handle by calling Connection.close(), the
  Connection Manager takes the necessary steps to clean up and
  destroy the Managed Connection, rather than returning it to the
  connection pool - this ensures that the broken connection cannot
  be reused. It will also potentially remove all of the other free
  connections in the pool, depending on the Purge Policy that has
  been defined on the connection pool.
  
  
  Now, when the issue reported in this APAR occurred, an
  application had:
  
  - Called ConnectionFactory.createConnection() to get a
  ConnectionWrapper representing a handle to a Managed Connection
  from a Connection Factory's connection pool.
  - And had then called
  Connection.setExceptionListener(ExceptionListener) to register
  its own Exception Listener.
  
  This resulted in the
  ConnectionWrapper.setExceptionListener(ExceptionListener) method
  being called, which set the application's ExceptionListener on
  the underlying JMS Connection, like this:
  
  ConnectionWrapper:
      |->contains ManagedConnectionImpl
      |      |-> contains JMS Connection
      |              |-> ExceptionListener: Application's
  Exception Listener
      |
      |-> contains JMS Connection
      |      |-> ExceptionListener: Application's Exception
  Listener
  
  
  This meant that if a connection error occurred on the JMS
  Exception, the application's ExceptionListener was called rather
  than the ManagedConnectionImpl.onException(JMSException) method.
  As a result, the Connection Manager did not receive a
  CONNECTION_ERROR_OCCURRED event for the Managed Connection, and
  so was unaware that it was broken and could no longer be used.
  This caused it to be left in the connection pool.
  
  The next time the application called
  Connection.createConnection(), it could get back a
  ConnectionWrapper representing a connection handle to the broken
  Managed Connection. If this happened, then when the application
  tried to use this to create a JMS Session, the attempt would
  fail with a JMSException containing reason code 2009
  (MQRC_CONNECTION_BROKEN). If the application was using a
  Connection Factory that had been configured to use:
  
  - Either a Connection Name List
  - Or a Client Channel Definition Table (CCDT)
  
  then this would result in the application being unable to create
  a JMS Connection to a queue manager, even though other queue
  managers mentioned in either the Connection Name List or entries
  in the CCDT were available.
  

Problem conclusion

• To resolve this issue, the IBM MQ JMS 2.0 resource adapter and
  IBM MQ JMS 3.0 resource adapter for Jakarta Messaging have been
  updated so that:
  
  - If an enterprise application has obtained a connection handle
  to a Managed Connection in a connection pool.
  - And registers its own ExceptionListener with the connection
  handle.
  
  then if a connection error occurs on the underlying JMS
  connection the
  "com.ibm.mq.connector.outbound.ManagedConnectionImpl" object
  will:
  
  - Send a CONNECTION_ERROR_OCCURRED event to the Connection
  Manager, to inform it that the JMS Connection has been broken.
  - And then drive the application's ExceptionListener.
  
  This ensures that the application is notified of the connection
  error (via its ExceptionListener), and that the Managed
  Connection is destroyed rather than being returned to the
  connection pool when the connection handle is closed.
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v9.3 LTS   9.3.0.5
  v9.x CD    9.3.2
  
  The latest available maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  MQ BASE V9.2

• Fixed component ID

  5724H7281

Applicable component levels