PH56962: APPLICATION FAILS WITH J2CA0045E/J2CA0027E ERRORS AFTER WEBSPHERE APPLICATION SERVER UPGRADE FROM V9.0.5.13 TO 9.0.5.15

APAR status

• Closed as program error.

Error description

• WebSphere Application Server was upgraded from v9.0.5.13 to
  v9.0.5.15. After the upgrade, the application throws J2CA0045E
  errors. Reverting back to v9.0.5.13 resolves the errors.
  
  SystemOut.log shows the following errors:
  
  CWSJY0003W: JMSCC3036: An exception has been delivered to the
  connection's exception listener: '
  Message : com.ibm.msg.client.jms.DetailedJMSException:
  JMSWMQ1107: A problem with this connection has occurred.
  An error has occurred with the IBM MQ JMS connection.
  Use the linked exception to determine the cause of this error.
                          Class : class
  com.ibm.msg.client.jms.DetailedJMSException
                          Stack : com.ibm.msg.client.wmq.common.i
  nternal.Reason.reasonToException(Reason.java:595)
                                : com.ibm.msg.client.wmq.common.i
  nternal.Reason.createException(Reason.java:215)
                                : com.ibm.msg.client.wmq.internal
  .WMQConnection.consumer(WMQConnection.java:801)
                               ....
  Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001:
  IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason
  '2009' ('MQRC_CONNECTION_BROKEN').
                          Class : class com.ibm.mq.MQException
                          Stack : com.ibm.msg.client.wmq.common.i
  nternal.Reason.createException(Reason.java:203)
                                : com.ibm.msg.client.wmq.internal
  .WMQConnection.consumer(WMQConnection.java:801)
                                : com.ibm.mq.jmqi.remote.api.Remo
  teHconn.callEventHandler(RemoteHconn.java:3107)
                                : com.ibm.mq.jmqi.remote.api.Remo
  teHconn.driveEventsEH(RemoteHconn.java:653)
                               ...
  J2CA0056I: The Connection Manager received a fatal connection
  error from the Resource Adapter for resource
  xxxConnectionFactory. The exception is: state STATE_ACTIVE_FREE
  com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A
  problem with this connection has occurred.
  An error has occurred with the IBM MQ JMS connection.
  Use the linked exception to determine the cause of this
  error.:com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed
  with compcode '2' ('MQCC_FAILED') reason '2009'
  ('MQRC_CONNECTION_BROKEN').
  
  
  FFDC:
  [9/3/23 0:36:36:545 EDT]     FFDC Exception:com.ibm.websphere.c
  e.j2c.ConnectionWaitTimeoutExceptionSourceId:Max connections rea
  Reporter:java.lang.String@452f94db
  com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException:
  J2CA1010E: Connection not available; timed out waiting for 180
  seconds.
   at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool
  .java:1836)
   at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3970)
   at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3186)
   at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(Connect
  ionManager.java:1605)
   at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(Connec
  tionManager.java:1061)
  
  Reporter:PoolManager name:xxxConnectionFactory
  PoolManager object:-1738707851
  Total number of connections: 10 (max/min 10/0, reap/unused/aged
  180/1800/0, connectiontimeout/purge 180/EntirePool)
                                (testConnection/inteval false/0,
  stuck timer/time/threshold 0/0/0, surge time/connections 0/-1)
                                (pool paused false, prePopulate
  alternate false, resourceFailBackEnabled true,
  isAlternateResourceEnabled false, disableDatasourceFailoverAlarm
  false, startFailBack false)
  
  (isPartialResourceAdapterFailoverSupportEnabled false,
  isAlteranteResourcePoolManager false,
  resourceAvailabilityTestRetryInterval 10, currentInusePool null,
  currentMode 100, alternate jndiName null)
  Shared Connection information (shared partitions 200)
   No shared connections
  
  Free Connection information (free distribution table/partitions
  5/1)
   No free connections
  
  UnShared Connection information
   No unshared connections
  
  Below connections is/are in middle stage of status changing:
  
  [REMOVING]  MCWrapper id 6fdae407  Managed connection
  com.ibm.ejs.jms.JMSManagedConnection@3c105bd0
   managed connection factory =
  com.ibm.ejs.jms.WMQJMSRAManagedConnectionFactory@21e12ed7
   connectorName = JMS$Customer
  Notification$JMSManagedConnection@4
   Pooled Object type = JMSConnection
   XACapable = true
   physical connection =
  com.ibm.mq.jms.MQXATopicConnection@b57e7596
   credential =
  javax.resource.spi.security.PasswordCredential@d8782d6a
   open connection handles = []  State:STATE_ACTIVE_FREE
  
  If the transaction involves database, you might see the
  following errors as well:
  J2CA0027E: An exception occurred while invoking rollback on an
  XA Resource Adapter from DataSource jdbc/xxxdatasource, within
  transaction ID {XidImpl: formatId(57415344), gtrid_length(36),
  bqual_length(54),
  data(0000018a5953eae9000000015bd1a0a0844f0514534de703423ccc7529
  1f786954dbda6e0000018a5953eae9000000015bd1a0a0844f0514534de7034
  23ccc75291f786954dbda6e000000010000000000000000000000000001)}: o
  is unavailable. ORA-17008 SQLErr (0)
   at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource
  .java:1112)
   at oracle.jdbc.xa.client.OracleXAResource.rollback(OracleXARes
  ource.java:1050)
   at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.rollback(WSRdb
  XaResourceImpl.java:1366)
   at com.ibm.ejs.j2c.XATransactionWrapper.rollback(XATransaction
  Wrapper.java:1332)
   ..
   Caused by: java.sql.SQLRecoverableException: Closed Connection
   at oracle.jdbc.driver.PhysicalConnection.needLine(PhysicalConn
  ection.java:3247)
   at oracle.jdbc.driver.T4CXAResource.doTransaction(T4CXAResourc
  e.java:817)
   at oracle.jdbc.driver.T4CXAResource.doRollback(T4CXAResource.j
  ava:687)
   at oracle.jdbc.xa.client.OracleXAResource.rollback(OracleXARes
  ource.java:1045)
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server using jms connection factories.      *
  ****************************************************************
  * PROBLEM DESCRIPTION: J2CA0045E occur after                   *
  *                      jms resource adapter detects failed     *
  *                      connection during                       *
  *                      matchManagedConnection processing.      *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  The jms resource adapter detects failed connections during a
  heartbeat thread or during matchManagedConnection processing
  which can result in connections that are not destoryed. If the
  failed connections build up in the connection pool, J2CA0045E
  can occur without recovering.   The only way to recover from
  this issue is to restart the server.
  

Problem conclusion

• The failing connection are removed from the connection pool
  avoiding the build up of failed connections.
  
  The fix for this APAR is targeted for inclusion in fix pack
  8.5.5.25 and 9.0.5.18. For more information, see 'Recommended
  Updates for WebSphere Application Server':
  https://www.ibm.com/support/pages/node/715553
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBS APP SERV N

• Fixed component ID

  5724H8800

Applicable component levels