APAR status
Closed as program error.
Error description
WebSphere Application Server was upgraded from v9.0.5.13 to v9.0.5.15. After the upgrade, the application throws J2CA0045E errors. Reverting back to v9.0.5.13 resolves the errors. SystemOut.log shows the following errors: CWSJY0003W: JMSCC3036: An exception has been delivered to the connection's exception listener: ' Message : com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has occurred. An error has occurred with the IBM MQ JMS connection. Use the linked exception to determine the cause of this error. Class : class com.ibm.msg.client.jms.DetailedJMSException Stack : com.ibm.msg.client.wmq.common.i nternal.Reason.reasonToException(Reason.java:595) : com.ibm.msg.client.wmq.common.i nternal.Reason.createException(Reason.java:215) : com.ibm.msg.client.wmq.internal .WMQConnection.consumer(WMQConnection.java:801) .... Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN'). Class : class com.ibm.mq.MQException Stack : com.ibm.msg.client.wmq.common.i nternal.Reason.createException(Reason.java:203) : com.ibm.msg.client.wmq.internal .WMQConnection.consumer(WMQConnection.java:801) : com.ibm.mq.jmqi.remote.api.Remo teHconn.callEventHandler(RemoteHconn.java:3107) : com.ibm.mq.jmqi.remote.api.Remo teHconn.driveEventsEH(RemoteHconn.java:653) ... J2CA0056I: The Connection Manager received a fatal connection error from the Resource Adapter for resource xxxConnectionFactory. The exception is: state STATE_ACTIVE_FREE com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has occurred. An error has occurred with the IBM MQ JMS connection. Use the linked exception to determine the cause of this error.:com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN'). FFDC: [9/3/23 0:36:36:545 EDT] FFDC Exception:com.ibm.websphere.c e.j2c.ConnectionWaitTimeoutExceptionSourceId:Max connections rea Reporter:java.lang.String@452f94db com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: J2CA1010E: Connection not available; timed out waiting for 180 seconds. at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool .java:1836) at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3970) at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3186) at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(Connect ionManager.java:1605) at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(Connec tionManager.java:1061) Reporter:PoolManager name:xxxConnectionFactory PoolManager object:-1738707851 Total number of connections: 10 (max/min 10/0, reap/unused/aged 180/1800/0, connectiontimeout/purge 180/EntirePool) (testConnection/inteval false/0, stuck timer/time/threshold 0/0/0, surge time/connections 0/-1) (pool paused false, prePopulate alternate false, resourceFailBackEnabled true, isAlternateResourceEnabled false, disableDatasourceFailoverAlarm false, startFailBack false) (isPartialResourceAdapterFailoverSupportEnabled false, isAlteranteResourcePoolManager false, resourceAvailabilityTestRetryInterval 10, currentInusePool null, currentMode 100, alternate jndiName null) Shared Connection information (shared partitions 200) No shared connections Free Connection information (free distribution table/partitions 5/1) No free connections UnShared Connection information No unshared connections Below connections is/are in middle stage of status changing: [REMOVING] MCWrapper id 6fdae407 Managed connection com.ibm.ejs.jms.JMSManagedConnection@3c105bd0 managed connection factory = com.ibm.ejs.jms.WMQJMSRAManagedConnectionFactory@21e12ed7 connectorName = JMS$Customer Notification$JMSManagedConnection@4 Pooled Object type = JMSConnection XACapable = true physical connection = com.ibm.mq.jms.MQXATopicConnection@b57e7596 credential = javax.resource.spi.security.PasswordCredential@d8782d6a open connection handles = [] State:STATE_ACTIVE_FREE If the transaction involves database, you might see the following errors as well: J2CA0027E: An exception occurred while invoking rollback on an XA Resource Adapter from DataSource jdbc/xxxdatasource, within transaction ID {XidImpl: formatId(57415344), gtrid_length(36), bqual_length(54), data(0000018a5953eae9000000015bd1a0a0844f0514534de703423ccc7529 1f786954dbda6e0000018a5953eae9000000015bd1a0a0844f0514534de7034 23ccc75291f786954dbda6e000000010000000000000000000000000001)}: o is unavailable. ORA-17008 SQLErr (0) at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource .java:1112) at oracle.jdbc.xa.client.OracleXAResource.rollback(OracleXARes ource.java:1050) at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.rollback(WSRdb XaResourceImpl.java:1366) at com.ibm.ejs.j2c.XATransactionWrapper.rollback(XATransaction Wrapper.java:1332) .. Caused by: java.sql.SQLRecoverableException: Closed Connection at oracle.jdbc.driver.PhysicalConnection.needLine(PhysicalConn ection.java:3247) at oracle.jdbc.driver.T4CXAResource.doTransaction(T4CXAResourc e.java:817) at oracle.jdbc.driver.T4CXAResource.doRollback(T4CXAResource.j ava:687) at oracle.jdbc.xa.client.OracleXAResource.rollback(OracleXARes ource.java:1045)
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server using jms connection factories. * **************************************************************** * PROBLEM DESCRIPTION: J2CA0045E occur after * * jms resource adapter detects failed * * connection during * * matchManagedConnection processing. * **************************************************************** * RECOMMENDATION: * **************************************************************** The jms resource adapter detects failed connections during a heartbeat thread or during matchManagedConnection processing which can result in connections that are not destoryed. If the failed connections build up in the connection pool, J2CA0045E can occur without recovering. The only way to recover from this issue is to restart the server.
Problem conclusion
The failing connection are removed from the connection pool avoiding the build up of failed connections. The fix for this APAR is targeted for inclusion in fix pack 8.5.5.25 and 9.0.5.18. For more information, see 'Recommended Updates for WebSphere Application Server': https://www.ibm.com/support/pages/node/715553
Temporary fix
Comments
APAR Information
APAR number
PH56962
Reported component name
WEBS APP SERV N
Reported component ID
5724H8800
Reported release
900
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2023-09-14
Closed date
2023-11-14
Last modified date
2023-11-14
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBS APP SERV N
Fixed component ID
5724H8800
Applicable component levels
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"9.0","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
15 November 2023