APAR status
Closed as program error.
Error description
In WebSphere Application Server v8.5.5.x when IO Error: Socket connect timed out error is thrown due to database/network issue, Java Batch application consumes all the connections in the jdbc connection pool which leads to the J2CA0045E errors. SystemOut.log shows the following error: J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/xxxds. CWLRB4700E: [Long Running Job Scheduler com.ibm.ws.batch.SchedulerStoreFactory getConnection failed]: com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: J2CA1010E: Connection not available; timed out waiting for 180 seconds. at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java :1696) at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcD ataSource.java:661) at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcD ataSource.java:611) .. Caused by: com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: J2CA1010E: Connection not available; timed out waiting for 180 seconds. at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java :1829) at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3861) at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3109) at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionMa nager.java:1588) at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionM anager.java:1050) Following FFDCs were produced: FFDC Exception:java.sql.SQLRecoverableException SourceId:com.ibm.ws.rsadapter.spi.InternalGenericDataStoreHelper .getPooledCon ProbeId:1298 java.sql.SQLRecoverableException: IO Error: Socket connect timed out at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:854) at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection .java:793) at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExt ension.java:57) at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:747) at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDa taSource.java:406) at oracle.jdbc.xa.client.OracleXADataSource.getPooledConnection(Ora cleXADataSource.java:525) at oracle.jdbc.xa.client.OracleXADataSource.getPooledConnection(Ora cleXADataSource.java:255) at oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleX ADataSource.java:153) at oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleX ADataSource.java:110) at com.ibm.ws.rsadapter.spi.InternalGenericDataStoreHelper$1.run(In ternalGenericDataStoreHelper.java:1365) at com.ibm.ws.security.util.AccessController.doPrivileged(AccessCon troller.java:118) at com.ibm.ws.rsadapter.spi.InternalGenericDataStoreHelper.getPoole dConnection(InternalGenericDataStoreHelper.java:1384) at com.ibm.ws.rsadapter.spi.WSRdbDataSource.getPooledConnection(WSR dbDataSource.java:2154) at com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.getConne ction(WSManagedConnectionFactoryImpl.java:1809) at com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createMa nagedConnection(WSManagedConnectionFactoryImpl.java:1579) at com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createMa nagedConnection(WSManagedConnectionFactoryImpl.java:1129) at com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(Fr eePool.java:2215) at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java :1885) at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3861) at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3109) at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionMa nager.java:1588) at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionM anager.java:1050) at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcD ataSource.java:644) at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcD ataSource.java:611) at com.ibm.ws.extensionhelper.db.impl.DatabaseHelperImpl$DSWrapper. getConnection(DatabaseHelperImpl.java:1603) at com.ibm.ws.extensionhelper.db.impl.DatabaseHelperImpl.getConnect ion(DatabaseHelperImpl.java:751) at com.ibm.ws.batch.SchedulerStoreFactory.getConnection(SchedulerSt oreFactory.java:376) at com.ibm.ws.batch.SchedulerSingleton.getConnection(SchedulerSingl eton.java:1055) at com.ibm.ws.batch.SchedulerSingleton.submitJob(SchedulerSingleton .java:1391) at com.ibm.ws.batch.SchedulerSingleton.submitJob(SchedulerSingleton .java:1232) at com.ibm.ws.batch.SchedulerSingleton.submitModifiableJobFromRepos itory(SchedulerSingleton.java:12524) at com.ibm.ws.batch.JobSchedulerBean.submitModifiableJobFromReposit ory(JobSchedulerBean.java:500) FFDC Exception:com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutExceptio n SourceId:Max connections reached ProbeId:869 Reporter:java.lang.String@7d267580 com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: J2CA1010E: Connection not available; timed out waiting for 180 seconds. at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java :1829) at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3861) .... at com.ibm.ws.batch.SchedulerStoreFactory.getConnection(SchedulerSt oreFactory.java:376) at com.ibm.ws.batch.SchedulerSingleton.getConnection(SchedulerSingl eton.java:1055) at com.ibm.ws.batch.SchedulerSingleton.getJobNumberSequence(Schedul erSingleton.java:2272) at com.ibm.ws.batch.SchedulerSingleton.syncedJobID(SchedulerSinglet on.java:2586) at com.ibm.ws.batch.SchedulerSingleton.getJobID(SchedulerSingleton. java:2188) at com.ibm.ws.batch.SchedulerSingleton.getEpsJob(SchedulerSingleton .java:13301) at com.ibm.ws.batch.SchedulerSingleton.getEpsJob(SchedulerSingleton .java:13290) at com.ibm.ws.batch.SchedulerSingleton.submitModifiableJobFromRepos itory(SchedulerSingleton.java:1
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server * * Java Batch * **************************************************************** * PROBLEM DESCRIPTION: Connection pool exhausted after * * StaleConnection exceptions occur, and * * the ConnLeakLogic trace shows the * * connections originating from Java * * Batch * * com.ibm.ws.batch.SchedulerStoreFactory * * .getConnection * **************************************************************** * RECOMMENDATION: * **************************************************************** When a getConnection attempt is made by the Java Batch runtime, there is the possibility that a StaleConnectionException can be encountered. This operation retries a number of times and attempts to getConnection again. However, if a subsequent attempt is then successful, there is a code bug where the operation does not continue, and the getConnection loop continues without using or closing the connections. Eventually, this can result in running out of connections.
Problem conclusion
A code update has been made to properly continue after a successful getConnection attempt that is preceded by a StaleConnectionException. Code updates have also been to properly get/use/close connections in affected areas. The fix for this APAR is targeted for inclusion in fix pack 8.5.5.25 and 9.0.5.18. For more information, see 'Recommended Updates for WebSphere Application Server': https://www.ibm.com/support/pages/node/715553
Temporary fix
Comments
APAR Information
APAR number
PH55523
Reported component name
WEBS APP SERV N
Reported component ID
5724H8800
Reported release
850
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2023-07-03
Closed date
2023-10-12
Last modified date
2023-10-12
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBS APP SERV N
Fixed component ID
5724H8800
Applicable component levels
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.5","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
13 October 2023