PH57438: WEBSPHERE APPLICATION SERVER HANGS INDEFINITELY DUE TO EXHAUSTED ORB THREAD POOL CAUSED BY MISBEHAVING RESOURCE ADAPTER

APAR status

• Closed as program error.

Error description

• Oracle/Weblogic Tuxedo JCA Adapter is installed and configured
  in WebSphere Application Server. Intermittently, the
  application server jvm hangs indefinitely until the server is
  restarted.
  
  The SystemOut.log shows one ORB.thread.pool thread hung in the
  Oracle Tuxedo JCA Resource Adapter Code:
  
  WSVR0605W: Thread "ORB.thread.pool : 4" (00015059) has been
  active for 605506 milliseconds and may be hung.  There is/are 1
  thread(s) in total in the server that may be hung.
  at java.lang.Object.wait(Native Method)
  at java.lang.Object.wait(Object.java:167)
  at weblogic.wtc.jatmi.TuxXidRply.get_specific_reply(TuxXidRply
  .java:275)
  at com.oracle.tuxedo.adapter.tdom.TuxedoJCAResource.internalCo
  mmit(TuxedoJCAResource.java:428)
  at com.oracle.tuxedo.adapter.tdom.TuxedoJCAResource.rollback(T
  uxedoJCAResource.java:1092)
  at com.oracle.tuxedo.adapter.tdom.TuxedoJCAResource.internalPr
  epare(TuxedoJCAResource.java:949)
  at com.oracle.tuxedo.adapter.tdom.TuxedoJCAResource.prepare(Tu
  xedoJCAResource.java:802)
  at com.ibm.ejs.j2c.XATransactionWrapper.prepare(XATransactionW
  rapper.java:1152)
  at com.ibm.tx.jta.impl.JTAXAResourceImpl.prepare(JTAXAResource
  Impl.java:234)
  at com.ibm.tx.jta.impl.RegisteredResources.prepareResource(Reg
  isteredResources.java:1211)
  at com.ibm.tx.jta.impl.RegisteredResources.distributePrepare(R
  egisteredResources.java:1472)
  at com.ibm.tx.jta.impl.TransactionImpl.prepareResources(Transa
  ctionImpl.java:1489)
  at com.ibm.tx.jta.impl.TransactionImpl.internalPrepare(Transac
  tionImpl.java:1438)
  at com.ibm.ws.Transaction.JTS.TransactionWrapper.prepare(Trans
  actionWrapper.java:205)
  at com.ibm.ws.Transaction.JTS.WSCoordinatorImpl.prepare(WSCoor
  dinatorImpl.java:144)
  at com.ibm.ws.Transaction.JTS._WSCoordinatorImplBase._invoke(
  WSCoordinatorImplBase.java:50)
  at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(Ser
  verDelegate.java:631)
  at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.j
  ava:501)
  at com.ibm.rmi.iiop.ORB.process(ORB.java:624)
  at com.ibm.CORBA.iiop.ORB.process(ORB.java:1582)
  at
  com.ibm.rmi.iiop.Connection.doRequestWork(Connection.java:3151)
  at com.ibm.rmi.iiop.Connection.doWork(Connection.java:3016)
  at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:63)
  at com.ibm.ws.giop.threadpool.WorkQueueElement.dispatch(WorkQu
  eueElement.java:174)
  at com.ibm.ws.giop.filter.GiopFilterChain.processMessage(GiopF
  ilterChain.java:203)
  at com.ibm.ws.giop.threadpool.PooledThread.handleRequest(Poole
  dThread.java:81)
  at com.ibm.ws.giop.threadpool.PooledThread.run(PooledThread.ja
  va:102)
  at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
  
  
  While the rest of the ORB.thread.pool threads were hung in the
  WAS transaction code:
  
  ThreadMonitor W   WSVR0605W: Thread "ORB.thread.pool : 5"
  
  (0001506e) has been active for 755486 milliseconds and may be
  hung.  There is/are 2 thread(s) in total in the server that may
  be hung.
  at com.ibm.ws.Transaction.JTS.WSCoordinatorImpl.rollback(WSCoo
  rdinatorImpl.java:179)
  at com.ibm.ws.Transaction.JTS._WSCoordinatorImplBase._invoke(
  WSCoordinatorImplBase.java:67)
  at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(Ser
  verDelegate.java:631)
  at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.j
  ava:501)
  at com.ibm.rmi.iiop.ORB.process(ORB.java:624)
  at com.ibm.CORBA.iiop.ORB.process(ORB.java:1582)
  at
  com.ibm.rmi.iiop.Connection.doRequestWork(Connection.java:3151)
  at com.ibm.rmi.iiop.Connection.doWork(Connection.java:3016)
  at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:63)
  at com.ibm.ws.giop.threadpool.WorkQueueElement.dispatch(WorkQu
  eueElement.java:174)
  at com.ibm.ws.giop.filter.GiopFilterChain.processMessage(GiopF
  ilterChain.java:203)
  at com.ibm.ws.giop.threadpool.PooledThread.handleRequest(Poole
  dThread.java:81)
  at com.ibm.ws.giop.threadpool.PooledThread.run(PooledThread.ja
  va:102)
  at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  WebSphere Application Server users          *
  *                  of JTS Distributed Transactions             *
  ****************************************************************
  * PROBLEM DESCRIPTION: The ORB runs out of threads after a     *
  *                      hang in a Resource Adapter processing   *
  *                      a transaction completion JTS protocol   *
  *                      request                                 *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  An issue in a Resource Adapter caused it to hang while
  processing a transaction completion JTS protocol request, for
  example processing a prepare request.  The transaction being
  completed was imported to the application server via JTS on an
  IIOP request.
  In such a case the superior transaction coordinator resends
  transaction completion protocol requests via the JTS protocol.
  Such protocol requests are processed on ORB threads and will
  be periodically resent by the superior until completion.
  The Transaction Service held an intrinsic lock while making
  the call to the Resource Adapter.  Each time the superior
  resent a JTS protocol request a new ORB thread was used to
  process the new request and each of these threads blocked
  while trying to obtain the same intrinsic lock.  Eventually
  the ORB thread pool ran out of threads.
  The root cause of the issue was the hanging Resource Adapter,
  however the Transaction Service JTS component must be able to
  cope with such an issue without adversely affecting other work.
  

Problem conclusion

• The handling of inbound JTS protocol requests was augmented
  with a mechanism for detecting multiple concurrent requests
  for the same transaction allowing a fast fail mechanism to be
  employed avoiding the unlimited consumption of threads.
  
  The fix for this APAR is targeted for inclusion in fix packs
  8.5.5.26 and 9.0.5.20. For more information, see 'Recommended
  Updates for WebSphere Application Server':
  https://www.ibm.com/support/pages/node/715553
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBS APP SERV N

• Fixed component ID

  5724H8800

Applicable component levels