Fixes are available
8.5.5.15: WebSphere Application Server V8.5.5 Fix Pack 15
9.0.0.11: WebSphere Application Server traditional V9.0 Fix Pack 11
9.0.5.0: WebSphere Application Server traditional Version 9.0.5 Refresh Pack
9.0.5.1: WebSphere Application Server traditional Version 9.0.5 Fix Pack 1
9.0.5.2: WebSphere Application Server traditional Version 9.0.5 Fix Pack 2
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
9.0.5.3: WebSphere Application Server traditional Version 9.0.5 Fix Pack 3
8.5.5.20: WebSphere Application Server V8.5.5.20
8.5.5.18: WebSphere Application Server V8.5.5 Fix Pack 18
8.5.5.19: WebSphere Application Server V8.5.5 Fix Pack 19
8.5.5.16: WebSphere Application Server V8.5.5 Fix Pack 16
8.5.5.21: WebSphere Application Server V8.5.5.21
APAR status
Closed as program error.
Error description
When the root coordinator sends a prepare request via WS-AT to a downstream WS-AT Participant, and this is holding a lock. For some reason the prepare operation hangs on something. Later we can see many WebContainer threads hanging on WS-AT code as follows: 3XMTHREADINFO "WebContainer : 62" J9VMThread:0x000000000E55CB00, j9thread_t:0x000003FFA2AFFC10, java/lang/Thread:0x00000000960A44F0, state:B, prio=5 3XMJAVALTHREAD (java/lang/Thread getId:0xB77, isDaemon:true) 3XMTHREADINFO1 (native thread ID:0xB44A, native priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread flags:0x00000201) 3XMTHREADINFO2 (native stack address range from:0x000003FFD6492000, to:0x000003FFD64D2000, size:0x40000) 3XMCPUTIME CPU usage total: 0.485163181 secs 3XMTHREADBLOCK Blocked on: com/ibm/ws/Transaction/JTS/TransactionWrapper@0x00000000951E99C8 Owned by: "WebContainer : 10" (J9VMThread:0x0000000004340600, java/lang/Thread:0x000000009A026AE0) 3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at com/ibm/ws/Transaction/wstx/WSATParticipantPortDistImpl.rollback Operation(WSATParticipantPortDistImpl.java:210(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/Transaction/wstx/WSATParticipantBindingImpl.rollbackO peration(WSATParticipantBindingImpl.java:183(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/Transaction/wstx/ns0410/ParticipantPortSoapBindingImp l.rollbackOperation(ParticipantPortSoapBindingImpl.java:104(Comp iled Code)) 4XESTACKTRACE at sun/reflect/GeneratedMethodAccessor277.invoke(Bytecode PC:40(Compiled Code)) 4XESTACKTRACE at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod AccessorImpl.java:56(Compiled Code)) 4XESTACKTRACE at java/lang/reflect/Method.invoke(Method.java:620(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/dispatchers/java/JavaDispatcher.in vokeMethod(JavaDispatcher.java:203(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/dispatchers/java/JavaDispatcher.in vokeOperation(JavaDispatcher.java:158(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/dispatchers/SoapRPCProcessor.proce ssOneway(SoapRPCProcessor.java:453(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/dispatchers/SoapRPCProcessor.proce ssMessage(SoapRPCProcessor.java:395(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/dispatchers/BasicDispatcher.proces sMessage(BasicDispatcher.java:134(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/dispatchers/java/SessionDispatcher .invoke(SessionDispatcher.java:204(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/PivotHandlerWrapper.invoke(PivotHa ndlerWrapper.java:264(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/handlers/jaxrpc/JAXRPCHandler.invo ke(JAXRPCHandler.java:153(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/handlers/WrappedHandler.invoke(Wra ppedHandler.java:64(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/PivotHandlerWrapper.invoke(PivotHa ndlerWrapper.java:264(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/PivotHandlerWrapper.invoke(PivotHa ndlerWrapper.java:264(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/engine/WebServicesEngine.invoke(WebServic esEngine.java:336(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webservices/sysengine/SystemEngineRouter.doPost(Syste mEngineRouter.java:147(Compiled Code)) ...
Local fix
N/A
Problem summary
**************************************************************** * USERS AFFECTED: WebSphere Application Server users of Web * * Services Atomic Transactions * **************************************************************** * PROBLEM DESCRIPTION: The WebContainer runs out of threads * * after a hang in a Resource Adapter * * processing a transaction completion * * protocol request * **************************************************************** * RECOMMENDATION: * **************************************************************** An issue in a Resource Adapter caused it to hang while processing a transaction completion protocol request, for example processing an JTA-XA prepare request. The transaction being completed was imported to the application server via Web Service Atomic Transaction (WS-AT) Context on a web service request. In such a case the superior transaction coordinator resends transaction completion protocol requests via the WS-AT protocol. Such protocol requests are processed on WebContainer threads and will be periodically resent by the superior until it receives the corresponding WS-AT protocol response. The WS-AT component held an intrinsic lock while making the call to the Resource Adapter. Each time the superior resent a WS-AT protocol request a new WebContainer thread was used to process the new request and each of these threads blocked while trying to obtain the same intrinsic lock. Eventually the WebContainer ran out of threads. The root cause of the issue was the hanging Resource Adapter, however the WS-AT component must be able to cope with such an issue without adversely affecting other work.
Problem conclusion
The handling of inbound WS-AT protocol requests was augmented with a mechanism for detecting multiple concurrent requests for the same transaction allowing a fast fail mechanism to be employed avoiding the unlimited consumption of threads. The fix for this APAR is currently targeted for inclusion in fix packs 8.5.5.15 and 9.0.0.11. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
Comments
APAR Information
APAR number
PH02450
Reported component name
WEBS APP SERV N
Reported component ID
5724H8800
Reported release
850
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2018-09-04
Closed date
2018-10-30
Last modified date
2018-10-30
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBS APP SERV N
Fixed component ID
5724H8800
Applicable component levels
R800 PSY
UP
R850 PSY
UP
R900 PSY
UP
Document Information
Modified date:
28 April 2022