Direct link to fix
APAR status
Closed as program error.
Error description
In some scenarios, especially when an SCA web service export receives SOAP requests and there are operations on a BPEL-related resource originated by BPEL REST APIs, you might notice that some threads timed out. You'll see that a deadlock occurred if you analyze the Java core dumps, which look like the following example: 3XMTHREADINFO "WebContainer : 104" J9VMThread:0x00000000006BE200, omrthread_t:0x00007F6B68DB6510, java/lang/Thread:0x00000001AA0DE750, state:P, prio=5 3XMJAVALTHREAD (java/lang/Thread getId:0x407, isDaemon:true) 3XMTHREADINFO1 (native thread ID:0x6B1C, native priority:0x5, native policy:UNKNOWN, vmstate:P, vm thread flags:0x00020001) 3XMTHREADINFO2 (native stack address range from:0x00007F6B1C5D4000, to:0x00007F6B1C615000, size:0x41000) 3XMCPUTIME CPU usage total: 1315.220940107 secs, current category="Application" 3XMTHREADBLOCK Parked on: java/util/concurrent/locks/ReentrantReadWriteLock$NonfairSync@0x 0000000101715DB0 Owned by: <unknown> 3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at sun/misc/Unsafe.park(Native Method) 4XESTACKTRACE at java/util/concurrent/locks/LockSupport.park(LockSupport.java:186 (Compiled Code)) 4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndChe ckInterrupt(AbstractQueuedSynchronizer.java:847(Compiled Code)) 4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireQue ued(AbstractQueuedSynchronizer.java:881(Compiled Code)) 4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquire(Ab stractQueuedSynchronizer.java:1210(Compiled Code)) 4XESTACKTRACE at java/util/concurrent/locks/ReentrantReadWriteLock$WriteLock.lock (ReentrantReadWriteLock.java:954(Compiled Code)) 4XESTACKTRACE at com/ibm/xml/sdo/helper/TypeHelperMap.cleanupHelperContext(TypeHe lperMap.java:627) 4XESTACKTRACE at com/ibm/xml/sdo/helper/TypeHelperImpl.cleanupHelperContext(TypeH elperImpl.java:311) 4XESTACKTRACE at com/ibm/wsspi/sdox/ReferenceSharingScopeManager.cleanupHelperCon text(ReferenceSharingScopeManager.java:71) 4XESTACKTRACE at com/ibm/ws/box/BOCore.clearScope(BOCore.java:306) 4XESTACKTRACE at com/ibm/ws/box/BORALCacheCleaner.clearScope(BORALCacheCleaner.ja va:38) 4XESTACKTRACE at com/ibm/ws/al/util/BOCacheHelper.clearBOCache(BOCacheHelper.java :51) 4XESTACKTRACE at com/ibm/ws/al/scope/ScopeContextManager.removeContext(ScopeConte xtManager.java:904) 5XESTACKTRACE (entered lock: com/ibm/ws/al/scope/ScopeContextManager@0x0000000100618D38, entry count: 3) 4XESTACKTRACE at com/ibm/ws/al/scope/ScopeContextManager.refreshRALCache(ScopeCon textManager.java:555) 5XESTACKTRACE (entered lock: com/ibm/ws/al/scope/ScopeContextManager@0x0000000100618D38, entry count: 2) 4XESTACKTRACE at com/ibm/ws/al/scope/ScopeContextManager.restoreContext(ScopeCont extManager.java:580) 5XESTACKTRACE (entered lock: com/ibm/ws/al/scope/ScopeContextManager@0x0000000100618D38, entry count: 1) 4XESTACKTRACE at com/ibm/wsspi/al/ALContext.unset(ALContext.java:134) 4XESTACKTRACE at com/ibm/bpe/api/ClientObjectWrapperALHelper.unsetALContext(Clien tObjectWrapperALHelper.java:73) 4XESTACKTRACE at com/ibm/bpe/api/ClientObjectWrapper.getObject(ClientObjectWrappe r.java:345(Compiled Code)) 4XESTACKTRACE at com/ibm/bpe/api/ClientObjectWrapper.getObject(ClientObjectWrappe r.java:174) 4XESTACKTRACE at com/ibm/bpe/api/rest/framework/BFMRestImpl.getVariable(BFMRestIm pl.java:2670) 4XESTACKTRACE at com/ibm/bpe/api/rest/framework/BFMRestServlet.doGet(BFMRestServl et.java:2251) 4XESTACKTRACE at javax/servlet/http/HttpServlet.service(HttpServlet.java:575(Comp iled Code)) ... 3XMTHREADINFO "WebContainer : 102" J9VMThread:0x0000000000683F00, omrthread_t:0x00007F6B68DAA0B0, java/lang/Thread:0x00000001AA0DE600, state:B, prio=5 3XMJAVALTHREAD (java/lang/Thread getId:0x405, isDaemon:true) 3XMTHREADINFO1 (native thread ID:0x6B1A, native priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread flags:0x00000201) 3XMTHREADINFO2 (native stack address range from:0x00007F6B1C656000, to:0x00007F6B1C697000, size:0x41000) 3XMCPUTIME CPU usage total: 1182.721685078 secs, current category="Application" 3XMTHREADBLOCK Blocked on: com/ibm/ws/al/scope/ScopeContextManager@0x0000000100618D38 Owned by: "WebContainer : 104" (J9VMThread:0x00000000006BE200, java/lang/Thread:0x00000001AA0DE750) 3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at com/ibm/ws/al/scope/ScopeContextManager.getWOLib(ScopeContextMan ager.java:110(Compiled Code)) 5XESTACKTRACE (entered lock: com/ibm/ws/al/scope/ScopeContextManager@0x0000000100618D38, entry count: 1) 4XESTACKTRACE at com/ibm/ws/al/scope/ScopeContextManager.getContextWOLib(ScopeCon textManager.java:233(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/al/ScopeContextAdapter.getContext(ScopeContextAdapter .java:59(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/al/ScopeContextAdapter.p_queryURLsSingleScope(ScopeCo ntextAdapter.java:202(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/al/ScopeContextAdapter.queryURLsSingleScopeNoLib(Scop eContextAdapter.java:186(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/al/ArtifactLoaderAdapter.queryURLsSingleScopeNoLib(Ar tifactLoaderAdapter.java:104(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/box/BOScope.getArtifacts(BOScope.java:123(Compiled Code)) 5XESTACKTRACE (entered lock: com/ibm/ws/box/BOScope@0x000000018AA81128, entry count: 1) 4XESTACKTRACE at com/ibm/ws/box/BOResourceLocator.locateResources(BOResourceLocat or.java:358(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/box/BOResourceLocator$1.run(BOResourceLocator.java:16 4(Compiled Code)) 4XESTACKTRACE at java/security/AccessController.doPrivileged(AccessController.jav a:638(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/box/BOResourceLocator.getAllSources(BOResourceLocator .java:162(Compiled Code)) 5XESTACKTRACE (entered lock: com/ibm/ws/box/BOResourceLocator@0x000000018AA81218, entry count: 1) 4XESTACKTRACE at com/ibm/ws/box/BOScopeManager.getModuleAndLibrarySources(BOScope Manager.java:238(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/box/BOScopeManager.resolveSchema(BOScopeManager.java: 145(Compiled Code)) 4XESTACKTRACE at com/ibm/xml/sdo/helper/HelperContextImpl.resolveAndLoad(HelperCo ntextImpl.java:509(Compiled Code)) 5XESTACKTRACE (entered lock: com/ibm/xml/xci/type/TypeRegistry@0x0000000188A333D8, entry count: 2) 4XESTACKTRACE at com/ibm/xml/sdo/helper/TypeHelperMap.loadTypes(TypeHelperMap.jav a:923(Compiled Code)) 4XESTACKTRACE at com/ibm/xml/sdo/helper/TypeHelperMap.getTypeFromMutableMap(TypeH elperMap.java:885(Compiled Code)) 5XESTACKTRACE (entered lock: com/ibm/xml/xci/type/TypeRegistry@0x0000000188A333D8, entry count: 1) 4XESTACKTRACE at com/ibm/xml/sdo/helper/TypeHelperMap.getType(TypeHelperMap.java: 854(Compiled Code)) 4XESTACKTRACE at com/ibm/xml/sdo/helper/TypeHelperImpl.getType(TypeHelperImpl.jav a:462(Compiled Code)) 4XESTACKTRACE at com/ibm/xml/sdo/helper/TypeHelperImpl.getType(TypeHelperImpl.jav a:886(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/sca/internal/scdl/wsdl/managed/ManagedWrapperTypeHelp er.getLocalType(ManagedWrapperTypeHelper.java:175(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/sca/internal/scdl/wsdl/managed/ManagedWrapperTypeHelp er.isWrapperType(ManagedWrapperTypeHelper.java:119(Compiled Code)) 4XESTACKTRACE at com/ibm/wsspi/sca/scdl/impl/OperationTypeAdapterImpl.isWrapperTy pe(OperationTypeAdapterImpl.java:178(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/sca/internal/jaxws/util/WebServiceLazyParsingUtil.mat chSOAPResponseOutput(WebServiceLazyParsingUtil.java:473(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/sca/internal/jaxws/handler/JaxWsExportHandler.createS oapResponse(JaxWsExportHandler.java:788(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/sca/internal/jaxws/handler/JaxWsExportHandler.process Message(JaxWsExportHandler.java:428(Compiled Code)) 4XESTACKTRACE at com/ibm/wsspi/sca/jaxws/service/WsExtensionProvider.invoke(WsExt ensionProvider.java:131(Compiled Code)) 4XESTACKTRACE at com/ibm/wsspi/sca/jaxws/service/GenericJaxWsExportProvider.invok e(GenericJaxWsExportProvider.java:94(Compiled Code)) 4XESTACKTRACE at sun/reflect/GeneratedMethodAccessor273.invoke(Bytecode PC:40(Compiled Code)) 4XESTACKTRACE at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod AccessorImpl.java:55(Compiled Code)) 4XESTACKTRACE at java/lang/reflect/Method.invoke(Method.java:508(Compiled Code)) 4XESTACKTRACE at org/apache/axis2/jaxws/server/dispatcher/JavaDispatcher.invokeTa rgetOperation(JavaDispatcher.java:120(Compiled Code)) ... Here the thread "WebContainer : 102" had the lock in XML (com.ibm.xml.sdo.helper.TypeHelperMap) and was trying to obtain the lock of ScopeContextManager, but the thread "WebContainer : 104", which was trying to obtain the lock in XML (com.ibm.xml.sdo.helper.TypeHelperMap), had already obtained the lock of ScopeContextManager. As a result, a deadlock occurred.
Local fix
Problem summary
This issue occurs because the Artifact Loader cleaned up some business object and XML caches in its own thread lock, but did not consider the thread lock in the XML processing.
Problem conclusion
A fix that removes such a deadlock is available for IBM BPM V8.6.0.0 cumulative fix 2018.03.
Temporary fix
Comments
APAR Information
APAR number
JR62343
Reported component name
RPA WITH AA
Reported component ID
5737E8100
Reported release
J00
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2020-05-18
Closed date
2020-10-30
Last modified date
2020-10-30
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
RPA WITH AA
Fixed component ID
5737E8100
Applicable component levels
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSMGNY","label":"IBM Robotic Process Automation with Automation Anywhere"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"19.0.0.1"}]
Document Information
Modified date:
14 December 2020