Fixes are available
7.0.0.19: WebSphere Application Server V7.0 Fix Pack 19
8.0.0.1: WebSphere Application Server V8.0 Fix Pack 1
7.0.0.21: WebSphere Application Server V7.0 Fix Pack 21
8.0.0.2: WebSphere Application Server V8.0 Fix Pack 2
8.0.0.3: WebSphere Application Server V8.0 Fix Pack 3
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
8.0.0.4: WebSphere Application Server V8.0 Fix Pack 4
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
8.0.0.5: WebSphere Application Server V8.0 Fix Pack 5
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.19: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.21: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
Obtain the fix for this APAR.
APAR status
Closed as program error.
Error description
Customer reports an ABENDSEC3 RSN04130004 in the Adjunct region. Dump shows a TCB ran a request in the Adjunct region for 60 minutes. The call stack for this TCB (native and java) is: Entry Offset Function ----- ------ -------- 1c5aaed0 000020c4 CEEOPCT 1c88fed8 000000c0 pthread_cond_timedwait 1e983bb0 0000044e monitor_wait_original 1e987700 00000084 j9thread_monitor_wait_abortable 1e961560 00000010 callMonitorWaitAbortable 1e8e9c08 00000418 Z_OBJECTMONITORENTERBLOCKING 1e94adf0 00000014 objectMonitorEnterBlocking 1e8db838 fffdcac4 RUNCALLINMETHOD 1e918058 0000003e gpProtectedRunCallInMethod 1e917e30 0000001c signalProtectAndRunGlue 1e9f1218 0000071a j9sig_protect 1e91ae50 000000a4 gpCheckCallin 1e9135a8 0000005e callVirtualObjectMethodA 1cbbc5f8 00001230 ORBEJSBridge::invoke(void*) 1cd40988 00000ea8 threadDispatch(BOSS_Object_Key*, Internal_CORBA_Request&,ORB_Request*) 1cd55060 00001ba2 SR_ExecutionThread::RemoveAndProcessWork( ThreadCleanUp*,TCB*) 1cd57ff8 000001b8 SR_ExecutionRoutine 1c5a6df0 00000f50 CELQPCMM Java: com/ibm/ws/sib/utils/ThreadLocalStack.push ThreadLocalStack.java:87 com/ibm/ws/sib/utils/ras/SibTr.push SibTr.java:448 com/ibm/ws/sib/admin/impl/HAManagerMessagingEngineImpl.membershi pChanged HAManagerMessagingEngineImpl.java:669 com/ibm/ws/hamanager/proxy/DispatchHAGroupCallbackImpl.membershi pChanged DispatchHAGroupCallbackImpl.java:166 com/ibm/ws/hamanager/proxy/_DispatchHAGroupCallbackImplBase._inv oke _DispatchHAGroupCallbackImplBase.java:140 com/ibm/ws390/orb/CommonBridge.invoke CommonBridge.java:1190 com/ibm/ws390/orb/ORBEJSBridge.invoke ORBEJSBridge.java:270 sun/reflect/GeneratedMethodAccessor33.invoke sun/reflect/DelegatingMethodAccessorImpl.invoke DelegatingMethodAccessorImpl.java:37 java/lang/reflect/Method.invoke Method.java:600 com/ibm/ws390/orb/parameters/ORBEJSBridgeInvoke.ORBEJSBridgeInvo keParmSetter ORBEJSBridgeInvoke.java:186 The extracted javacore shows TCB 8B64F8 is waiting for a monitor owned by the SIB Activation Thread on TCB 8b5a48 (again, in the adjunct) 2LKMONINUSE sys_mon_t:0x0000000A352CC700 infl_mon_t: 0x0000000A352CC770: 3LKMONOBJECT com/ibm/ws/sib/admin/impl/JsMessagingEngineImpl @0x000000081CBB92E0/0x000000081CBB92F8: Flat locked by "SIB Activation Thread" (0x000000080943ED00), entry count 1 3LKWAITERQ Waiting to enter: 3LKWAITER "WebSphere t=008b64f8" (0x0000000809428600) TCB 8B5A48 has gone outbound with a request: TCB 8b5a48 caa c13dfecf0 pthread_id 1d4579000000007a SIB Activation Thread J9VMThread 80943ed00 (daemon) Dsa Entry Offset Function --- ----- ------ -------- c13bfd1e0 1cf065d0 00000000 bboosout(BBOOSOUT_Functions,ORB_Request_SharedMemberData*,long long,int) c13bfd4e0 1cd7b480 00000594 ORB_Request::comm_outbound_request() c13bfd720 1cdce160 00000c0e Java_com_ibm_ws390_orb_ClientDelegate_invokeRequestCFW c13bfe420 1e8dbdf0 e1724210 RUNJAVATHREAD c13bfe860 1e953ae8 000000e0 javaProtectedThreadProc c13bfe960 1e9f1218 0000071a j9sig_protect c13bff160 1e9565e8 00000072 javaThreadProc c13bff260 1e9855e0 0000051c thread_wrapper c13bff360 1c5a6df0 00000f50 CELQPCMM Java stack: ----------- com/ibm/ws390/orb/ClientDelegate.invokeRequestCFW ClientDelegate.java com/ibm/ws390/orb/ClientDelegate.commonInvoke ClientDelegate.java:1018 com/ibm/ws390/orb/ClientDelegate.invoke ClientDelegate.java:865 org/omg/CORBA/portable/ObjectImpl._invoke ObjectImpl.java:484 com/ibm/ws/cluster/zos/_ControllerClusterManagerImpl_Stub.publis h _ControllerClusterManagerImpl_Stub.java:544 com/ibm/ws/cluster/zos/SRDescriptionManager.publish SRDescriptionManager.java:287 com/ibm/ws/cluster/zos/SRDescriptionManager.publish SRDescriptionManager.java:364 com/ibm/ws/cluster/zos/ZDescriptionManager.publish ZDescriptionManager.java:196 com/ibm/websphere/cluster/topography/DescriptionA.updateMemento DescriptionA.java:160 com/ibm/ws/cluster/topography/ClusterDescriptionImpl.addMember ClusterDescriptionImpl.java:387 com/ibm/ws/cluster/service/ClusterManagementImpl.joinCluster ClusterManagementImpl.java:283 com/ibm/ws/cluster/zos/ZosClusterManagementImpl.joinCluster ZosClusterManagementImpl.java:108 com/ibm/ws/cluster/service/ClusterMemberServiceImpl.joinCluster ClusterMemberServiceImpl.java:503 com/ibm/ws/sib/trm/wlm/server/Join.join Join.java:211 com/ibm/ws/sib/trm/wlm/server/Join.toShortToLongName Join.java:136 com/ibm/ws/sib/trm/wlm/server/Manager.start Manager.java:144 com/ibm/ws/sib/trm/TrmMeMainImpl.start TrmMeMainImpl.java:266 com/ibm/ws/sib/admin/impl/JsMessagingEngineImpl.start JsMessagingEngineImpl.java:609 com/ibm/ws/sib/admin/impl/HAManagerMessagingEngineImpl.activate HAManagerMessagingEngineImpl.java:995 com/ibm/ws/sib/admin/impl/JsActivationThread.run JsActivationThread.java:92 THIS IS THE LOOPING TCB in the Controller region, processing the request on TCB=8b5a48 TCB 8b6e00 caa a4f7fecf0 pthread_id 1d952000000006d WebSphere t=008b6e00 J9VMThread 8098df000 (daemon) Dsa Entry Offset Function --- ----- ------ -------- b2fafa4e0 1fe29838 fd722688 RUNCALLINMETHOD b2fafa920 1fe66058 0000003e gpProtectedRunCallInMethod b2fafaa20 1fe65e30 0000001c signalProtectAndRunGlue b2fafab20 1ff3f218 0000071a j9sig_protect b2fafb320 1fe68e50 000000a4 gpCheckCallin b2fafb460 1fe615a8 0000005e callVirtualObjectMethodA b2fafb560 1cbc15f8 00001230 ORBEJSBridge::invoke(void*) b2fafc560 1cd45988 00000ea8 threadDispatch(BOSS_Object_Key*, Internal_CORBA_Request&, ORB_Request*) b2fafd1a0 1cd47ba8 00000a56 ACR_ExecutionThread:: ProcessInboundRequest(acrwObj*, ThreadCleanUp*,BOSS_Object_Key&, Interna... b2fafe3a0 1cd577e8 0000086e ACR_ExecutionThread:: RemoveAndProcessWork(ThreadCleanUp*) b2faff220 1cd599c8 0000022e ACR_ExecutionRoutine b2faff360 1c5a9df0 00000f50 CELQPCMM Java stack: ----------- java/util/HashMap.findNonNullKeyEntry HashMap.java:526 java/util/HashMap.putImpl HashMap.java:622 java/util/HashMap.put HashMap.java:605 java/util/HashSet.add HashSet.java:95 com/ibm/ws/cluster/zos/CRDescriptionManagerImpl.publish CRDescriptionManagerImpl.java:206 com/ibm/ws/cluster/zos/ControllerClusterManagerImpl.publish ControllerClusterManagerImpl.java:582 com/ibm/ws/cluster/zos/_ControllerClusterManagerImpl_Tie.publish _ControllerClusterManagerImpl_Tie.java:255 com/ibm/ws/cluster/zos/_ControllerClusterManagerImpl_Tie._invoke _ControllerClusterManagerImpl_Tie.java:84 sun/reflect/GeneratedMethodAccessor15.invoke sun/reflect/DelegatingMethodAccessorImpl.invoke DelegatingMethodAccessorImpl.java:37 java/lang/reflect/Method.invoke Method.java:600 com/ibm/ws390/orb/RIRProtectedProxyHandler.invoke RIRProtectedProxyHandler.java:186 $Proxy0._invoke com/ibm/ws390/orb/CommonBridge.invoke CommonBridge.java:1190 com/ibm/ws390/orb/ORBEJSBridge.invoke ORBEJSBridge.java:270 sun/reflect/GeneratedMethodAccessor11.invoke sun/reflect/DelegatingMethodAccessorImpl.invoke DelegatingMethodAccessorImpl.java:37 java/lang/reflect/Method.invoke Method.java:600 com/ibm/ws390/orb/parameters/ORBEJSBridgeInvoke.ORBEJSBridgeInvo keParmSetter ORBEJSBridgeInvoke.java:186 Evidence suggest the loop on TCB=8b5a48 ran for 1 hour.
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server z/OS on versions 7.0 & 8.0 * **************************************************************** * PROBLEM DESCRIPTION: Infinite loop in HashMap * **************************************************************** * **************************************************************** Unprotected concurrent access to a hashmap in the WLM class com/ibm/ws/cluster/zos/CRDescriptionManagerImpl can cause an infinite loop in the underlying HashMap inplementation. A customer hitting this problem would see symptoms as described above, any stack should have the following piece present: com/ibm/ws/cluster/zos/CRDescriptionManagerImpl.publish CRDescriptionManagerImpl.java:206
Problem conclusion
The WLM code in question had synchronization added around the usage of the hashmap to prevent the infinite loop from occurring. APAR PM35764 is currently targeted for inclusion in Service Level (Fix Pack) 7.0.0.19, 8.0.0.1 of WebSphere Application Server Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
Comments
APAR Information
APAR number
PM35764
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
700
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2011-03-28
Closed date
2011-07-08
Last modified date
2011-10-04
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500
Applicable component levels
R700 PSY UK71280
UP11/09/10 P F109
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
Document Information
Modified date:
27 October 2021