Fixes are available
6.1.0.33: Java SDK 1.5 SR12 FP1 Cumulative Fix for WebSphere
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for AIX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for HP-UX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for IBM i
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Linux
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Solaris
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Windows
7.0.0.13: Java SDK 1.6 SR8FP1 Cumulative Fix for WebSphere Application Server
6.1.0.35: Java SDK 1.5 SR12 FP2 Cumulative Fix for WebSphere
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for AIX
7.0.0.15: Java SDK 1.6 SR9 Cumulative Fix for WebSphere Application Server
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for HP-UX
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for IBM i
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Linux
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Solaris
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Windows
6.1.0.37: Java SDK 1.5 SR12 FP3 Cumulative Fix for WebSphere
7.0.0.17: WebSphere Application Server V7.0 Fix Pack 17
7.0.0.17: Java SDK 1.6 SR9 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.19: WebSphere Application Server V7.0 Fix Pack 19
7.0.0.21: WebSphere Application Server V7.0 Fix Pack 21
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
6.1.0.47: WebSphere Application Server V6.1 Fix Pack 47
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
6.1.0.39: Java SDK 1.5 SR12 FP4 Cumulative Fix for WebSphere Application Server
6.1.0.41: Java SDK 1.5 SR12 FP5 Cumulative Fix for WebSphere Application Server
6.1.0.43: Java SDK 1.5 SR13 Cumulative Fix for WebSphere Application Server
6.1.0.45: Java SDK 1.5 SR14 Cumulative Fix for WebSphere Application Server
6.1.0.47: Java SDK 1.5 SR16 Cumulative Fix for WebSphere Application Server
7.0.0.19: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.21: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
Obtain the fix for this APAR.
APAR status
Closed as program error.
Error description
Customer runs an EJB invocation stress test on a server. The server has one servant region. For some reason, the servant region experiences a timeout. BBOO0327I IIOP REQUEST TIMEOUT: (022D):(FFFFEEEE):(0000081C): (007BD8B0):(STC01545):(2010/05/04 10:21:37.074040): (2010/05/04 10:21:37.075264):(2010/05/04 10:21:37.075315): (com.abc.def.ejb.EJSRemoteStateless_00000000): (evaluateXXX):(hostname=host.name.com port=33333):():():() BBOO0232W A request for Class Name 'com.abc.def.ejb.EJSRemoteStateless_00000000' and Method Name 'evaluateXXX', from hostname=host.name.com port=33333, has timed out. The servant process associated with the request will be terminated. Request Id(FFFFEEEE) Since there is only one servant region, the server issues BBOO0299I SERVER XXXPLEX/YYYY/SERVERA HAS NO SERVANTS. WORK IS BEING REJECTED. 2 seconds later the controller region ABENDs with S0C4-4. The S0C4 psw instruction counter points into method ORB_Request::setRTN_RSN_SevereError(ORB_Request::commRSN_Codes) The call stack looks like this: ORB_Request::construct_and_send_message(SessionHandle*,GIOP_ +00087E3C BBOBOA ORB_Request::comm_outbound_response() +00001194 BBOBOA ORB_Request_Registry::drainRegistry(unsigned int) +000009A2 BBOBOA cleanupForLastSR() +000008DA BBOBOA ACR_ExecutionThread::ProcessSrCleanup(acrwObj*) +00000132 BBOBOA ACR_ExecutionThread::RemoveAndProcessWork(ThreadCleanUp*) +0000132E BBOBOA ACR_ExecutionRoutine +0000053E BBOBOA CEEVROND +000011F4 CEEPLPKA CEEOPCMM +000009A2 CEEOPCMM The S0C4 occurs because the ORBR being used by method comm_outbound_response() has been freed. DSA for comm_outbound_response + 0x840 -> ORBR control block. The ORBR eyecatcher in the first 8 bytes say RBROOOBB. It should be BBOOORBR while the ORBR is allocated. The letters are reversed just before the storage is released. The Heap Pool Prefix for this ORBR indicates the storage has been freed because the free chain pointer is non-zero. hp 3 hp 3 reversed element free | eyecatcher | chain | | ptr | | | | V V V 5C7B7FD8. 00000003 5C7B96F8 D9C2D9D6 D6D6C2C2 | RBROOOBB| 5C7B7FE0. 00000000 00000000 00000000 00000000 |................| 5C7B7FF0. 00000000 00000000 |........ | The eyecatcher should be BBOOORBR. This ORBR control block is associated with a new request coming into the controller. The controller places the ORBR into the ORBR Registry. Since there are no servant regions, the controller returns it to the requestor and removes it from the ORBR Registry . Because the servant region has recently ended due to a timeout, another thread in the controller is running ORB_Request_Registry::drainRegistry() to remove every ORBR in the ORBR Registry. There is a small window where drainRegistry() can obtain an ORBR that is being freed concurrently on another thread. This causes the S0C4.
Local fix
Run with multiple servant regions to minimize the chance that all servants will be unavailable at the same time.
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server V6.0.1 * **************************************************************** * PROBLEM DESCRIPTION: ABEND0C4/ABENDS0C4 in WebSphere * * Application Server for z/OS * * Controller following a BBOO0299I * * message. * **************************************************************** * RECOMMENDATION: * **************************************************************** Message BBOO0299I is issued when a server has lost its last servant and autopause is enabled (control_region_dreg_on_no_srs set to 1). The ORB_Request registry is drained of work at this time (ORB_Request_Registry::drainRegistry). There exists a timing window where drainRegistry can process a newly arrived request at the same time as the request is being forwarded or rejected because of the server's paused state. Both of these threads will attempt to cleanup the ORB_Request. An additional symptom is a dump issued with ERRNO=C9C212F7. This is a dump taken after detecting an incorrect ORB_Request object.
Problem conclusion
Code has been modifed in the inbound request paths to not register inbound requests unless we are accepting work for servants. APAR PM13797 is currently targeted for inclusion in Service Level (Fix Pack) 6.0.2.43 of WebSphere Application Server V6.0.1. Please refer to URL: //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970 for Fix Pack availability.
Temporary fix
Comments
APAR Information
APAR number
PM13797
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
601
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2010-05-05
Closed date
2010-07-13
Last modified date
2010-10-02
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500
Applicable component levels
R601 PSY UK60277
UP10/09/23 P F009
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
Document Information
Modified date:
25 October 2021