Fixes are available
APAR status
Closed as program error.
Error description
This issue appears to occur only in a multicell environment when load is placed on IHS during a recycle of some set of app servers in one of the cells. Odrlib attempts to rebuild the tree leading to a crash of odrlib in the odrTargetSelect.tsRemoveVirtualClusterFromMultiVCs() method. Here are two relevant stacks from this failure: #0 0x00007f2f504cdd4a in odrListGetHead (list=0x0) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrL ist.c:237 #1 0x00007f2f504cdece in odrListFind (list=0x0, obj=0x7f2f4294f070) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrL ist.c:366 #2 0x00007f2f504ce0fa in odrListFindAndRemove (list=0x0, obj=0x7f2f4294f070) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrL ist.c:479 #3 0x00007f2f504dd9ba in tsRemoveVirtualClusterFromMultiVCs (ts=0x7f2f40001940, vc=0x7f2f4294f070) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT argetSelector.c:2774 #4 0x00007f2f50499453 in vcAddServerApp (vc=0x7f2f4294f070, sapp=0x7f2f4068e8e0) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/Virt ualCluster.c:267 #5 0x00007f2f504d986a in handleLinkServerToWebModuleRequest (server=0x7f2f41cb5d80, req=0x7f2f4036c670) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT argetSelector.c:1345 #6 0x00007f2f504d99b5 in tsLinkServerToWebModule (server=0x7f2f41cb5d80, wm=0x7f2f4294f7d0, add=1 '\001') at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT argetSelector.c:1372 #7 0x00007f2f504a6f4c in odrLinkServerToWebModule (server=0x7f2f41cb5d80, owm=0x7f2f429504e0, add=1 '\001') at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr. c:2596 #8 0x00007f2f504a8ce7 in jsonHandleRelationships (json=0x7f2f40b06280, pi=0x7f2f4e8e2ba0) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr. c:3358 #9 0x00007f2f504a8fdd in jsonHandleOp (json=0x7f2f40b2c570, pi=0x7f2f4e8e2ba0) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr. c:3410 #10 0x00007f2f504a91f1 in jsonHandle (json=0x7f2f403929b0, pi=0x7f2f4e8e2ba0) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr. c:3446 #11 0x00007f2f504a9dbd in odrHandleJson (json=0x7f2f403929b0, jsonStr=0x7f2f401fc190 "{\n \"add\": {\n \"relationships\": {\n \"server:webModule\": [\n", ' ' <repeats 12 times>, "{\n", ' ' <repeats 15 times>, "\"server\": \"/cell/WDE/node/WDEFNCC001/server/WDEFNCC001C1\",\n", ' ' <repeats 15 times>, "\"webModule\": \"/cell/WDE/app"..., c=0x5587fa015ae0) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr. c:3676 #12 0x00007f2f504ac01c in odrClientProcessHandleMessage (odr=0x5587f9fb78a0, msg=0x7f2f4e8e2d30) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr. c:4591 #13 0x00007f2f504cfb94 in odrClientProcessRun (parm=0x5587fa013660) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrM ultiProcess.c:422 #14 0x00007f2f504e0a9a in odrStartRoutineWrapper (arg=0x5587fa013570) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT hread.c:60 #15 0x00007f2f5fb8ddd5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007f2f5f6b2ead in clone () from /lib64/libc.so.6 and #0 0x00007f7e4331bc76 in odrHashCompute (key=0x44572f6c6c65632f <Address 0x44572f6c6c65632f out of bounds>) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrH ash.c:287 #1 0x00007f7e4331bccb in odrHashGet (hm=0x7f7e34001670, key=0x44572f6c6c65632f <Address 0x44572f6c6c65632f out of bounds>) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrH ash.c:300 #2 0x00007f7e4333e840 in tsGetListOfVcForMultiVc (ts=0x7f7e34000fa0, multiVc=0x7f7e34aea2b0) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT argetSelector.c:2744 #3 0x00007f7e4333e9a9 in tsRemoveVirtualClusterFromMultiVCs (ts=0x7f7e34000fa0, vc=0x7f7e342de040) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT argetSelector.c:2772 #4 0x00007f7e432fa453 in vcAddServerApp (vc=0x7f7e342de040, sapp=0x7f7e34010ac0) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/Virt ualCluster.c:267 #5 0x00007f7e4333a86a in handleLinkServerToWebModuleRequest (server=0x7f7e34292f50, req=0x7f7e34011310) at /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT argetSelector.c:1345 (edited) ...
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server WAS ND edition- Virtual * * Enterprise/IM Component * **************************************************************** * PROBLEM DESCRIPTION: In a multi-cell environment, after one * * of the cells is restarted, ODRLIB * * component inside WAS IHS intermittently * * crashes. * **************************************************************** * RECOMMENDATION: * **************************************************************** In the ODRLIB component of IHS there is code that is used to select a target for incoming requests using stored information about the topology of the cells for which it is sending the requests. The software that manages that information had a defect when a cell was removed and then added back which led to this problem.
Problem conclusion
Changes were made to code that manages the target selection endpoints so that when a cell is removed and then added back, the correct endpoint object is selected. The fix for this APAR is currently targeted for inclusion in fix pack 9.0.5.1. Please refer to the Recommended Updates page for delivery information: https://www- 01.ibm.com/support/docview.wss?rs=180&uid=swg1PH14613
Temporary fix
Comments
APAR Information
APAR number
PH14613
Reported component name
WEBS APP SERV N
Reported component ID
5724H8800
Reported release
900
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2019-07-17
Closed date
2019-10-22
Last modified date
2020-01-21
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBS APP SERV N
Fixed component ID
5724H8800
Applicable component levels
R900 PSY
UP
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"9.0","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
02 November 2021