A fix is available
APAR status
Closed as program error.
Error description
IRR401I 878 ABEND DURING RACF PROCESSING OF LOCATE REQUEST FOR ENTRY MQSTARC1.CSQ1.A0022474 IRR401I 878 ABEND DURING RACF PROCESSING IRR413I RACF MANAGER REQUEST ID WAS RCK06001 IEC614I PRLSE FAILED - RC 016, DIAGNOSTIC INFORMATION IS (08160878), CSQ1MSTR,P20119,XXXARC1.CSQ1.A0022474 IEC216I A14-04,IFG0202E,CSQ1MSTR,CSQ1MSTR,SYS01527,2936,P20119, MQSTARC1.CSQ1.A0022474 CSQJ104E xxxxx CSQJDS04 RECEIVED ERROR STATUS DUMP TITLE=JOB=CSQ1MSTR,CAS ESTAE-5695DF105 R210,ABEND878, +0000,FMID=HDZ2110 ,MAINT=NONE IEC342I CATALOG ABEND OCCURRED L3 found: DISPLAY QSTATUS processing has got an ILAT latch (CSQIESEL) and CSQM1DQQ is suspended with CSQEOCRQ waiting for CSQEWRKT to locate the problem structure task. However, CSQEWRKT is waiting for the ECTA latch for the BOOK structure, which is held by the BOOK structure task, STRTSK04. STRTSK04 is suspended in CSQEROUT waiting for structure backup/ recovery task BKPREC04, which is waiting for BKPREC0E which is processing the recovery of the problem structure and is suspended with CSQIESEL waiting for the ILAT latch mentioned above (CSQIESEL is invoked by CSQERCF2). This has therefore resulted in a deadly-embrace, and has lead to a build-up of DXWB control blocks (as CSQEDSG1 is waiting for the ECTA latch for the BOOK structure). Additional Symptoms: QMGR restart failed with messages CSQY224I and CSQY225E SP229 KEY7 Subpool 229 Key 7
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of WebSphere MQ for z/OS Version 7 * * Release 1 Modification 0. * **************************************************************** * PROBLEM DESCRIPTION: Structure recovery processing and * * DISPLAY QSTATUS processing hang. * * Additional symptoms may include the * * queue manager going short on storage * * and abending with S878 and * * 5C6-00E2000B. * **************************************************************** * RECOMMENDATION: * **************************************************************** A command to recover 2 or more structures is being issued or scheduled by auto-recovery. Recovery runs on the backup recovery task for the structure with the oldest backup, for instance APPL1. As multiple structures are being recovered, CSQERRPB schedules CSQERCF2 on the backup recovery tasks for the other structures, before calling CSQERCF2 for structure APPL1. Once completed, it will suspend on backup recovery task ROBs to wait for the other backup recovery tasks to finish CSQERCF2 processing. CSQERCF2 processing for structure APPL2 has not yet reached the point where it tries to acquire a latch on the queue synonym chain. As CSQERCF2 has completed for structure APPL1, all shared queues have been restored, and flagged as cached in the CF structure. CSQERCF2 has scheduled a close request to the structure task, which disconnects from the structure and checks if the task can now go dormant. As the task is no longer used, it queues a request to the backup recovery task to quiesce and suspends, waiting for the task to go away. When a DISPLAY QSTATS command comes in with a wild card, matching a queue hosted on structure APPL1, CSQM1DQQ is invoked for the queue, detects that it is cached in the CF and attempts to get the most recent values from the CF, causing it to schedule an open request to the structure task for APPL1. This processing is delayed, due to the structure task waiting for the backup recovery task to finish processing. The DISPLAY QSTATS command is in the middle of CSQIESEL processing at this point, holding the relevant queue synonym chain latch. CSQERCF2 for structure APPL2 now reaches the point where it needs to restore the shared queues, and calls CSQIESEL to iterate over the queues. It tries to latch the queue synonym chain latch that is currently held by the thread processing the DISPLAY QSTATS command. The dead lock occurs: - The structure task APPL1 is waiting for the associated backup recovery task to complete processing - The backup recovery task for APPL1 is waiting for the backup recovery task for APPL2 to complete processing. - The backup recovery task for APPL2 is waiting for the queue synonym chain latch held by the thread processing the DISPLAY QSTATUS command. - The command processing the DISPLAY QSTATUS command is waiting for the structure task for APPL1 to become available.
Problem conclusion
The code was changed to prevent the structure task from attempting to quiesce the backup recovery task if recovery processing is still in progress. 100Y CSQEROUT CSQERRPB
Temporary fix
********* * HIPER * *********
Comments
APAR Information
APAR number
PI11896
Reported component name
WMQ Z/OS V7
Reported component ID
5655R3600
Reported release
100
Status
CLOSED PER
PE
NoPE
HIPER
YesHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2014-02-17
Closed date
2014-07-31
Last modified date
2014-09-03
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
UI20146
Modules/Macros
CSQEROUT CSQERRPB
Fix information
Fixed component name
WMQ Z/OS V7
Fixed component ID
5655R3600
Applicable component levels
R100 PSY UI20146
UP14/08/30 P F408 ¢
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
[{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SG19M","label":"APARs - z\/OS environment"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1","Edition":"","Line of Business":{"code":"","label":""}}]
Document Information
Modified date:
03 September 2014