A fix is available
APAR status
Closed as program error.
Error description
5C6-00C510A4 issued by queue manager in CFSTRUCT due to an IXLFORCE call returning with IXLRSNCODEREBUILDINPROGRESS . A structure can only be recovered when none of the connections to it is in a "loss of connectivity" state. Thus, the long delay between the initial loss of connectivity and the adequate processing of this event, triggered by the queue managers receiving the EEPLXESRECOMMENDACTION events, creates a rather large timing window for this issue to occur. As can be seen in the syslog for each of these systems, an attempt to initiate a duplexing build of a structure was made after the queue managers had lost connectivity, but before their connections could be recovered. Thus the duplexing build of the structures failed, as indicated by the IXC538I message issued in the syslog, and the systems went into a wait mode - setting a trigger for events on the structure to initiate another duplexing rebuild when the structure is in a different state. Once all the queue managers have received the EEPLXESRECOMMENDACTION event for a given structure, recoveries, that were pending so far due to remaining connections to the queue manager as indicated by CSQE149I messeages, start to get processed. As part of the processing, two IXLFORCE calls are issued: the first one to force all the connections of the structure, and the second one to delete the structure itself. After the first IXLFORCE call, the structure is in a state where a duplexing build can take place, and the system goes ahead and initiates the duplexing build straight away. Thus, the queue manager's second IXLFORCE call to delete the structure now fails with IXLRSNCODEREBUILDINPROGRESS. The 00C510A4 abend is issued by the structure task. . DUMP TITLE=xxxx,ABN=5C6-00C510A4,U=SYSOPR,C=R3600.710.CFM -CSQECONN,M=CSQGFRCV,LOC=CSQELPLM.CSQECONN+00003C90
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of WebSphere MQ for z/OS Version 7 * * Release 1 Modification 0. * **************************************************************** * PROBLEM DESCRIPTION: CF structure recovery fails and the * * queue manager abends with 5C6-00C5104A, * * followed by abend 5C6-00C5104D and * * terminates with reason 00C510AB. * **************************************************************** * RECOMMENDATION: * **************************************************************** There are multiple queue managers in a QSG. When the systems loose connectivity to the coupling facility, the connectors for the different queue managers go into a "loss of connectivity" state. If a duplexing build is initiated whilst at least one of the connectors is still in that state, it will fail and wait for the situation to be resolved. As part of recovery processing, two IXLFORCE calls are issued: the first one to force all the connections of the structure, and the second one to delete the structure itself. After the first IXLFORCE call, the structure is in a state where a duplexing build can take place, and the system goes ahead and initiates the duplexing build straight away. Thus, the queue manager's second IXLFORCE call to delete the structure now fails with IXLRSNCODEREBUILDINPROGRESS. The 00C510A4 abend is issued by the structure task.
Problem conclusion
The code was changed to tolerate the IXLRSNCODEREBUILDINPROGRESS reason code from the IXLFORCE call. The queue manager will attempt it several times before ending the CF structure recovery with message CSQE101I, structure in use. Message CSQE159I was added to indicate that the structure is being rebuild by the system and the queue manager goes into a wait. The follow message will be added to the info center at WebSphere MQ > Reference > Messages > Messages and reason codes for z/OS > Messages for WebSphere MQ for z/OS > Coupling Facility manager messages (CSQE...) CSQE149I: csect-name Waiting for other queue managers to disconnect from structure structure-name Explanation The queue manager has lost connectivity to the coupling facility, in which the identified structure is allocated, but cannot delete the structure or initiate a system-managed rebuild because a structure rebuild is currently in progress. System action The queue manager will periodically retry the attempted operation until the structure rebuild is finished. 100Y 101Y 102Y 103Y 104Y 105Y CSQECONN CSQFEDIC CSQFEDIE CSQFEDIF CSQFEDIK CSQFEDIU CSQFMTXC CSQFMTXE CSQFMTXF CSQFMTXK CSQFMTXU
Temporary fix
********* * HIPER * *********
Comments
APAR Information
APAR number
PM92899
Reported component name
WMQ Z/OS V7
Reported component ID
5655R3600
Reported release
100
Status
CLOSED PER
PE
NoPE
HIPER
YesHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2013-07-12
Closed date
2013-07-30
Last modified date
2013-10-04
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
UK96254 UK96255 UK96256 UK96257 UK96258 UK96259
Modules/Macros
CSQECONN CSQFEDIC CSQFEDIE CSQFEDIF CSQFEDIK CSQFEDIU CSQFMTXC CSQFMTXE CSQFMTXF CSQFMTXK CSQFMTXU
Fix information
Fixed component name
WMQ Z/OS V7
Fixed component ID
5655R3600
Applicable component levels
R100 PSY UK96254
UP13/09/27 P F309
R101 PSY UK96255
UP13/09/27 P F309
R102 PSY UK96256
UP13/09/27 P F309
R103 PSY UK96257
UP13/09/27 P F309
R104 PSY UK96258
UP13/09/27 P F309
R105 PSY UK96259
UP13/09/27 P F309
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
[{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SG19M","label":"APARs - z\/OS environment"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1","Edition":"","Line of Business":{"code":"","label":""}}]
Document Information
Modified date:
04 October 2013