PH54826: MQ Z/OS: CSQE038E CSQEWUOW ADMIN STRUCTURE IS FULL. THERE ARE MANY ESAL CONTROL BLOCKS FOR A CHANNEL.

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• In the reported case, symptoms in the MSTR job log include:
  
  CSQE038E CSQ1 CSQEWUOW Admin structure is full
  
  CSQ2004E CSQ1 CSQ2QCP0 ERROR USING QUEUE Q.OTMA,
  MQRC=2003 (MQRC_BACKED_OUT)
  
  IEA794I SVC DUMP HAS CAPTURED:
  DUMPID=013 REQUESTED BY JOB (CSQ1MSTR)
  DUMP TITLE=CSQ1,ABN=5C6-00F20042,U=SYSOPR  ,C=MQ900.920.XMC -CS
             Q2QCP0,M=CSQGFRCV,LOC=CSQ2IEPL.CSQ2QCP0
  
  CSQ2004E CSQ1 CSQ2QCP1 ERROR USING QUEUE
  Q.OTMA.REPLY, MQRC=2003 (MQRC_BACKED_OUT)
  
  IEA794I SVC DUMP HAS CAPTURED:
  DUMPID=015 REQUESTED BY JOB (CSQ1MSTR)
  DUMP TITLE=CSQ1,ABN=5C6-00F20046,U=SYSOPR  ,C=MQ900.920.XMC -CS
             Q2QCP1,M=CSQGFRCV,LOC=CSQ2IEPL.CSQ2QCP1
  
  Reason code 00F20042 means CSQ2QCP0 got an MQCLOSE error.
  Reason code 00F20046 means CSQ2QCP1 got an MQCLOSE error.
  
  When the queue manager was recycled and the channel ended,
  the CHIN log had this message for the queue the channel was
  putting to:
  
  CSQX038E CSQ1 CSQXRESP Unable to put message to
  CSQ1.INQUIRE.REPLY, MQCC=2 MQRC=2053 (MQRC_Q_FULL)
  
  
  The original dumps did not include the CSQ_ADMIN structure.
  For another instance when list entries were building in
  CSQ_ADMIN, a dump with trace active was captured to include the
  MQ jobs and the structure.
  
  The CSQ_ADMIN structure was filled with thousands of list
  entries for eSALE control blocks with ESAL eyecatchers.  These
  were for receiver channels that were trying to put messages to
  a shared queue that had reached MAXDEPTH and was full.  Based
  on message retry settings of MRTMR(1000), which is the default
  of 1 second, and MRRTY(999999999), where the default is 10),
  the channel was retrying the MQPUT to the queue indefinitely as
  there were problems with the queue getter.
  
  Queue manager trace showed that the channel adapter thread
  created an eSALE (SALE) control block for the serialized MQPUT
  (module CSQESAPP with trace entries CSQESAP1 and IXLWSALE).
  The adapter then tried to write the message to the application
  CF structure, but this failed with ixlRsnCodeListFull,
  which gets mapped to MQRC_Q_FULL. An eTROP control block was
  then deleted, but the deletion of the eSALE was deferred until
  the channel ended. When the MRRTY value is so large, a huge
  number of eSALE blocks builds up.
  
  This problem only occurs when the serialized application (the
  channel in this case) is doing the puts as part of a UOW that
  touches multiple shared queues. The eSALE processing is working
  as designed in this scenario. However, this APAR is being
  raised to investigate improving the processing if possible
  while still maintaining correct shared queue serialization.
  

Local fix

• Set a lower value for MRRTY so that the channel does not retry
  indefinitely, and fix the getting application so that the
  target queue does not fill.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All users of IBM MQ for z/OS Version 9       *
  *                 Release 2 Modification 0 and Release 3       *
  *                 Modification 0.                              *
  ****************************************************************
  * PROBLEM DESCRIPTION: Various symptoms occur due to the MQ    *
  *                      admin structure filling. These can      *
  *                      include                                 *
  *                       - CSQE038E 'Admin Structure Full'      *
  *                       - Applications performing MQ work      *
  *                         involving shared queues failing with *
  *                         return codes such as:                *
  *                         MQRC_BACKED_OUT (MQRC2003),          *
  *                         MQRC_RESOURCE_PROBLEM (MQRC2102)     *
  *                       - IMS bridge tasks abending            *
  *                         5C6-00F20042 or 5C6-00F20046         *
  ****************************************************************
  A serialized application (for example a shared receiver channel)
  attempted to put/get a message on a shared queue, but failed.
  The application determined that the MQRC returned was expected
  to be a transient error, and retried the operation many times.
  
  Each time the application tries the failing operation, a queue
  SALE was written to the admin structure, however unless the
  failing operation was the first/only operation in the unit of
  work, the SALE is not cleared by the failing attempts.
  This results in SALE control blocks with eyecatcher eSAL
  building up in the admin structure for the duration of the
  application's unit of work.
  
  When the application eventually succeeds in putting the message,
  or gives up and aborts the unit of work, these additional SALEs
  are removed from the admin structure, however the accumulated
  SALE blocks can cause the admin structure to fill, resulting
  in a variety of symptoms affecting other users of shared queues
  in the queue sharing group.
  

Problem conclusion

• CSQETHDP is changed to prevent the creation of multiple SALEs
  in the problem scenario.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UI97267 UI97268

Modules/Macros

• CSQETHDP
  

Fix information

• Fixed component name

  IBM MQ Z/OS V9

• Fixed component ID

  5655MQ900

Applicable component levels

• R200 PSY UI97268

     UP24/06/27 P F406 ¢

• R300 PSY UI97267

     UP24/06/27 P F406 ¢

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.