II02744: MSGHASP050 JES2 RESOURCE SHORTAGE CODE= LBUF TPBUF ETC TIPS ON DIAGNOSING THESE RESOURCE SHORTAGES.

APAR status

• Closed as canceled.

Error description

• (THIS INFO IS VALID FOR ALL JES2 RELEASES THRU  HJE4420)
  (HJE4410 AND HJE4420 USE A TOTALLY NEW $CPOOL OPERATION)
  
  *** JES2INFO APAR TO HELP YOU DIAGNOSE MSGHASP050 RESOURCE
  SHORTAGE PROBLEMS.  THE FIRST STEP IN DIAGNOSING WHY YOU ARE
  RECEIVING A RESOURCE SHORTAGE IS TO GET A DUMP OF THE JES2
  ADDRESS SPACE WHEN THE SHORTAGE IS OCCURRING. THIS MAY BE
  OBTAINED EITHER VIA AN MVS DUMP COMMAND OR VIA A SLIP TRAP FOR
  THE PARTICULAR SHORTAGE SET ...FOR EXAMPLE SEE THE HASPMISC CODE
  WHERE THE CMB SHORTAGE IS SET. ALL THE RESOURCE SHORTAGES MAY BE
  EVALUATED IN THE SAME WAY. HERE IS WHAT TO DO WITH YOUR DUMP.
  
  1. FIND THE HCT. IF YOU ARE RUNNING UNDER MVS/XA THIS WILL BE AT
     X'6000' IN THE JES2 ADDRESS SPACE. IF YOU ARE RUNNING MVS/SP,
     THEN YOU WILL HAVE TO USE JES2 REG11.  TYPICALLY IT WILL BE
     AN ADDRESS LIKE 125000, 105000, OR F5000.
  
  (2) FIND THE HCT POINTER TO THE BPM (BUFFER POOL MAP) WHICH
      YOU NEED TO DESCRIBE THE POOL WHICH HAS RUN OUT...
      ACQUIRE A LISTING OF HASPDOC, AND FIND $BFRMAP FOR LBUFS
      AND $TPBFMAP FOR TP BUFFERS..  FOR RELEASE HJE3313,
      $BFRMAP = HCT + 2F8
      $TPBFMAP= HCT + 480
      THE FIRST BUFFER IN THE MAP WILL BE THE 'BPM'......
      $BFRMAP POINTS TO THE LOCAL OR HASP BUFFER POOL MAP...
      IN 2.1.5 IT IS AT X'3A0' INTO THE HCT. IN 2.2.1 IT IS
      AT 348. IF YOU ARE SHORT OF TPBUF'S THEN YOU WILL NEED
      THE $TPBFMAP WHICH IS AT 4C8 FOR 2.1.5 OR  4AC IN THE
      B21 RELEASE. PLEASE NOTE THAT USERMODS IN THE HCT CAN
      ALTER THESE VALUES AND YOU SHOULD LOOK IN YOUR HASPDOC
      TO VERIFY THE OFFSETS.
  
  (3) NOW, GO TO THE MAP (WHERE THE MAP POINTER SHOWS)....
      YOU SHOULD SEE "BPM". THERE ARE ALL THE FIELDS IN THIS
      SMALL AREA WHICH CAN GET YOU VERY CLOSE TO THE PROBLEM:
      AT BPM + C IS A POINTER TO THE FIRST BUFFER. BPMBFR1
      AT BPM + 10 IS THE PTR TO THE LAST BUFFER . BPMLAST
      AT BPM + 18 (2 BYTES) IS THE SIZE OF EACH BUFFER.BPMBFSIZ
      AT BPM + 1C IS A COUNT OF AVAILABLE BUFFERS. BPMBUFCT.
      AT BPM + 20 IS THE MAP. THIS IS A BIT MAP OF ALL THE
         BUFFERS YOU HAVE IN YOUR POOL. FOR EXAMPLE, IF YOU HAVE
         TPBUF SET TO 200, THEN 200 BITS WILL BE REQUIRED. 200
         BITS AT 8 BITS A BYTE = 19 BYTES HEX OR 25 DECIMAL.
         THEREFORE, YOU NEED ONLY WORRY ABOUT THE NUMBER OF BITS
         WHICH ACCOUNT FOR YOUR NUMBER OF BUFFERS. IF THESE BITS
         ARE ZERO, THEN THE BUFFER IS ALLOCATED (IN USE) AND IF
         THEY ARE ONE, THEN THEY ARE FREE (AVAILABLE).
         NOTE THAT THE FIRST BUFFER WILL ALWAYS BE ALLOCATED.
  
  (4) NOW, TAKE YOUR BPMBFSIZ(2 BYTES) AND YOUR FIRST BUFFER
      POINTER, AND KEEP ADDING THE BUFSIZ TO THE FIRST POINTER,
      THEN AGAIN, ETC. WITH A HEX CALCULATOR WITH THE BPMBFSIZ IN
      MEMORY, YOU CAN DO A RECALL, ADD, RECALL ADD, ETC AND STEP
      RIGHT THRU THE POOL. WITH A HIGHLIGHTER PEN, YOU CAN MARK
      YOUR DUMP WITH A MARK ON THE FIRST BYTE OF EACH BUFFER THRU
      STORAGE UNTIL YOU GET UP TO THE LAST BUFFER ADDRESS. NOW YOU
      HAVE DONE THE EASY PART.
      NOTE: BUFFERS MAY START ON A  PAGE BOUNDARY SO EACH TIME
      YOU ADD THE BUFSIZE TO THE START OF THE LAST BUFFER YOU MAY
      NEED TO JUMP UP TO THE NEXT PAGE BOUNDARY TO GET TO THE NEXT
      BUFFER.
  
  (5) EYEBALL THRU THESE BUFFERS LOOKING FOR REPETITIVE PATTERNS..
      AGAIN, THE HIGHLITER CAN BE A FRIEND. KNOWING THAT THE FIRST
      HEX 68 BYTES OR SO OF THE BUFFER MAY BE EITHER AN IOB (OFTEN
      STARTS WITH A 420000, OR THE WORD BUF IN 2.1.5) OR AN RPL
      (EITHER BUF FOR 1.3.6 OR 2.1.5 OR 00202270 OR 00202370 OR
      WHATEVER IN THE RPL), STEP DOWN THRU THESE BUFFERS MAKING
      NOTES WHENEVER YOU FIND DUPLICATE INFORMATION. THIS WILL
      TAKE SOME TIME AND YOU MAY NOT FIND ANY DUPLICATED INFO.
      (NOTE: ALL CURRENT RELEASES OF JES2 THRU HJE4420+ HAVE
       A STANDARD BUFFER PREFIX THAT BEGINS WITH THE 'BUF'
       CHARACTER STRING..  THE ACTUAL BUFFER BEGINS AT BUF + X'18
       SO, IF THE BUFFER ADDRESS WAS  C5000, THE ACTUAL BUFFER
       WOULD BEGIN AT  C5018 ...  IF THE BUFFER REPRESENTS
       AN MVS IOB, THEN THE FIRST NIBBLE OF BYTE 18 WILL BE 4
       42 41 44 46 48 ....  (IOBFLAG1) ..... IF THE BUFFER
       REPRESENTS AN MVS/RPL, THEN, BYTE 18 WILL ALWAYS BE X'00'
       00201700  00202200 00202300 00201F00 ... (RPLID=00)...
       BYTES 18 19 1A 1B SHOULD NEVER BE 00000000 ON AN ACTIVE
       BUFFER..... THE BUFFER IS ACTIVE IF THE ASSOCIATED BPM BIT
       IS OFF......)
  
  (6) WHEN YOU FIND MANY BUFFERS SHOWING THE SAME KINDS OF
      INFO, THEN IT IS TIME TO FIND OUT WHAT PROGRAM, OR USERMOD,
      OR EXIT, OR WHATEVER CREATED THESE DUPLICATE ENTRIES AND
      IN TURN FAILED TO FREBUF THEM.
  
  (7) BUFFER MANAGEMENT:
        In JES2 releases HJE4410 and HJE4420, the buffer pool
        has been replaced by a $CPOOL (CELL POOL).  Now JES2 V4
        uses MVS CELL pools to manage buffer pools.  $CPOOL can
        be issued in the JES2 main task to obtain and release
        these buffers.  They are:
          * BSC  (BSC teleprocessing buffers)
          * NHB  (8K NETWORK HEADER BUFFER STORAGE)
          * CB   (Control block buffers)
          * HASP (local data buffer)
          * PAGE (4096-byte (1K) buffer)
          * PP   (Print/punch buffer)
          * VTAM (SNA teleprocessing buffer)
  
  With IPCS, one can easily locate these buffers in the dump
  following these steps below:
  a) Select option 2 (ANALYSIS) under IPCS PRIMARY OPTION MENU
  b) Select option 6 (MVS COMPONENT DATA) under IPCS MVS ANALYSIS
       OF DUMP CONTENTS
  c) type: S JES2 on command line under IPCS MVS DUMP COMPONENT
       DATA ANALYSIS
  d) Select 'JES2 CONTROL BLOCKS' under JES2 COMPONENT DATA
       ANALYSIS
  e) Select 'JES2 CELLPOOL' under JES2 CONTROL BLOCK LIST
  .
  At this point, you will see a list of cell pool control
  blocks (such as $CPNX, $CPMR and $CPEBE).  For EXAMPLE,
  Let's find the VTAM tp buffer pool..
  
  $CPNX + '34 = CPIVTAM ($CPMR)
  $CPMR + '30 = CPMCPEBE (CPEB)
  CPEB  + '14 = CPEXXADD (FIRST BUFFER IN POOL)
  
  For control block reference in HASPDOC, see these:
  $CPNX is mapped by $CPINDEX in HASPDOC
  $CPMR is mapped by $CPMASTR in HASPDOC
  CPEB  is mapped by $CPEBE   in HASPDOC
  ----------------------------------------------------------------
  ================================================================
  =====A D D I T I O N A L  D I A G N O S T I C  H I N T S: ======
  ================================================================
  
  A. IF THE SHORTAGE CAME ON QUICKLY, THEN IT IS POSSIBLE THAT
     SOME FUNCTION WENT INTO A LOOP GETTING BUFFERS AND ATE THEM
     UP ALL AT ONCE. FREQUENTLY THIS CAN ALSO HANG THE SYSTEM. IF
     THIS IS THE CASE YOU USUALLY WILL FIND AN IDENTICAL PATTERN
     IN THE BUFFER POOL AND CAN GO DIRECTLY TO THE FUNCTION THAT
     ATE THE BUFFERS. FOR EXAMPLE, A NEW USER EXIT. IT IS USUALLY
     VALUABLE AT THIS TIME TO LOOK AT THE SYSLOG TO SEE IF THERE
     WERE ANY PARTICULAR STRANGE MESSAGES OR COMMANDS ISSUED OR
     INVOCATIONS OF SPECIAL PROCESSING WHICH IMMEDIATELY PRECEED
     THE FIRST ISSUANCE OF THE $HASP050 RESOURCE SHORTAGE MESSAGE.
  
  B. IN SOME CASES A $DBUFDEF COMMAND ISSUED IMMEDIATELY AFTER THE
     APPEARANCE OF MSGHASP050 WILL RESULT IN MSGHASP840 SHOWING A
     NUMBER OF FREE BUFFERS (BUFFREE=) THAT IS GREATER THAN THE
     NUMBER OF BUFFERS DEFINED.  (THIS COMMAND IS ONLY VALID IN
     HJE1367, HJE2215 AND LATER).  IF THIS HAPPENS, THEN THE
     MSGHASP050 DID NOT REALLY REPRESENT A BUFFER SHORTAGE AT
     ALL (THAT IS, DO NOT BOTHER LOOKING FOR A PROCESSOR THAT HAS
     FREED HUNDREDS OF BUFFERS IN A SECOND).  THE JES2 RESOURCE
     MANAGER PROCESSOR SUBTRACTS THE FREE BUFFERS FROM THE TOTAL
     DEFINED, CREATING A NEGATIVE NUMBER WHICH IS USED IN LATER
     CALCULATIONS.  THE ERRONEOUS RESULTS APPEAR TO BE A VERY
     HIGH (POSITIVE) NUMBER OF BUFFERS USED, WHEREAS IT IS
     REALLY A HIGH NEGATIVE NUMBER.  THE LOGIC OF THE
     CALCULATIONS IS NOT IN ERROR: THE NUMBER OF BUFFERS
     FREE SHOULD NEVER EXCEED THE NUMBER DEFINED.  THEREFORE,
     THIS PHENOMENON MEANS THAT SOME FUNCTION HAS BEEN
     FREEING BUFFERS THAT IT DID NOT ACTUALLY OWN, OR FREEING
     THE SAME BUFFERS MORE THAN ONCE.  THIS IS USUALLY CAUSED
     BY A USER MODIFICATION OR EXIT, SO PLEASE CHECK THESE
     CAREFULLY BEFORE REQUESTING SUPPORT CENTER ASSISTANCE.
  
  C. IF THE SHORTAGE CAME ON SLOWLY, BUT NEVER WAS ALLEVIATED,
     THEN IT IS POSSIBLE THAT SOME FUNCTION IS NOT LOOPING, IS
     DOING HIS MAIN PROCESSING FINE, BUT HAS FORGOTTEN TO DO THE
     $FREBUF TO PUT THE BUFFER BACK ON THE FREE LIST AGAIN. IN
     THIS CASE, YOU WILL ALSO SEE THE REPETITIVE PATTERN, ALTHOUGH
     WITH DIFFERENT DATA. YOU SHOULD AGAIN INVESTIGATE THE JES2 OR
     USER EXIT OR MOD FUNCTION WHICH CREATED THESE BLOCKS. IF IT
     LOOKS LIKE A JES2 BLOCK, THEN THE JES2 PLM (LOGIC MANUAL) CAN
     HELP YOU FIGURE OUT WHAT MODULE CREATES THESE BLOCKS.
  
  D. IF THE SHORTAGE COMES ON SLOWLY, GOES AWAY SOMETIMES, AND
     APPEARS TO BE QUITE INTERMITTENT.... THEN YOU MAY NOT SEE
     ANY REPETITIVE PATTERN OF BUFFERS USED. IT MAY BE THAT IN
      FACT YOU DO HAVE A VALID SHORTAGE AND THAT YOU SHORT
      GUESSED THE SYSTEM REQUIREMENTS AND SHOULD ADD MORE BUFFERS
      IN YOUR JES2 PARAMETERS. YOU MAY WISH TO CONSULT WITH YOUR
      SE TO HELP YOU DECIDE HOW MANY ARE NEEDED FOR YOUR SYSTEM'S
      INDIVIDUAL NEEDS. IN THIS CASE, THE SHORTAGE MAY OCCUR
      WHEN THE SYSTEM IS GOING THRU HEAVY PROCESSING OR HEAVY
      LOAD.
  
  E.  ONE LAST THOUGHT, . . . . IF THE BUFFER COUNT SHOWS NONE
      AVAILABLE, BUT YOUR BPMMAP (AT BPM + 20) SHOWS LOTS OF EM
      AVAILABLE (1'S MEAN FREE O'S MEAN ALLOCATED), A POSSIBLE
      OVERLAY OF THE FREE COUNT OR OVERLAY OF A BUFFER CHAIN
      MAY HAVE OCCURRED. THIS MOST OFTEN OCCURS WHEN A USERMOD
      OR USEREXIT HAS OVERRUN ITS DATA BOUNDARY AND HAS WRITTEN
      DATA INTO THE NEXT BUFFER (WHEN IT WAS STILL A "LIVE"
      BUFFER.
  
  ==============================================================
  =======   Additional Hints for 100% CMB Utilization    =======
  ==============================================================
    The BUFNUM parameter on the CONDEF initialization statement
  specifies the number of console message buffers to be provided
  to JES2.  The number is used to define the maximum number of
  buffers that JES2 will obtain from extended private storage.
  The same number is used as the maximum number of buffers JES2
  can dynamically get from ECSA for command processing.  For
  example, if you specify BUFNUM=52, then the maximum number of
  commands that can be queued for JES2 is 52.
    The $D CONDEF command can be used to display the number of
  free and inuse message buffers.  However, it does not display
  the status of the command buffers.  (There are not 'free'
  ECSA command buffers; when the command has been processed, the
  CMB is freemained.)
    If a situation of 100% utilization of CMBs is reached, a
  hotstart of JES2 will reinitialize the private area buffer
  pool, and will free queued command buffers.  It is recommended
  to use the command $PJES2,ABEND,FORCE rather than $PJES2,ABEND.
  The latter command is processed by the JES2 command processor,
  and if there is a CMB shortage, JES2 may not be able to handle
  new commands.  The FORCE option is recognized before being
  queued to JES2, and results in a CALLRTM being issued from
  the Console address space.  JES2 will abend with an abend02D.
  Jobs in execution will continue to execute.  JES2 can be
  restarted, and the CMB inuse values will be reset.
    For analysis of CMB shortage problems, a dump of the JES2
  address space, including CSA and the private region must be
  obtained.  This can be obtained by an MVS "DUMP COMM= " command
  entered before the $PJES2,ABEND,FORCE, or it can be obtained
  by replying "DUMP" to the msgHASP098 "ENTER TERMINATION
  OPTION".
    Within the dump, CMBs can be found on the following chains:
  In the HCCT:
      CCTCOMMQ   (These are ECSA command buffers queued for JES2)
  In the HCT:
      $BUSYQUE (These may be either ECSA or private)
      $BUSYRQ  (These will be private)
      $COMMQUE  (These may be ECSA or private)
      $CONWQ   (These will be private)
      $DOMQUE       "  "   "   "
      $DOMQUEA      "  "   "   "
  ================================================================
  BEFORE CALLING LEVEL 1 TO HAVE A SEARCH DONE, OR BEFORE SCANNING
  YOUR INFO MVS DATABASE, EVALUATING THE BUFFER POOL FOR WHICHEVER
  BUFFERS ARE SHORT IS ADVISABLE BECAUSE OTHERWISE YOU WILL HAVE
  NO IDEA WHAT CAUSED YOUR SHORTAGE. IF THE BUFFERS LOOK LIKE
  THEY HAVE BEEN CREATED BY JES2, THEN DEFINITELY DO YOUR SEARCH
  OR CALL LEVEL1 FOR A GO AT THE DATABASE. IF HOWEVER YOUR
  BUFFERS SEEM TO BE CAUSED BY A USEREXIT, THEN A GOOD TEST TO
  PROVE THE GUILT OR INNOCENSE OF A MOD OR EXIT IS TO RUN FOR
  A WHILE WITH THE MOD OR THE EXIT DISABLED BEFORE MAKING ANY
  ASSUMPTIONS. SOME OTHER AREAS YOU MIGHT WANT TO INVESTIGATE...
  OEM PRODUCTS WHICH MODIFY JES2. $RESHORT BIT S (HCT + 8A6 APPROX
  IN RELEASES OF JES2 ***PRIOR**** TO 1.3.5/2.1.5.)  ALSO YOU MAY
  WISH TO READ THE $GETBUF LOGIC AND $FREBUF LOGIC IN HASPNUC.
  THE PLM ALSO GIVES VALUABLE INFORMATION ON THIS PROCESSING.
  SEE APAR OY34023 FOR BUFFER POOL TUNING NOTE, RECOMMENDATION
  IS VALID FOR ALL RELEASES.
  

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

• INFORMATION APAR.
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels