IT45074: An MFT agent can receive BFGSS0024E due to receiving a 2127 when performing a PDS Member transfer

APAR status

• Closed as program error.

Error description

• MFT Agent received message BFGSS0024E due to receiving a 2127
  (MQRC_ADAPTER_STORAGE_SHORTAGE) reason code when making an API
  call while an agent is trying to put a message to its state
  queue.
  This reason code is returned to an application when a storage
  allocation in the application address space fails. Review of
  the storage usage of the application address space shows that
  there is a large amount of storage allocated from subpool 1,
  key 8. The allocated data looks to be 31-bit thunked storage
  allocated when a 64-bit application makes an MQ api call.
  The increasing size of the messages which are held on the
  SYSTEM.FTE.STATE.* queues, is believed to be causing the
  storage increase.
  
  z/OS; HEAP64; JAVA; MFT; MQ;OUTOFMEMORY
  

Local fix

• Changing the default HEAP64 KEEP values to FREE.
  
   - The _CEE_RUNOPTS environment variable can be set in the
  .wmqfterc script following documentation
  
  https://www.ibm.com/docs/en/ibm-mq/9.3?topic=zos-environment-var
  iables-mft.
  
      For example you would add to the script:
          export
  _CEE_RUNOPTS="HEAP64(1M,1M,FREE,32768,32768,FREE,4096,4096,FREE)
  "
  
   - The same export specification can be added alongside the
  other exports in ++MQ_HLQ++.SCSQFCMD(BFGPROF)
   - Using the CEEOPTS DD on the fteStartAgent JCL
  

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of IBM MQ Managed File Transfer (MFT),
  running on z/OS platform.
  
  
  Platforms affected:
  z/OS
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  Having an MFT Agent perform a transfer PDS dataset with a number
  of members may result in the agent failing/hanging and
  generating FFDC caused by OutOfMemoryError or
  StackOverflowError.
  
  During the transfer the Agent look to maintain the state of the
  transfer by putting a message on the SYSTEM.FTE.STATE.* queue.
   These messages contain state information about the current
  managed transfer, such as how many items have been transferred.
  It gets updated periodically as the transfer proceeds, and will
  gradually grow as more members are transferred. Each get and put
  of one of these messages will require a message buffer to be
  allocated in below the bar storage by the adapter. These are
  freed back to LE(Language Environment) at the end of the MQ API
  call, but LE(Language Environment) will keep the buffer around
  to be reused for future allocations. The next time the state
  message is got and put, it will be slightly bigger which will
  mean that LE can't reuse the previous buffer and will have to
  acquire a new one. This continues until all the transfers
  complete, or until storage is exhausted in the application
  address space, at which point MQRC_ADAPTER_STORAGE_SHORTAGE will
  be returned to the application.
  
  The output 0.log can also report the following:
  
  22/11/2023 15:04:21:827 GMT! 00000024 FTEStateStore E
  BFGSS0024E: The agent has received a reason code of '2127' from
  the message queue interface (MQI) while accessing
  'SYSTEM.FTE.STATE.AGENT0@QM1'. The agent cannot continue
  processing and will now end.
  
  FFDC:
  
  Filename:
  /usr/mqmfte/mqft/logs/QM1/agents/AGENT1/logs/ffdc/FFDC.FTE.20231
  115063132331.2301065843213282597.log
  Level:      MQMFT 9.2.0.10 V920-PH52699-L230217
  Time:       15/11/2023 06:31:32:331 GMT
  Thread:     44 (CommandHandler)
  Class:      com.ibm.wmqfte.thread.FTEThread
  Method:     uncaughtException
  Probe:      FFDC_001
  Cause:      java.lang.StackOverflowError
  java.lang.StackOverflowError
  .at java.util.regex.Pattern$Curly.match(Pattern.java:4252)
  .at java.util.regex.Pattern$GroupHead.match(Pattern.java:4683)
  .at java.util.regex.Pattern$Branch.match(Pattern.java:4629)
  .at java.util.regex.Pattern$BranchConn.match(Pattern.java:4593)
  .at java.util.regex.Pattern$GroupTail.match(Pattern.java:4742)
  .at java.util.regex.Pattern$Curly.match0(Pattern.java:4304)
  .....
  
  This reason code is returned to an application when a storage
  allocation in the application address space fails. Analysing the
  storage usage of the application address space shows that there
  is a large amount of storage allocated from subpool 1, key 8.
  The allocated data looks to be 31-bit thunked storage allocated
  when a 64-bit application makes an MQ api call.
  

Problem conclusion

• The fteStartAgent script has been updated to include the
  _CEE_RUNOPTS environment variable & HEAP64 option to avoid the
  issue.
  
  Changing the default HEAP64 KEEP values to FREE will mean that
  the buffer will be freed back to the system when the adapter
  frees the storage back to LE. This prevents the problem, but it
  is expected that this will have a performance cost, which the
  customer should be aware of.
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v9.2 LTS   9.2.0.30
  v9.3 LTS   9.3.0.20
  v9.4 LTS   9.4.0.0
  
  The latest available maintenance can be obtained from
  'IBM MQ Recommended Fixes'
  https://www.ibm.com/support/pages/recommended-fixes-ibm-mq
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'IBM MQ
  Planned Maintenance Release Dates'
  https://ibm.biz/mqplannedmaintenance
  
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  MQ BASE V9.2

• Fixed component ID

  5724H7281

Applicable component levels