What to do if your MFT agent ABENDS with a java.lang.OutOfMemoryError due to native memory exhaustion

While processing a number of managed transfer requests, such as file-to-file, message-to-file or file-to-message transfers, the agent abnormally ends (ABENDS) reporting a java.lang.OutOfMemoryError, and at the time your total RAM memory was not fully utilized. This exception has been caused by native memory exhaustion.

Diagnosing the problem

When this issue occurs, the affected agent ABENDs and generates two files that provide details on the root cause:

An ABEND file. The name of this file conforms to the naming convention ABEND.FTE.date_timestamp.identifier.log.
On Multiplatforms, the file is written to the MQ_DATA_PATH/mqft/logs/coordination_qmgr_name/agents/agent_name/logs/ffdc directory.

On z/OS®, the file is written to the USS location $BFG_CONFIG/mqft/logs/coordination_qmgr_name/agents/agent_name/logs/ffdc
A Javacore file. The name of this file has the following format: javacore.datestamp.timestamp.pid.identifier.txt
On Multiplatforms, the file is written to the MQ_DATA_PATH/mqft/logs/coordination_qmgr_name/agents/agent_name directory.

On z/OS, the file is written to the USS location $BFG_CONFIG/mqft/logs/coordination_qmgr_name/agents/agent_name directory.

The ABEND and Javacore pair contain information similar to the examples shown below:

Example: Pair one

Abend file


Filename: C:\ProgramData\IBM\MQ\mqft\logs\COORDQM\agents\AGENT1\logs\ffdc\ABEND.FTE.20200109113518046.1764802189777906538.log
Level: p900-005-180821
Time: 09/01/2020 11:35:18:046 GMT
Thread: 96 (TransferSender[414d51204d44424b525030372020202045fbd6532ebfaa02])
Class: com.ibm.wmqfte.thread.FTEThread
Instance: 55b455b4
Method: uncaughtException
Probe: ABEND_001
Cause: java.lang.OutOfMemoryError: native memory exhausted

java.lang.OutOfMemoryError: native memory exhausted
at com.ibm.mq.jmqi.local.internal.base.Native.MQPUT(Native Method)
at com.ibm.mq.jmqi.local.LocalMQ.MQPUT(LocalMQ.java)
at com.ibm.wmqfte.wmqiface.WMQQueueImpl.put(WMQQueueImpl.java)
at com.ibm.wmqfte.wmqiface.WMQQueueImpl.put(WMQQueueImpl.java)
at com.ibm.wmqfte.transfer.impl.TransferSenderRunnable.doTransfer(TransferSenderRunnable.java)
at com.ibm.wmqfte.transfer.impl.TransferSenderRunnable.run(TransferSenderRunnable.java)
at java.lang.Thread.run(Thread.java)
at com.ibm.wmqfte.thread.FTEThread.run(FTEThread.java)

Javacore file


NULL ------------------------------------------------------------------------
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "native memory exhausted" received
1TIDATETIME Date: 2020/01/09 at 11:35:18
1TIFILENAME Javacore filename: C:\ProgramData\IBM\MQ\mqft\logs\COORDQM\agents\AGENT1\javacore.20200109.113518.14148.0002.txt

Example: Pair two

ABEND file


Filename: C:\ProgramData\IBM\MQ\mqft\logs\COORDQM\agents\AGENT1\logs\ffdc\ABEND.FTE.20200109143700286.3177895731698464509.log
Level: p900-005-180821
Time: 09/01/2020 14:37:00:286 GMT
Thread: 918 (AgentStatusPublisher)
Class: com.ibm.wmqfte.thread.FTEThread
Instance: bc10bc1
Method: uncaughtException
Probe: ABEND_001
Cause: java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 12

java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 12
at java.lang.Thread.startImpl(Native Method)
at java.lang.Thread.start(Thread.java)

Javacore file


NULL ------------------------------------------------------------------------
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 12" received
1TIDATETIME Date: 2020/01/09 at 14:37:00
1TIFILENAME Javacore filename: C C:\ProgramData\IBM\MQ\mqft\logs\COORDQM\agents\AGENT1\javacore.20200109.143700.2652.0003.txt

Why this problem occurs

This issue occurs due to exhaustion of the native heap memory on the system where the agent is running.

See How MFT agents use Java heap and native heap memory for more information on the distinctions between Java heap memory and native heap memory.

Avoiding the problem

There are a number of actions that you can take to help reduce the likelihood of an MFT agent stopping due to a


java.lang.OutOfMemoryError

, caused by exhaustion of native memory:

Reduce the size of the Java heap for the JVM running the MFT agent.
The greater the size of the allocated Java heap, the less memory is available to the native heap. Reducing the size of the Java heap used by an agent can free up more memory for the native heap.
By default, the Java heap of an agent is set to 512 MB. If you have changed this to make it a larger value, consider reducing it, and testing with your production-like workload.
- To lower or change the Java heap when running the agent as a normal process:
  Set the BFG_JVM_PROPERTIES environment variable to pass options directory to the JVM. For example, on Windows, to set the maximum heap size to 1024 MB run the following command before using the fteStartAgent command:
```
set BFG_JVM_PROPERTIES="-Xmx1024M"
```
  For more information about how to set Java system properties using the BFG_JVM_PROPERTIES environment variable, see Java system properties for MFT.
- To lower or change the Java heap when running the agent as a Windows service:
  To pass options to the JVM running the agent as a Windows service, modify the agent using the -sj parameter specified on the fteModifyAgent command.
  The following example uses the fteModifyAgent command with the -sj parameter, to set the maximum size of the Java heap for a JVM running a Windows service configured agent:
```
fteModifyAgent.cmd -agentName AGENT1 -s -su user1 -sp passw0rd -sj -Xmx1024M
```
  You can check this has been successfully set, by reviewing the output0.log file of the agent, after the agent has been restarted. In the Start Display Current Environment section, a value of 1024 MB will be reported, as follows:
```
The maximum amount of memory that the Java virtual machine will attempt to use is: '1024'MB
```
Restrict native memory use
Often, java.lang.OutOfMemoryErrors caused by native heap exhaustion are seen if an agent connects to its agent queue manager using the BINDINGS transport. When the agent has been configured to use the BINDINGS transport, the agent calls native methods whenever it needs to communicate with the queue manager.
This means that native memory usage grows as the workload of the agent increases, due to more connections to the queue manager and increased message communication. In this situation, reducing the workload can help. To do this, set the following agent properties to a lower value than the default 25:
- maxSourceTransfers
- maxDestinationTransfers
This reduces the number of concurrent transfers that can occur, and therefore decreases the maximum concurrent workload for the agent.
Configure the agent to use the CLIENT transport when connecting to its agent queue manager. You can do this by setting the following agent properties:
- agentQMgrHost
- agentQMgrPort
- agentQMgrChannel
You can find information about these properties in The MFT agent.properties file topic.
This ensures that all communication between the agent and the queue manager takes place over TCP/IP, rather than native code, which reduces the amount of native memory used by the agent.
Important: Taking this action also decreases performance. By using a TCP/IP connection to the local host, rather than native code, the configuration is not as efficient when the agent requires interactions with the queue manager.