MFT resource monitoring concepts

An overview of the key concepts of the Managed File Transfer resource monitoring feature.

Resource monitors

You create a resource monitor by using the fteCreateMonitor command, which creates and starts a new resource monitor from the command line. The resource monitor is associated with a Managed File Transfer agent, and is only active when that agent is running. When the monitoring agent stops, so does the resource monitor. If the agent is already running when the resource monitor is created, the resource monitor starts immediately. The monitoring agent must also be the source agent of the task that is initiated by the resource monitor.

Resource monitor names must be unique within their agent. The resource monitor name must be a minimum of one character in length and must not contain asterisk (*), percent (%) or question mark (?) characters. The case in which a resource monitor name is supplied is ignored and the resource monitor name is converted to uppercase. If you try to create a resource monitor with a name that is already present, the request is ignored and the attempt is logged to the resource monitor log topic.

Note: You cannot create a resource monitor with a task definition that contains scheduled transfers.

[MQ 9.2.2 Mar 2021] For Long Term Support, and for Continuous Delivery before IBM® MQ 9.2.2, the only way to stop a resource monitor is by stopping the agent that is running the monitor operation. To restart a resource monitor, you must restart the agent altogether. From IBM MQ 9.2.2, you can start and stop resource monitors without needing to stop or restart an agent. For more information, see Starting and stopping resource monitors.

There is no restriction on the number of resource monitors that can be created on an agent, and all run with the same priority. Consider the implications of overlapping monitored resources, conflicting trigger conditions and how frequently the resources are polled.

Overlapping resource monitors can cause:

Possible contention on the source location/items.
Possible duplicate transfer requests for same source items.
Unexpected errors or failures for transfers due to conflicts of source items.

If multiple monitors scan the same location and can trigger on the same items, you can potentially end up with the problem of two different monitors submitting managed transfer requests for the same item.

Resource monitors look at the contents of resources after every poll interval period. The contents of the resource are compared with the trigger conditions and if those conditions are satisfied, the task associated with the resource monitor is called.

The task is started asynchronously. If there is a condition match, and the task is started, the resource monitor continues to poll for further changes to the resource contents. So for example, if a match occurred because a file called reports.go arrived in a monitored directory, the task would be started once. At the next poll interval, even if the file still exists, the task is not started again. However, if the file is deleted and then placed in the directory again, or the file is updated (such that the last modified date attribute is changed), the next trigger condition check causes the task to be called again.

Before IBM MQ 9.1.5, if a resource monitor performs a poll that takes longer than the polling interval, this means that the next poll starts as soon as the current one finishes with no gap in between, which could have an effect on how quickly resource monitors submit work to an agent. This could cause performance issues if the items that are found during the first poll are still there when the second one takes place.

[MQ 9.2.0 Jul 2020] From IBM MQ 9.1.5, the resource monitor uses the ScheduledExecutorService and initiates the next poll only after the completion of the previous poll plus the configured poll interval time. This means that there will always be a gap in between the poll intervals, rather than having another poll starting straight away after the previous poll if the poll time was longer than the poll interval.

[MQ 9.2.0 Jul 2020] From IBM MQ 9.1.3, if a file has failed to transfer, you can clear the resource monitor history, which allows another transfer request to be submitted without the need to either delete the file and place it in the directory again, or update the file to change its last modified date attribute. Clearing the history is useful, for example, in situations where there is need for the file to be transferred but modifying the file is not possible. For more information, see Clearing resource monitor history.

Resources

Resource monitors in Managed File Transfer can poll the contents of the following two types of resource:

Directories or nested directory structures

A common scenario is to monitor a directory for the presence of a trigger file. An external application might be processing multiple files and placing them in a known source directory. When the application has completed its processing, it indicates that the files are ready to be transferred, or otherwise acted upon, by placing a trigger file into a monitored location. The trigger file can be detected by a Managed File Transfer resource monitor and the transfer of those files from the source directory to another Managed File Transfer Agent is initiated.

By default, the specified directory is monitored. To also examine sub-directories set the recursion level in the fteCreateTransfer command.

Two examples of monitoring a directory are as follows:

Monitor for a trigger file (for example trigger.file) and then transfer a wildcard (for example, *.zip).
Monitor for *.zip and then transfer ${FilePath} (for example, the file that triggered the transfer). For more information about variable substitution, see Customizing MFT resource monitor tasks with variable substitution.

Note: Do not create a monitor that monitors for *.zip, and then transfers *.zip. The monitor tries to start a transfer of *.zip for every .zip file on your system. That is, the monitor generates * number of transfers for *.zip.

For an example of creating a resource monitor to monitor a directory, see Monitoring a directory and using variable substitution.

IBM MQ queues

An example of monitoring a queue is that an external application might be generating messages and placing them on a known queue with the same group ID. When the application has completed putting messages on the queue, it indicates that the group is complete. The complete group of messages can be detected by a Managed File Transfer resource monitor and the transfer of the group of messages from the source queue to a file is initiated. For an example of creating a resource monitor to monitor a queue, see Example: Configuring an MFT resource.

Note: You can specify only one monitor per queue. If you specify more than one monitor to poll an IBM MQ queue, unpredictable behavior occurs.

Monitoring data sets is not supported.

Conditions and trigger conditions

The condition is met when the resource contains a value that matches some other string or pattern. Conditions can be one of the following:

Match on file name (pattern)
No match on file name (pattern)
File size
Match if file size remains the same for a number of polls

File name matching can be expressed as:

Exact string match
Simple wildcard match as described in Using wildcard characters with MFT
Regular expression match

File names can also be excluded from file name matching by using a wildcard or Java regular expression that identifies file names that are never matched.

When a matching file is detected, its last modified time stamp is retained. If subsequent polls detect that the file has been changed, the trigger condition is satisfied again, and the task is started. If the condition is to detect when a file does not exist, if no file in the monitored directory matches the file name pattern, the task is started. If a file is then added to the directory that does match the file name pattern, the task is only started if the file is then deleted.

Tasks

Managed File Transfer supports the following two types of task that you can configure to be started by resource monitors:

File transfer tasks: File transfer tasks are defined in the same way as any other file transfer. A useful way to generate the task XML required by a monitor is to run the fteCreateTransfer command with the -gt parameter. This command generates a task definition as an XML document, including the transfer specification. You then pass the name of the task XML document as the value for the -mt parameter on the fteCreateMonitor command. When the fteCreateMonitor is run, it reads the task XML document. After the fteCreateMonitor is run, any changes that are made to the task XML file are not used by the monitor.; When using a file transfer task, you can select how many trigger conditions are batched into a task. The default is for one trigger condition to start one task. You can run the fteCreateMonitor command with the -bs option to select the number of trigger conditions that are batched together into one task.
Command tasks: Command tasks can run Ant scripts, call executable programs, or run JCL jobs. For more information, see Configuring MFT monitor tasks to start commands and scripts.

Trigger files

You can use the contents of a trigger file in a resource monitor to define a set of files to transfer in a single transfer request. Each time a matching trigger file is detected, its contents are parsed for source file paths and optionally for destination file paths. These file paths are then used to define file items in the task transfer XML file that you specify, which is submitted as single transfer request to the agent. The definition of the resource monitor determines whether trigger content is enabled.

The format of each trigger file is a single file path to transfer on each line of text. The default format for the line is either a single source file path or a source and destination file path separated by a comma.

For more information and examples, see Using a trigger file.

Starting and stopping resource monitors

For Long Term Support, and for Continuous Delivery before IBM MQ 9.2.2, the only way to stop a resource monitor is by stopping the agent that is running the monitor operation. To restart a resource monitor, you must restart the agent altogether. For more information, see Starting an MFT agent and Stopping an MFT agent.

From IBM MQ 9.2.2, you can start and stop resource monitors without needing to stop or restart an agent by using the fteStartMonitor and fteStopMonitor commands. This is useful, for example, in the following situations:

If an agent has multiple resource monitors, and only some of them have encountered errors but the remaining resource monitors are still working fine so you just want to restart the failed resource monitors.
If you want to stop a resource monitor to carry out some maintenance work, or if the resource monitor is not required for certain time and you do not want it to run unnecessarily, thus consuming precious system resources.

[MQ 9.2.2 Mar 2021] For more information, see Starting an MFT resource monitor and Stopping an MFT resource monitor.

Table 1. Behavior of a resource monitor depending on which command is run
Command	Behavior of resource monitor
fteStartMonitor	If the agent is running, then the resource monitor is started if it is currently stopped.
fteStopMonitor	If the agent is running, then the resource monitor is stopped if it is currently started.
fteStartAgent	The resource monitor is started as part of agent start up, regardless of earlier calls to fteStopMonitor.
fteStopAgent	Any resource monitors that are running are stopped.

Backing up and restoring resource monitors

You can back up the resource monitors that you have already defined so that you can re-use them in the future. There are various options that you can use as follows:

Use the fteCreateMonitor command with the -ox parameter to export a resource monitor configuration to an XML file, and with the -ix parameter to restore a resource monitor by importing the resource monitor configuration from an XML file.
Use the fteListMonitors command with the -ox to export the definition for a single resource monitor to an XML file.
Use the fteListMonitors command with the -od to export multiple resource monitor definitions to a specified directory. Each resource monitor definition is saved to separate XML file. You can also use the -od option to export a single resource monitor definition to a specified directory.

For more information, see Backing up and restoring MFT resource monitors.

Resource monitor logging

From IBM MQ 9.1.0, Managed File Transfer includes resource monitor logging. For more information, see Logging MFT resource monitors.