Configuring the collection of message flow statistics by using a .yaml configuration file

Configure the collection of accounting and statistics data for one or more active message flows by setting the statistics properties in the node.conf.yaml or server.conf.yaml configuration file for your integration node or integration server.

Before you begin

Read the topics Message flow statistics and accounting data and Configuring the collection of message flow accounting and statistics data.
Configure an integration server, by following the instructions described in Configuring an integration server by modifying the server.conf.yaml file.
Create your message flow and deploy it to your integration server. For more information, see Creating a message flow and Deploying integration solutions to a production environment.
Ensure that the publication of events is enabled, and the pub/sub broker is correctly configured. For an integration node, the default configuration enables publication of events by using the built-in MQTT pub/sub broker. If IBM MQ is installed and a default IBM MQ queue manager is specified on the integration node or independent integration server, the IBM MQ pub/sub broker is used for the publication of events. For more information, see Configuring the publication of event messages and Configuring the built-in MQTT pub/sub broker.

About this task

You can configure message flow accounting and statistics data collection by modifying properties in the server.conf.yaml or node.conf.yaml files. For example, you can start collecting data for a new message flow that you deploy to an integration server for which you are already collecting data. When you restart the integration server, the new settings that you define in the .yaml configuration file come into effect.

You can configure the collection of snapshot data, archive data, or both, by setting properties in the Statistics section of the configuration file, as shown in the following example:

Statistics:
  Resource:
    reportingOn: true
    outputFormat: 'csvFile'
    filePath: ''
    numberOfFiles: 4
    sizeOfFile: 25
    averages: true
  Snapshot:
    publicationOn: 'active'      # choose 1 of : active|inactive, explictly set 'active'. If unset will default to 'inactive'
    #accountingOrigin: 'none'    # choose 1 of : none|basic, default none
    nodeDataLevel: 'basic'       # choose 1 of : none|basic|advanced, explictly set 'basic'. If unset will default to 'none'
    outputFormat: 'json'         # choose comma separated list of one or more of : csv,json,xml,usertrace. Explictly set to 'json' for WebUI. If unset will default to ''
    #threadDataLevel: 'none'     # choose 1 of : none|basic. If unset will default to 'none'
  Archive:
    #archivalOn: 'inactive'      # choose 1 of : active|inactive, default inactive
                                 # Also set Events.OperationalEvents.MQ|MQTT for outputFormat xml to be published to MQ/MQTT
    #accountingOrigin: 'none'    # choose 1 of : none|basic
    #majorInterval: 60           # Sets the interval in minutes at which archive statistics are published
    #nodeDataLevel: 'none'       # choose 1 of : none|basic|advanced
    #outputFormat: 'usertrace'   # comma separated list of : csv,xml,usertrace
    #threadDataLevel: 'none'     # choose 1 of : none|basic
  csv:
    filePath: ''
    numberOfFiles: 4
    sizeOfFile: 25
    averages: true

Alternatively, you can configure the collection of message flow accounting and statistics data dynamically, by running the mqsichangeflowstats command. The settings that are specified by the command take effect immediately, without restarting the integration server. However, it is not possible to alter the csv section by using the mqsichangeflowstats command or any other command. You can alter the csv section only if you update the .conf.yaml file. For more information, see Modifying message flow accounting and statistics data collection settings by using the mqsichangeflowstats command.

Procedure

Complete the following steps to configure the collection of accounting and statistics data for message flows by uncommenting and setting the statistics properties in the configuration file for your integration node or server. This configuration is then applied to all message flows when they are deployed or when the integration server is restarted:

Open the appropriate node.conf.yaml or server.conf.yaml configuration file by using a YAML editor.

You can edit the file by using the built-in YAML editor that is provided in the IBM App Connect Enterprise Toolkit, either by double-clicking the file in the Application Development view or by right-clicking the file and selecting Open with > YAML editor. If you choose to edit the file by using a plain text editor, ensure that you do not include any tab characters (which are not valid in YAML) and use a YAML validation tool to validate the contents of your file.

You can start collection of snapshot data, archive data, or both, by setting properties in the Statistics section of the configuration file (node.conf.yaml or server.conf.yaml):

To turn on the collection of snapshot statistics data, uncomment and set the following properties in the appropriate .yaml file:
1. Enable the collection of snapshot statistics data by setting the publicationOn property to active. This setting takes effect when the integration server is restarted. However, it can be changed dynamically (without restarting the integration server) by using the mqsichangeflowstats command.
  
  When a node.conf.yaml or server.conf.yaml file is created, the publicationOn property is explicitly set to active by default, and snapshot statistics for the integration node or server are published to the web user interface. If the publicationOn property is unset (with a value of ''), the publication of snapshot statistics is turned off.
  
  In node.conf.yaml and server.conf.yaml files that were created prior to IBM App Connect Enterprise Version 11.0.0.8, the publicationOn property was set to inactive by default, so the publication of statistics was turned off. To enable the publication of snapshot statistics for those integration nodes or servers, edit the relevant .conf.yaml file and set the publicationOn property to active.
2. Decide the target destination and specify the appropriate format in the outputFormat property.
  If you want to display statistics in the web user interface, you must specify json as one of the values in the outputFormat property. The web user interface can display message flow statistics and accounting data for independent integration servers and integration servers that are managed by an integration node. The data that it receives must be in JSON format.
  
  When a node.conf.yaml or server.conf.yaml file is created, the outputFormat property is explicitly set to json by default. If this property is unset, the output format is set to ''.
  
  You can choose multiple destinations by specifying multiple comma-separated values:
  - usertrace
    The data is sent to the user trace log, which must be enabled before you can view the data.
  - csv
    The data is sent to a set of cyclic files in the work directory of the integration node or integration server.
    For independent integration servers, by default the data to files are written to the following location:
    serverWorkDirectory/config/common/resourceStats/ResourceStats_integration_server_serverName_resource.txt.n
    where:
    
    serverWorkDirectory is the value of the --work-dir parameter.
    
    serverName is the value of the --name parameter.
    
    resource is the name of the reported resource, such as JVM or JDBCConnectionPools.
    Integration node servers by default write the data to files at the following location:
    MQSI_WORKPATH/common/stats/
    where MQSI_WORKPATH is the common work path. For example, C:\ProgramData\IBM\MQSI\ (Windows), or /var/mqsi/ (Linux® and UNIX systems).
  - json
    The data is sent to the IBM App Connect Enterprise web user interface. If IBM MQ and MQTT are enabled, data is also sent there. When a node.conf.yaml or server.conf.yaml file is created, the outputFormat property is explicitly set to json by default.
  - xml
    If IBM MQ and MQTT are enabled, the data is also sent there.
3. Optional: Decide whether you want to associate data collection with a particular accounting origin. To report an accounting origin for the data, set the accountingOrigin property to basic.
  Accounting and statistics data can be accumulated and reported about an identifier that is associated with a message in a message flow. This identifier is the accounting origin, which provides a method of producing individual accounting and statistics data for multiple accounting origins that generate input to message flows. The default for this property is none (no support). For more information, see Setting message flow accounting and statistics accounting origin.
4. Optional: Decide whether you want to collect thread-related statistics. To include thread-related data, set the threadDataLevel property to basic. The default value for this property is none.
5. Optional: Decide whether you want to collect node-related statistics. If you do, you can also collect information about terminals for the nodes. To include node-related data, set the nodeDataLevel property to basic, or to include node-related and terminal-related data, specify advanced.
When the collection of snapshot statistics data is turned on, an up-to-date snapshot of information is collected every 20 seconds. You can view the snapshot data in the web user interface. For more information, see Viewing message flow statistics and accounting data.
To configure the collection of archive statistics data, uncomment and set the following properties in the appropriate .yaml file:
1. Set the archivalOn property to active.
  This setting takes effect when the integration server is restarted or when the message flow is deployed. However, it can be changed dynamically (without restarting the integration server) by using the mqsichangeflowstats command.
2. Set the outputFormat property to one or more of the following values (separated by commas and enclosed by single quotation marks):
  - usertrace
  - csv
  - xml
  By default, the publication of archive statistics data is turned off. If you start collection (by setting the archivalOn property to active), the default output format is usertrace.
3. Optional: Specify the interval (in minutes) at which archive statistics are published, by setting the majorInterval property. The default is 60 minutes.
4. Optional: To report an accounting origin for the data, set the accountingOrigin property to basic.
  Accounting and statistics data can be accumulated and reported about an identifier that is associated with a message in a message flow. This identifier is the accounting origin, which provides a method of producing individual accounting and statistics data for multiple accounting origins that generate input to message flows. The default for this property is none (no support). For more information, see Message flow accounting and statistics accounting origin.
5. Optional: To include thread-related data in the statistics, set the threadDataLevel property to basic. The default value for this property is none.
6. Optional: To include node-related data in the statistics, set the nodeDataLevel property to basic, or to include node-related and terminal-related data in the statistics, specify advanced.
To configure the settings for the csv files, uncomment and set the following properties in the appropriate .yaml file:
1. Set the filePath property to the absolute file path of where the statistics files are written.
2. Set the numberOfFiles property to the number of rotatable log files that are produced (default is 4).
3. Set the sizeOfFile property to the size of each rotatable log file in MB (default is 25).
4. Set the averages property to true or false to indicate whether averages are written to the file.
Restart the integration node or integration server for the changes to take effect. Changes to the server.conf.yaml and node.conf.yaml files take effect only when the modified integration node or server is restarted.
The state of statistics collection that is configured in the .yaml file is applied to all flows that are deployed when the integration server starts. It is also applied to any additional flows that are deployed after the integration server starts.