Monitoring a log file
Use the following procedure to collect data from a log file:
- On the Agent Initial Data Source page (Figure 124) or the Data Source Location page, click Logged Data in the Monitoring Data Categories area.
- In the Data Sources area, click A Log File.
- Click Next.
- On the Log File Information page (Figure 125),
type the name of the log file you want to monitor in the Log File Information area.
The file name must be fully qualified. Optionally, part of the log file name can come from a runtime configuration property. To create a log file name, click Insert Configuration Property and select a configuration property (Figure 246). The file can also be a dynamic file name. For more information, see Appendix I. Dynamic file name support.
- In the Field Identification area, click
one of the following options:
- Fixed number of characters
- When selected, limits the number of characters.
With this option, each attribute is assigned the maximum number of characters it can hold from the log file. For example, if there are three attributes A, B, and C (in that order), and each attribute is a String of maximum length 20, then the first 20 bytes of the log record go into A, the second 20 into B, and the next 20 into C.
- Tab separator
- When selected, you can use tab separators.
- Space separator
- When selected, multiple concurrent spaces can be used as a single separator.
- Separator Text
- When selected, type in separator text.
- Begin and End Text
- When selected, type in both Begin and End text.
- XML in element
- When selected, type the name of the XML element to use as the
record, or click Browse to define the element.
If you clicked Browse, the XML Browser window is displayed (Figure 126). If you use the browse functionality, the Agent Builder identifies all of the possible attributes of the record by looking at the child tags and their attributes.
Note:Unless you click Advanced and fill out the information in that window, the information that you fill out here assumes the following:- Only one log file is monitored at a time.
- Each line of the log file contains all the fields necessary to fill the attributes to be defined.
- (Optional) Click Advanced on the Log File Information
page to do the following using the Advanced Data Source Properties
page (Figure 127):
- Monitor more than one file at a time, or monitor files with different names on different operating systems or monitor file names with names that match regular expressions.
- Draw a set of fields from more than one line in the log file.
- Choose event filtering and summarization options.
- Produce output summary information. This summary produces an additional attribute group at each interval. For more information about this attribute group, see Log File Summary. This function has been deprecated by the options available in the Event Information tab.
- To monitor more than one log file, click Add and
type the name. If more than one file is listed, a unique label must
be entered for each file. The label can be displayed as an attribute
to indicate which file generated the record. It must not contain spaces.
(Optional) To select the operating systems on which each log file is to be monitored, do the following:
- Click in the Operating systems column for the log file.
- Click Edit.
- In the Operating Systems window, select the operating systems.
- Click OK to save your changes and return to the Advanced Data Source Properties page.
- Select the File names match
regular expression check box if the file name you are providing
is a regular expression that will be used to find the file instead
of being a file name. See Appendix G. ICU regular expressions for
more information.
If you do not check this box, then the name needs to be an actual file name, or it must be a pattern following the rules for file-name patterns described in Dynamic file name syntax.
- In the When multiple files match drop-down
list, select one of the following options:
- The file with the highest numerical value in the file name
- The biggest file
- The most recently-updated file
- The most recently-created file
- All files that match
Note:When you select All files that match, the agent identifies all files in the directory that match the dynamic file-name pattern, and monitors updates to all of the files in parallel. Data from all files is intermingled during the data collection process. The best practice is to add an attribute by selecting Log file name in Record Field Information to correlate log messages to the log files that contain the log messages. Ensure that all files that match the dynamic file-name pattern can be split into attributes in a consistent manner. If the log files selected cannot be coherently parsed, then the best practice is to select Entire record in Record Field Information to define a single attribute. See Step 8 for more information about specifying Record Field Information for attributes. - Choose how the file is processed. With Process all records when the file is sampled, you
can process all records in the entire file every time a situation
runs against the data source or a sample is taken. The same records
are reported every time unless they are removed from the file. With
this selection, event data is not produced when new records are written
to the file. A file must have at least two records to be processed
correctly. With Process new records appended to the
file, you can process new records appended to the file while
the agent is running. An event record is produced for every record
added to the file. If the file is replaced (the first record changes
in any way), the entire file is processed and an event is produced
for each record in the file.
Note:If you are appending records to an XML log file, the append records must contain a complete set of elements defined within the XML element you selected as Field Identification.
- If you chose to process new records appended to the file, you can also choose how new records are detected. With Detect new records when record count increases, new records can be detected when the number of records in the file increases, whether or not the size of the file changes. This is useful when an entire log file is pre-allocated before any records are written to the file. Note that this can be selected for files that are not pre-allocated, but it is less efficient than monitoring the size of the file. With Detect new records when the file size increases you can determine when a new entry has been appended to a file in the typical way. There might be a brief delay recognizing that a monitored file has been replaced.
- If you chose to detect new records based on the size of the file, you can also choose how to process a file that exists when the monitoring agent starts. Ignore existing records disables event production for any record in the file at the time agent starts. Process ___ existing records from the file specifies production of an event for a fixed number of records from the end of the file at the time the agent starts. Process records not previously processed by the agent specifies for restart data to be maintained by the monitoring agent so the agent knows which records were processed the last time it ran. Events are produced for any records appended to the file since the last time the agent was running. Note that this involves a little extra overhead each time a record is added to the file.
- If you chose to process records not previously processed by the agent, you can choose what to do when the agent starts and it appears that the existing file has been replaced. Process all records if the file has been replaced specifies the production of events for all records in the file if the current information about the monitored file and the information stored in the restart data do not match. Examples of mismatches include: the file name or file creation time has changed, the file size has decreased, the file last modification time is earlier than it was. Do not process records if the file has been replaced disables processing of any existing records in the file if the current information about the monitored file and the information stored in the restart data do not match.
- Click the Record Identification tab
(Figure 128) to interpret multiple lines in the log file
as a single logical record.
Note:The Record Identification tab does not display if you select XML in element as the field identification on the Log File Information page.
- Single line interprets each line as a single logical record.
- Separator line you can enter a sequence
of characters that identifies a line that separates one record from
another.
Note:The separator line is not part of the previous or the next record.
- Rule enables you to identify a maximum number
of lines that make up a record and optionally a sequence of characters
that indicate the beginning or end of a record. With Rule, you can specify the following properties:
- Maximum non-blank line defines the maximum number of non blank lines that can be processed by a rule.
- Type of rule can either be No text comparison (the Maximum lines per record indicates a single logical record), Identify the beginning of record (which marks the start of the single logical record), or Identify the end of record (which marks the end of the single logical record).
- Offset specifies the location within a line where the Comparison String must occur.
- Comparison Test can either be Equals, requiring a character sequence match at the specific offset, or Does not equal, indicating a particular character sequence does not occur at the specific offset.
- Comparison String defines the character sequence to be compared.
- Regular Expression enables you to identify
a pattern used to indicate the beginning or end of a record. By using Regular Expression, you can specify the following
properties:
- Comparison String defines the character
sequence to be matched.
--OR--
- Beginning or end of record:
- Identify the beginning of record marks the start of the single logical record.
- Identify the end of record marks the end of the single logical record.
- Comparison String defines the character
sequence to be matched.
- If you did select Process all records when the file is sampled in Step 6d, click the Filter Expression tab (Figure 129) to filter the data that will be returned as rows based on the values of one or more attributes and/or configuration variables. If you selected Process new records appended to the file in Step 6d you cannot create a filter expression. For more information about filtering data from an attribute group, see Filtering attribute groups
- If you selected Process new records
appended to the file in Step 6d,
click the Event Information tab Figure 130 to select event filtering and summarization
options. For more information, see (Event filtering and summarization).
Note:The Summary tab may be present if the agent was created with an earlier version of Agent Builder. The summary tab has now been deprecated by the Event Information tab
- On the Log File Information page (Figure 125), after you select the options for the log source, you can click Test Log File Settings to start and test the agent. For more information about testing, see Testing.
- If you did not use the test
functionality in Step (7) and
you typed the log file name in the Log File Information area
of the Log File Information page (Figure 125),
do the following:
- Click Next to display the Attribute Information page and define the first attribute in the attribute group.
- Specify the information on the Attribute Information page, and click Finish.
Note:When a Log File attribute group is added to an agent that is at the default minimum Tivoli® Monitoring version of 6.2.1 or later, a Log File Status attribute group is automatically included. For more information about the Log File Status attribute group, see Log File Status attribute group.Along with the fields that are applicable to all of the data sources (Table 6), the Attribute Information page for the Log File data source has some additional fields in the Record Field Information area.
The Record Field Information fields are:
- Next field
- Shows the next field after parsing, using the delimiters from the attribute group (or special delimiters for this attribute from the Advanced dialog).
- Remainder of record
- Shows the rest of the record after previous attributes have been parsed. This is the last attribute, except for possibly the log file name or log file label.
- Entire record
- Shows the entire record, which can be the only attribute, except for possibly the log file name or log file label.
- Log file name
- Shows the name of the log file.
- Log file label
- Shows the label assigned to the file on the advanced panel (above).
Note:Use the Derived Attribute Details tab only if you want a derived attribute, and not an attribute directly from the log file. - If you click Advanced in the Record Field Information area, the Advanced Log
File Attribute Information page (Figure 132)
is displayed.
- In the Attribute Filters section, specify
the criteria for data to be included or excluded. Filtering attributes
can enhance the performance of your solution by reducing the amount
of data processed. Click one or more of the attribute filters:
- Inclusive indicates that the attribute filter set is an acceptance filter, meaning that if the filter set succeeds, the record passes the filter and is output.
- Exclusive indicates that the attribute filter set is a rejection filter, meaning that if the attribute filter set succeeds, the record is rejected and is not output.
- Match all filters indicates that all filters defined to the filter set must match the attribute record in order for the filter set to succeed.
- Match any Filter indicates that if any of the filters defined to the filter set match the attribute record, the filter set succeeds.
- Use Add, Edit,
and Remove to define the individual filters
for an attribute filter set.
To add a filter, perform the following steps:
- Click Add, and complete the options in the
Add Filter window (Figure 133) as follows:
- The Filter criteria section defines the
base characteristics of the filter, including the following properties:
- Starting offset defines the position in the attribute string where the comparison is to begin.
- Comparison string defines the pattern string
against which the attribute is defined.
Type a string, pattern, or regular expression that will be used by the agent to filter the data read from the file. Depending on whether you choose for the filter to be inclusive or exclusive, the records that match the filter pattern will either be the only records returned to the Tivoli Enterprise Portal, subject to any override filters specified.
- Match entire value checks for an exact occurrence of the comparison string in the attribute string starting from the starting offset position.
- Match any part of value checks for the comparison string anywhere in the attribute string starting from the starting offset position.
- The comparison string is a regular expression indicates
the comparison string is a regular expression pattern that can be
applied against the attribute string.
Regular expression-filtering support is provided by using the International Components for Unicode (ICU) libraries to check whether the attribute value being examined matches the specified pattern.
To effectively use regular expression support, you must be familiar with the specifics of how ICU implements regular expressions, which is not identical to how regular expression support is implemented in Perl, grep, sed, Java regular expressions, and other implementations that you might have worked with in the past. See Appendix G. ICU regular expressions for guidance on creating regular expression filters.
- Define an override filter indicates that
you want to provide a more specific filter comparison that overrides
the base characteristics previously defined. This additional comparison
string is used to reverse the filter result. When the filter is Inclusive,
the override acts as an exclusion qualifier for the filter expression.
When the filter is Exclusive, the override acts as an inclusion qualifier
for the filter expression. (See Chapter 15, Step 9a for more details
on Inclusive versus Exclusive, and the examples in Step 2 of this
set of procedures). The override filter has the following properties:
- Starting offset defines the position in the attribute string where the comparison is to begin.
- Comparison string defines the pattern string
against which the attribute is matched.
Type a regular expression that will be used by the agent to filter the data read from the file. Depending on whether you choose for the filter to be inclusive or exclusive, the records that match the filter pattern are be eliminated from the records returned to the Tivoli Enterprise Portal, or they will be the only records returned to the Tivoli Enterprise Portal.
- Replacement value can be used to alter the raw attribute string with a new value. See Appendix G. ICU regular expressions for more details about special characters that can be used.
- Replace first occurrence replaces the first occurrence matched by the comparison string with new text.
- Replace all occurrences replaces all occurrences matched by the comparison string with new text.
- The Filter criteria section defines the
base characteristics of the filter, including the following properties:
- Click OK.
Example 1
If the attribute string is abc is easy as 123, then the replaced string that is displayed in the Tivoli Enterprise Portal as 123 is not as easy as abc.
Example 2
If the attribute string is "Unrecoverable Error reading from disk", and the filter is Inclusive, then the attribute is displayed in the Tivoli Enterprise Portal. If the attribute string is "No Errors Found during weekly backup", and the filter is Inclusive, then the attribute is not displayed in the Tivoli Enterprise Portal.
- Click Add, and complete the options in the
Add Filter window (Figure 133) as follows:
- In the Field Identification section of the
Advanced Log File Attribute Information page (Figure 132), specify how to override the attribute
group field delimiters for this one attribute only. Click one of the
attribute filters, and complete the required fields for the option:
- Number of characters: Enter the limit for the number of characters.
- Tab separator specifies the use of tab separators.
- Separator Text: Enter the separator text you want to use.
- Begin and End Text Enter both Begin text and End text.
- In the Summary section of the Advanced Log File Attribute Information page (Figure 132), click the Include attribute in summary attribute group check box to add the attribute to the summary attribute group. This attribute group is produced when a user turns on log attribute summarization.
- Click OK.
- In the Attribute Filters section, specify
the criteria for data to be included or excluded. Filtering attributes
can enhance the performance of your solution by reducing the amount
of data processed. Click one or more of the attribute filters:
- If you used the test functionality in Step 7, the Select key attributes page is displayed. On the Select key attributes page, select key attributes or indicate that this data source produces only one data row. See Selecting key attributes for more information.
- Do one of the following steps:
- If you are using the New Agent Wizard, click Next.
--OR--
- Click Finish to save the data source and open the Agent Editor.
- If you are using the New Agent Wizard, click Next.
![Add Filter window with "^([a-z]*) is ([a-z]*) as ([0-9]*)$" entered in the comparison string field, the Replacement value box checked, and "$3 is not as $2 as $1" entered in the replacement value field](addfilterexample1.gif)
Note:
When a Log File attribute group is added to an
agent that is at the default minimum Tivoli Monitoring version of 6.2.1 or later, a Log File Status
attribute group is automatically included. For more information about
the Log File Status attribute group, see Log File Status attribute group.