Avoiding process errors with large trace or log files by using log file rotation

If trace or log files become too large, processes might quit unexpectedly. If too many files are created, you might run out of disk space. Configure the number and size of trace and log files by configuring log file rotation.

You can control the size of log and trace files using one of the following criteria:

  • Maximum size
  • Time of day

You can configure the number of log files by configuring a pool of files, that is, a number of files that are written to in succession. The process of using new files when triggered is called log file rotation. All log files can be rotated, but not all components have log files. Some components have only trace files, which can also be rotated.

Restriction: Only processes that are managed by the domain process controller, ncp_ctrl, can have their log files rotated.

Setting the maximum size of log and trace files

Large log or trace files can cause process errors. The default maximum size for a log file is 1 GB on UNIX systems. As a guidance estimate for log files, assuming that each log file is 1 GB in size and six processes are set to full debug level, you would require 24 GB of disk space. (6 processes x 4 log or trace files each = 24 log or trace files x 1 GB = 24 GB).

Set the NDE_LOGFILE_MAXSIZE environment variable to determine the maximum size that a log or trace file can reach for a process. This setting applies to all Network Manager processes. When the log file reaches the maximum size, the next log file is used.

  • If log file rotation is turned off for a process, only two files are used.

  • If log file rotation is turned on for a process, the number of files configured in the pool are used.

You can either rotate log files according to file size or time of day. If log file rotation according to time of day is configured, the settings for file size are ignored.

The following example shows how to set the maximum log file size to 1 GB (the setting is in bytes).
setenv NDE_LOGFILE_MAXSIZE 1073741824
Note: Do not set a maximum size of less than 1000000 (1MB). Setting a maximum size of less than 1000000 (1MB) can cause the ncp_g_event and nco_p_monitor processes to fail.

You must restart Network Manager processes after you change the log file rotation environment variables so that the processes use the updated variables.

Important: On UNIX systems, make sure that you set the logging environment variables in the appropriate shell profile files for the account that Network Manager is running. Do not set them in the NCHOME/env.sh file because this file is not used when Network Manager starts.

Setting a time of day for log file rotation

To set log file rotation to occur at a certain time of day, set the NDE_LOGFILE_ROTATION_TIME environment variable to the time required, and set the NDE_LOGFILE_ROTATION_FORMAT environment variable to the required naming format. If you set a time of day, this setting takes precedence over the maximum file size setting.

The NDE_LOGFILE_ROTATION_TIME environment variable specifies the time after which a log file rotation occurs each day. Log files are rotated when the first write is made to the file after the time that is set by the NDE_LOGFILE_ROTATION_TIME environment variable. This environment variable is an integer, with a default of 0000.

The following example specifies log file rotation at the first write after midnight each day: 
setenv NDE_LOGFILE_ROTATION_FORMAT "%Y%m%d-%H%M"
setenv NDE_LOGFILE_ROTATION_TIME 0000
The following example shows how to rotate the log files at midnight each day, and append the old log file name with the literal character string "old"
setenv NDE_LOGFILE_ROTATION_FORMAT \'old\'
setenv NDE_LOGFILE_ROTATION_TIME 0000
Restriction: Literal characters must be escaped by quotation marks (' ') as described in http://userguide.icu-project.org/formatparse/datetime.

The NDE_LOGFILE_ROTATION_FORMAT environment variable configures how the log files are created.

  • If NDE_LOGFILE_ROTATION_FORMAT is set to "old", and a pool of log files is not configured for the process in question, two files are used.
  • If a pool of log files is configured for a process, the number of files in the pool is used.
  • If NDE_LOGFILE_ROTATION_FORMAT is set to a supported time stamp format, and a pool of log files is not configured for the process in question, a new file is created each day. Using a time stamp format without configuring a pool of log files can result in an unlimited number of log files.
    Restriction: The only time stamp format that is supported for use by Network Manager processes is the POSIX format. The following options are supported:
    %a
    Abbreviated weekday name, for example, Thu
    %A
    Full weekday name, for example, Thursday
    %b
    Abbreviated month name, for example, Aug
    %B
    Full month name, for example, August
    %d
    Day of the month, zero-padded, (01-31), for example, 23
    %e
    Day of the month, space-padded, ( 1-31), for example, 23
    %h
    Equivalent to %b.
    %H
    Hour in 24h format (00-23), for example, 14
    %I
    Hour in 12h format (01-12), for example, 02
    %j
    Day of the year (001-366), for example, 235
    %m
    Month as a decimal number (01-12), for example, 08
    %M
    Minute (00-59), for example, 55
    %p
    AM or PM designation, for example, PM
    %R
    24-hour HH:MM time, equivalent to %H:%M, for example, 14:55
    %S
    Second (00-61), for example, 02
    %T
    ISO 8601 time format (HH:MM:SS), equivalent to %H:%M:%S, for example, 14:55:02
    %u
    ISO 8601 weekday as number with Monday as 1 (1-7), for example, 4
    %V
    ISO 8601 week number (00-53), for example, 34
    %y
    Year, last two digits (00-99), for example, 01
    %Y
    Year, for example, 2001
    %Z
    Timezone name.

For example, %Y%m%d-%H%M generates rotate log files with the year, month, day, hour, and minute added. An example file is ncp_model.NCOMS.log_20100430-0000.

You must restart Network Manager processes after you change the log file rotation environment variables so that the processes use the updated variables.

Important: On UNIX systems, make sure that you set the logging environment variables in the appropriate shell profile files for the account that Network Manager is running. Do not set them in the NCHOME/env.sh file because this file is not used when Network Manager starts.

Configuring the number of log files created per process

The number of log files created by a process is determined on a per-process basis by the command-line variables -logfileusepool and -logfilepoolsize. If a Network Manager process is started with the -logfileusepool command line option, then the process uses a pool of log files. The number of files in the pool is defined by the value of the -logfilepoolsize command line option. If the -logfilepoolsize command line option is not specified, the default value of 10 is used. The minimum value of -logfilepoolsize is 2 and the maximum is 99.

The log files in the pool are deleted each time the process is restarted by the user or by the system. If you want to keep historical logs you need to use the NDE_LOGFILE_ROTATION_FORMAT without a log pool.

Important: If you have many processes that are configured to create a pool of their own log and trace files, the total number of log files created can be large. Ensure you have enough disk space. Ensure that your operating system allows enough files to be opened concurrently. On UNIX and Linux® operating systems, ensure that there are enough file handles allowed.
Restriction:

Do not start the ncp_g_event or nco_p_ncpmonitor processes with the -logfileusepool command line option if you have set the NDE_LOGFILE_ROTATION_FORMAT environment variable. Using log file pooling and a timed log file rotation together for these processes is not supported.

Example of starting a process with log file pool specified

The following example excerpt from the CtrlServices.cfg configuration file configures the ncp_disco process to use a pool of 5 log files:

insert into services.inTray (
    serviceName,
    binaryName,
    servicePath,
    domainName,
    argList,
    dependsOn,
    retryCount
)
values (
    "ncp_model",
     "ncp_model",
     "$PRECISION_HOME/platform/$PLATFORM/bin",
      "$PRECISION_DOMAIN",
       [ "-domain" , "$PRECISION_DOMAIN", "-latency" , "100000" , "-debug", "0-", "-messagelevel", "warn", "-logfileusepool", "-logfilepoolsize", "5" ],
       [ "ncp_config", "ncp_store", "ncp_class" ],
      5
);

What happens when log files are rotated

If a pool of log files is configured for a process, log file rotation occurs as follows:
  1. A number of log files are created for that process when it starts. The number of files that are opened is configured by the value of the -logfilepoolsize command line option. The process keeps the log files open for as long the process runs. By contrast, .trace files are only created when they are needed, and are not kept open. Files in the pool are named using the format process[.domain].log_ID or process[.domain].trace_ID, where ID is a two-digit number that starts from 01, and ranges up to the maximum number specified for the -logfilepoolsize option.
  2. If they already exist, the first N .log files are truncated (set to zero size), and the first N .trace files are deleted, where N is the number of log files specified for the -logfilepoolsize option for the process. If there are more than N .log or .trace files, for example from a previous time when the log pool size was greater, the remaining files are not affected. If you want to be sure to keep all previous .log or .trace files, copy them to a different location before starting or restarting the process.
    Note: An exception is the ncp_disco process. When the ncp_disco process is restarted (including on every full discovery) the old ncp_disco.DOMAIN.log_n files are renamed with an extension of .old appended to the file name
  3. The file with the lowest available file number is used for logging, regardless of which file was last used.
  4. When the criteria for log rotation are met, the logging system writes to the next file.
  5. When all the files in the pool are at maximum size, the logging system truncates the first file in the pool and starts writing to it again.
If a pool of log files is not configured, log file rotation occurs as follows:
  1. The ncp_ctrl process renames the log file from logfilename.log to the format set by the NDE_LOGFILE_ROTATION_FORMAT environment variable, or to logfilename.log_old if the NDE_LOGFILE_ROTATION_FORMAT environment variable is not set.
  2. The ncp_ctrl process generates a new log file named logfilename.log.
  3. When the new logfilename.log file reaches the maximum size, the ncp_ctrl process overwrites logfilename.log_old.