Accumulator is falling behind

38750099 - The accumulator was unable to aggregate all events/flows for this interval.

Explanation

This message appears when the system is unable to accumulate data aggregations within a 60-second interval.

Every minute, the system creates data aggregations for each aggregated search. The data aggregations are used in time-series graphs and reports and must be completed within a 60-second interval. If the count of searches and unique values in the searches are too large, the time that is required to process the aggregations might exceed 60 seconds. Time-series graphs and reports might be missing columns for the time period when the problem occurred.

You do not lose data when this problem occurs. All raw data, events, and flows are still written to disk. Only the accumulations, which are data sets that are generated from stored data, are incomplete.

User response

The following factors might contribute to the increased workload that is causing the accumulator to fall behind:
Frequency of the incomplete accumulations
If the accumulation fails only once or twice a day, the drops might be caused by increased system load due to large searches, data compression cycles, or data backup.

Infrequent failures can be ignored. If the failures occur multiple times per day, during all hours, you might want to investigate further.

High system load
If other processes use many system resources, the increased system load can cause the aggregations to be slow. Review the cause of the increased system load and address the cause, if possible.

For example, if the failed accumulations occur during a large data search that takes a long time to complete, you might prevent the accumulator drops by reducing the size of the saved search.

Large accumulator demands
If the accumulator intervals are dropped regularly, you might need to reduce the workload.

The workload of the accumulator is driven by the number of aggregations and the number of unique objects in those aggregations. The number of unique objects in an aggregation depends on the group-by parameters and the filters that are applied to the search.

For example, a search that aggregates for services filters the data by using a local network hierarchy item, such as DMZ area. Grouping by IP address might result in a search that contains up to 200 unique objects. If you add destination ports to the search, and each server hosts 5 - 10 services on different ports, the new aggregate of destination.ip + destination.port can increase the number of unique objects to 2000. If you add the source IP address to the aggregate and you have thousands of remote IP addresses that hit each service, the aggregated view might have hundreds of thousands of unique values. This search creates a heavy demand on the accumulator.

To review the aggregated views that put the highest demand on the accumulator:
  1. On the Admin tab, click Aggregated Data Management.
  2. Click the Data Written column to sort in descending order and show the largest views.
  3. Review the business case for each of the largest aggregations to see whether they are still required.