QRadar: Troubleshooting Network Activity Overflow Records

Troubleshooting

Problem

Overflow records seen in IBM QRadar® Network Activity tab.

Cause

An overflow record is created when the number of flows captured exceeds the licensed limit of the QFlow component.

Sometimes, flows received are less than the license deployed, but we still see overflow records. The reason is that the hardware limit is less than the flow license.

Diagnosing The Problem

Overflow records with a source IP address of 127.0.0.4 and a destination IP address of 127.0.0.5 are seen in Network Activity.

Resolving The Problem

Important: Do not make manual changes to these values by editing files directly. Editing files directly may have unexpected results, and may lead to your deployment being unsupported.

There are a few limits that affect:

1. DEPLOYMENT_FLOW_LIMIT - The deployment limit is calculated based on the license limit of flows for the QRadar environment.

2. HARDWARE_FLOW_LIMIT - The hardware limit is based on hardware specification - RAM, core, CPU, and so on.

3. QF_GOVERNOR - This value can be set by the user.

Hardware limit takes priority over deployment limit as hardware has a "hard limit".

If the deployment limit is higher than hardware, the hardware limit is used to process flows.

Step 1: Check the current limits

Run the following command on the flow collecting QRadar host to check limits. The number in the file name (xxxx) can vary:

cat /opt/qradar/conf/nva.qflow.qflow<xxxx>.conf | grep -E 'DEPLOYMENT_FLOW_LIMIT|HARDWARE_FLOW_LIMIT|QF_GOVERNOR'

DEPLOYMENT_FLOW_LIMIT=715000
HARDWARE_FLOW_LIMIT=69396
QF_GOVERNOR=

The limits are based on flows per minute (fpm):

DEPLOYMENT_FLOW_LIMIT=715000 fpm = 11,916 fps
HARDWARE_FLOW_LIMIT=69396 fpm = 1156 fps

Step 2: Check overflow stats

less /var/log/qradar.log | grep -i qflow | less

Example output:

Oct 11 09:52:00 siem [QRADAR] [119498] qflow: [INFO] [1633931460] IPFIX Flow Source Stats for default_Netflow:  received and processed 28069 packets.
Oct 11 09:52:00 siem [QRADAR] [119498] qflow: [INFO] [1633931460] Sent 53206 flows on transport connection to x.x.x.x:32010
Oct 11 09:52:00 siem [QRADAR] [119498] qflow: [INFO] [1633931460] Flows held over for the next reporting interval: 0
Oct 11 09:52:00 siem [QRADAR] [119498] qflow: [INFO] [1633931520] Current interval starting input flow count: 69396
Oct 11 09:52:00 siem [QRADAR] [119498] qflow: [INFO] [1633931520] Number of flows that should be reported in the interval: 69405
Oct 11 09:52:00 siem [QRADAR] [119498] qflow: [INFO] [1633931520] Total number of aggregatable flows received from all flow sources: 415058
Oct 11 09:52:00 siem [QRADAR] [119498] qflow: [INFO] [1633931520] Total number of non-aggregatable flows received from all flow sources: 0
Oct 11 09:52:00 siem [QRADAR] [119498] qflow: [INFO] [1633931520] Byte count: 2435932155
Oct 11 09:52:00 siem [QRADAR] [119498] qflow: [INFO] [1633931520] Packet count: 27088611
Oct 11 09:52:00 siem [QRADAR] [119498] qflow: [INFO] [1633931520] Overflow count: 9 (Compressed: 284936)
Oct 11 09:52:00 siem [QRADAR] [119498] qflow: [INFO] [1633931520] Superflow count: 95 (Compressed: 16221)

So hardware limit is 69,396 flows per minute. Flows are processed based on this value, but flows received are higher than the value, hence overflows are seen.

The issue can be resolved by updating QF_GOVERNOR (currently 0) which takes priority over the other 2 limits.

Step 3: Calculate the required QF_GOVERNOR value, the values are shown in the output in Step 2

Number of flows that should be reported in the interval: 69,405
Aggregation ratio = (Total number of aggregatable flows received from all flow sources - Overflow compressed - Superflow compressed) / Current interval starting input flow count = (415,058 - 284,936 - 16,221) / 69,396 = 1.64

For every 1.64 flows, they will be compressed down to 1 flow.

Flows after aggregation = Overflow compressed / Aggregation ratio
= 284,936 / 1.64 = 173,741

Flow limit to be larger (governor limit) = HARDWARE_FLOW_LIMIT + Flows after aggregation = 69,396 + 173,741 = 243,137

Step 4: Update the governor limit

Log in to console.
Navigate to Admin > System and License Management.
Ensure that the Display is set to Systems.
Select the Flow Collector or -Processor, which we want to configure, and choose Deployment Actions > Edit Host.
Click the cogwheel next to Component Management.
In the Flow Collector section, set the Maximum Number of Flows variable to a reasonable number (250,000) for this appliance. This relates to the result of the calculation in Step 3.
Click Save.
Deploy the changes.

Related Information

QRadar: Overflow records in Network Activity

Flow capacity limits

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsuAAA","label":"Flow Source"}],"ARM Case Number":"TS007151766","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Tips