Troubleshooting
Problem
When the FPS rate is hitting the license limit continuously, causing the pipeline and spillover to back up and getting cleared in intervals. Same thing happens after an EP or FP restart.
This article is for users who want to understand what is causing this behavior.
Environment
QRadar Flow processor
Resolving The Problem
If FPS rate hitting the license limit continuously, thus pipeline backing up, spillover also filling up and getting cleared regularly.
Symptom stack traces
::ffff:X.X.X [ecs-ec.ecs-ec] [02abe3ac-2f4f-472a-b139-0b72540e75f0/SequentialEventDispatcher] com.q1labs.sem.monitors.SourceMonitor: [WARN] [NOT:0000004000][X.X.X/- -] [-/- -][FlowSource] has detected a total of 156940237 dropped flow(s) (Flows off the wire). 65173.0 flow(s) were dropped in the last 60 seconds. Queue is at 100 percent capacity.
::ffff:X.X.X [ecs-ec.ecs-ec] [02abe3ac-2f4f-472a-b139-0b72540e75f0/SequentialEventDispatcher] com.q1labs.sem.monitors.SourceMonitor: [WARN] [NOT:0000004000][X.X.X/- -] [-/- -][FPS License] FPS on this system has been over license 120 times in the last 60 seconds (total of 729838 times since the last process restart). This message is shown when you are over your license for more than 75% of the last minute, indicating you are reaching your license limit; flows may be dropped if the rate does not decrease.
::ffff:X.X.X [ecs-ec.ecs-ec] [SourceMonitorTimerTask] com.q1labs.sem.monitors.SourceMonitor: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -]Incoming flow rate [5s: (966.00):(0.00) fps], [10s: (1305.90):(-0.00) fps], [15s: (1228.47):(-0.00) fps], [30s: (1148.57):(-0.00) fps], [60s: (1109.57):(0.02) fps], [300s: (1098.40):(111.76) fps], [900s: (1098.40):(111.76) fps]. Peak in the last 60s: (1645.80):(0.20) fps. Max Seen (2179.00):(21898.20) fps.License Threshold: 332.00
::ffff:X.X.X [ecs-ec.ecs-ec] [SourceMonitorTimerTask] com.q1labs.sem.monitors.SourceMonitor: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -]Incoming flow rate [5s: (1065.00):(0.00) fps], [10s: (1454.50):(-0.00) fps], [15s: (1295.47):(-0.00) fps], [30s: (1132.60):(-0.00) fps], [60s: (1052.33):(-0.00) fps], [300s: (1091.14):(91.34) fps], [900s: (1091.14):(91.34) fps]. Peak in the last 60s: (1844.00):(0.00) fps. Max Seen (2179.00):(21898.20) fps.License Threshold: 332.00
Event pipeline backing up
::ffff:X.X.X [ecs-ec.ecs-ec] [ReceiverServer:ecs-ec/EC/FlowSource] com.q1labs.sem.monitors.PipelineStatusMonitor: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -] ---- PIPELINE STATUS -- Initiated From: FlowSource
::ffff:X.X.X [ecs-ec.ecs-ec] [ReceiverServer:ecs-ec/EC/FlowSource] com.q1labs.sem.monitors.PipelineStatusMonitor: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -] EP (Filters: 0.00 pc) (Queues: 0.00 pc) (Sources: 0.00 pc)
::ffff:X.X.X [ecs-ec.ecs-ec] [ReceiverServer:ecs-ec/EC/FlowSource] com.q1labs.sem.monitors.PipelineStatusMonitor: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -] MPC (Filters: 0.00 pc) (Queues: 0.00 pc) (Sources: 0.00 pc)
::ffff:X.X.X [ecs-ec.ecs-ec] [ReceiverServer:ecs-ec/EC/FlowSource] com.q1labs.sem.monitors.PipelineStatusMonitor: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -] EC (Filters: 4.26 pc) (Queues: 2.22 pc) (Sources: 90.91 pc)
::ffff:X.X.X [ecs-ec.ecs-ec] [ReceiverServer:ecs-ec/EC/FlowSource] com.q1labs.sem.monitors.PipelineStatusMonitor: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -] 100.00 pc - Filter:EventCoalesceFilter (10000/10000)
::ffff:X.X.X [ecs-ec.ecs-ec] [ReceiverServer:ecs-ec/EC/FlowSource] com.q1labs.sem.monitors.PipelineStatusMonitor: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -] 100.00 pc - Source:FlowSource (100000/100000)
::ffff:X.X.X [ecs-ec.ecs-ec] [ReceiverServer:ecs-ec/EC/FlowSource] com.q1labs.sem.monitors.PipelineStatusMonitor: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -] 100.00 pc - Queue:FlowStack/FlowAsymmetricFilter (250/250)
::ffff:X.X.X [ecs-ec.ecs-ec] [ReceiverServer:ecs-ec/EC/FlowSource] com.q1labs.sem.monitors.PipelineStatusMonitor: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -] 100.00 pc - Queue:FlowSource (250/250)
::ffff:X.X.X [ecs-ec.ecs-ec] [ReceiverServer:ecs-ec/EC/FlowSource] com.q1labs.sem.monitors.PipelineStatusMonitor: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -] 100.00 pc - Queue:FlowStack (250/250)
The reason for these peaks happening at regular interval is due to spillover queue being cleared at those intervals:
::ffff:X.X.X [ecs-ec.ecs-ec] [FlowGovernerProcessor] com.q1labs.frameworks.queue.SpilloverQueue.FlowGovernerQueue: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -]spillQueue poll: needSignalNotFull = true
::ffff:X.X.X [ecs-ec.ecs-ec] [FlowGovernerProcessor] com.q1labs.frameworks.queue.SpilloverQueue.FlowGovernerQueue: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -]spillQueue poll: needSignalNotFull = true
::ffff:X.X.X [ecs-ec.ecs-ec] [FlowGovernerProcessor] com.q1labs.frameworks.queue.SpilloverQueue.FlowGovernerQueue: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -]spillQueue poll: needSignalNotFull = true
::ffff:X.X.X [ecs-ec.ecs-ec] [FlowGovernerProcessor] com.q1labs.frameworks.queue.SpilloverQueue.FlowGovernerQueue: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -]spillQueue poll: needSignalNotFull = true
::ffff:X.X.X [ecs-ec.ecs-ec] [FlowGovernerProcessor] com.q1labs.frameworks.queue.SpilloverQueue.FlowGovernerQueue: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -]spillQueue poll: needSignalNotFull = true
::ffff:X.X.X [ecs-ec.ecs-ec] [FlowGovernerProcessor] com.q1labs.frameworks.queue.SpilloverQueue.FlowGovernerQueue: [INFO] [NOT:0000006000][X.X.X/- -] [-/- -]spillQueue poll: needSignalNotFull = true
Resolution:
Increase your FPS license or fix flow bursts.
Increase your FPS license or fix flow bursts.
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsuAAA","label":"Flow Source"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
24 February 2023
UID
ibm16479659