Troubleshooting
Problem
This technical note investigates the phenomenon of events dropped by protocols.
Symptom
Logging shows that events dropped at ecs-ec-ingress with "License restrictions have been applied" even when the system sees incoming EPS rate below the licensed rate. These messages are closely preceded by a message that shows the same number of events dropped in the last 60 seconds, but specifically reference the protocol.
In this example, events are dropped at the Microsoft Azure Event Hubs protocol.
In this example, events are dropped at the Microsoft Azure Event Hubs protocol.
[ecs-ec-ingress.ecs-ec-ingress] [SourceMonitor-0/ecs-ec-ingress.ecs-ec-ingress] com.q1labs.sem.monitors.SourceMonitor: [WARN] [NOT:0000004000][x.x.x.x/- -] [-/- -][Microsoft Azure Event Hubs] has detected a total of 18402 dropped event(s). 1495 event(s) were dropped in the last 60 seconds. Queue is at 0 percent capacity.
[ecs-ec-ingress.ecs-ec-ingress] [SourceMonitor-3/ecs-ec-ingress.ecs-ec-ingress] com.q1labs.sem.monitors.SourceMonitor: [WARN] [NOT:0060005100][x.x.x.x/- -] [-/- -]A total of 18402 dropped raw event(s) have been detected. 1495 raw event(s) have been dropped in the last 60 seconds. License restrictions have been applied 2 times in the last 60 seconds.
In this example, events are dropped at the AmazonAWSRESTAPI protocol.
[ecs-ec-ingress.ecs-ec-ingress] [SourceMonitor-4/ecs-ec-ingress.ecs-ec-ingress] com.q1labs.sem.monitors.SourceMonitor: [WARN] [NOT:0000004000][x.x.x.x/- -] [-/- -][AmazonAWSRESTAPI] has detected a total of 31616449 dropped event(s). 3252 event(s) were dropped in the last 60 seconds. Queue is at 0 percent capacity.
[ecs-ec-ingress.ecs-ec-ingress] [SourceMonitor-1/ecs-ec-ingress.ecs-ec-ingress] com.q1labs.sem.monitors.SourceMonitor: [WARN] [NOT:0060005100][x.x.x.x/- -] [-/- -]A total of 32247953 dropped raw event(s) have been detected. 3252 raw event(s) have been dropped in the last 60 seconds. License restrictions have been applied 5 times in the last 60 seconds.
Cause
On each managed host, ecs-ec-ingress maintains a queue for each active protocol type.
If this queue fills and new events need to be added to the queue, those events are dropped and trigger the error messages.
Environment
This issue affects any protocol where the incoming queue size might be exceeded and the "License restrictions have been applied".
Note: QRadar on Cloud administrators do not have access to the Console command-line interface. You can SSH and check the logs of the local Data Gateway appliance only.
Note: QRadar on Cloud administrators do not have access to the Console command-line interface. You can SSH and check the logs of the local Data Gateway appliance only.
Diagnosing The Problem
From this sample error message, we can tell that 3252 events were dropped at the AmazonAWSRESTAPI protocol on the managed host with IP 10.11.11.11.
[ecs-ec-ingress.ecs-ec-ingress] [SourceMonitor-4/ecs-ec-ingress.ecs-ec-ingress] com.q1labs.sem.monitors.SourceMonitor: [WARN] [NOT:0000004000][10.11.11.11/- -] [-/- -][AmazonAWSRESTAPI] has detected a total of 31616449 dropped event(s). 3252 event(s) were dropped in the last 60 seconds. Queue is at 0 percent capacity.
[ecs-ec-ingress.ecs-ec-ingress] [SourceMonitor-1/ecs-ec-ingress.ecs-ec-ingress] com.q1labs.sem.monitors.SourceMonitor: [WARN] [NOT:0060005100][10.11.11.11/- -] [-/- -]A total of 32247953 dropped raw event(s) have been detected. 3252 raw event(s) have been dropped in the last 60 seconds. License restrictions have been applied 5 times in the last 60 seconds.
Resolving The Problem
Events dropping at the protocol stage can be an indication of poor performance in the event pipeline or incorrect tuning for the protocol. If you experience a "License restrictions have been applied" error paired with a protocol error, check your logs and contact Support for more troubleshooting assistance.
How to get further assistance
Administrators can verify this issue in /var/log/qradar.log on the managed host that makes the connection to the remote log source to collect events.
- Log in to QRadar as the root user:
- For QRadar SIEM - administrators can open an SSH session to the Console, then SSH to the QRadar managed host that makes the connection.
- For QRadar on Cloud - administrators can open an SSH session to their Data Gateway appliance.
- Review the logs to confirm the issue. For example,
/var/log/qradar.log | grep "License restrictions" - Open a case with Support. Provide system logs for the Console and the systems whose IP addresses are identified in the license restriction errors.
Alternate approach from the GUI:
- Log in to QRadar console GUI.
- Navigate to the Log Activity tab.
- In the Advanced Search text box, enter this query:
select UTF8(payload) from events where ( "deviceType"='147' AND qid='38750002' ) AND ( ( icu4jsearch('dropped', payload) != -1 AND icu4jsearch('has detected a total of', payload) != -1 ) AND icu4jsearch('ingress', payload) != -1 ) order by "startTime" desc last 24 hours - Click the Search button.
- Review the logs to confirm the issue.
- Open a case with Support. Provide system logs for the Console and any systems whose IP addresses are identified in license restriction error messages.
Results
A QRadar Support representative contacts you about your case.
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtEAAQ","label":"Log Activity"},{"code":"a8m0z000000cwtiAAA","label":"Performance"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
11 December 2023
UID
ibm17091952