Troubleshooting
Problem
A system administrator might notice that some events are failing to trigger rules that were expected to match.
Cause
There might be innumerous causes behind this problem. Throughout the article, we walk over the most common causes along with troubleshooting steps.
Environment
Before we start diagnosing the problem at a more granular level, we can determine whether the environment is experiencing pipeline performance issues. These issues could be spotted by the presence of notifications such as the following:
When such performance degradation occurs, QRadar skips rule correlation for a sample of events by routing them directly to storage. Affected events can be identified by an empty list of associated rules when carefully inspecting the Additional Information in the event details:
If you found no evidence of pipeline performance issues, continue reading through the following sections.
Otherwise, if you found that your environment is experiencing performance issues, target these issues for resolution first. Follow the steps mentioned on the User Response section of the documentation page linked previously.
Diagnosing The Problem
Once confirmed that the environment does not experience performance degradation, a useful next step is to verify that there are no flaws in the logic used. It is recommended to follow our guide QRadar: Troubleshooting rule tests with log activity searches for assistance with ensuring the events are being searched for correctly.
Once the search was optimized and the logic confirmed to match, we can proceed to a more granular review.
Orphaned Building Blocks
A user might wonder why a building block is not matching an event, even though the conditions are met. It can be because this building block is not referenced within any enabled rule in the system. In order to optimize system performance, QRadar does not test any building block, which is not referenced by an active rule on the system.
False Positive Rules
False Positive Rules trigger the CRE action "Bypass further rule correlation for this event or flow.", which prevents other rules or building blocks from triggering the event. An event can be determined as affected by opening the event details information page and analysing the Custom Rules field in the Additional Information section.
If you see FalsePositive: False Positive Rules and Building Blocks listed, then you can confirm that the event was triggered as being a false positive and skipping further rule correlation.
Note: Other Custom Rules can also cause similar behaviour when the rule is configured to trigger the CRE action "Bypass further rule correlation for this event or flow".
To prevent this event from being bypassed, you can make an exception in the false-positive Building Block or Rule that contains the criteria, which cause the rule to unintentionally mark this event as a false positive.
Events Bypassed by Routing Rules
Routing Rules have the ability to tag specific events for bypassing the Custom Rule Engine. There are two different options that can cause an event to bypass correlation. These options are "Bypass Correlation" and "Log Only (Exclude Analytics)". If an event is not triggering a rule expected to match, and the previous steps were confirmed not to be related: Make sure to check that none of the existing routing rules are causing this event to bypass rule correlation.
To inspect configured Routing Rules, log in to the QRadar GUI as an admin user. Navigate to the Admin Tab > System Configuration > Routing Rules. Once on the Routing Rules page, the rule criteria can be reviewed for enabled routing rules, which contain either of these options selected:
CRE Failed to Read Rules
In some instances, QRadar can fail to read some of the rules configured on the system. When this error occurs, the listed rules do not load and not be available for correlation. Refer to the following article for details as well as the recommended User Response:
Local and Global Stateful Rules
Each rule in QRadar can be configured to test locally or globally on the environment. This setting has the most impact with stateful tests, which are given a specific threshold used to trigger a rule.
For instance, a rule can be configured to trigger when matched at least 5 times within a predetermined period. This event may intrigue an analyst who sees 10 partial matches within the specified period and none of these fully triggering the rule.
This behaviour can occur if the rule is configured to test locally and the environment has multiple event processors. After further investigation, it is determined that the 10 events received were ditributed accross multiple processors. And, that none of the individual hosts received more than 4 events each, hence not fully meeting the rule criteria on any of them.
Chained Stateful Tests
Rule tests are always tested sequentially, including stateful tests. For instance, take the following Rule as an example:
In the previous case, the Rule would read as follows:
In the previous case, the Rule would read as follows:
- Test for the first conditional, and wait until there are at least 5 Username matches from the BB within 5 minutes
- When the first conditional is met, the second conditional waits will at least 5 Source IP's match from the BB within 5 minutes, not connected to the first conditional test
To address this problem, one would consider the following change:
This way, you are combining the same Username and Source IP to the same key pair, rather than finding 5 by Username and then 5 completely different Source IP's.
Interrupted Stateful Tests
Due to the nature of stateful tests, which perform tests over time, they are subject to being interrupted. The current state for the rule is stored in memory, meaning that any ecs-ep service disruptions or Full Configuration Deployments reset the rule's state. This behaviour can be most noticable with rules that test against large periods of time.
For instance, let's assume a rule is configured to match when 2 events are seen over a period of 24 hours. We see that 2 qualified events are received 12 hours apart of each other. If a service disrruption occurs between the receipt of the 2 events, we can notice that both events are qualified as partial matches. While the expectation, would be for the second event to fully match the rule. This expected behaviour does not occur due to the state reset.
A scheduled historical correlation profile with a disabled rule can be used in certain circunstances to ensure rules with large testing periods are not subject to interruptions. Refer to the documentation for best practices to use Historical Correlation Profiles and avoid extra system load.
Rule Tests Based on Dynamic Data
QRadar contains various sources of Dynamic Data, which keep changing over time, such as IP Geolocation, Asset Information, Network Hierarchy, Reference Data. However, rules are only evaluated once, when the event is collected. Users might notice during historical reviews that some events can contain the correct criteria to match certain rules, which it did not match. However, one needs to be careful when the criteria are based on Dynamic Data. As the fact that the event meets all the criteria now may not always mean that, it met the required criteria at the time it was collected into the system.
Resolving The Problem
Hopefully, the details provided along the course of this article were helpful to determine the root cause as to why the event did not trigger the intended rule. If the conclusions you reached by following this article are not conclusive, contact support for further assistance. Make sure to provide the following details to support:
- A Log File Bundle following steps 1A, 1B, or 1C from Getting Help: What information should be submitted with a QRadar service request?.
- A Full Export of the events affected by the problem by following the guidelines from Exporting events - IBM Documentation.
- Extra details with regards to the rule along with all detailed information collected throughout the course of this article.
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtrAAA","label":"Rules"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
24 August 2023
UID
swg21622847