QRadar: Where is performance degradation happening?

Diagnosing The Problem

1. Click the notifications to view the alerts:
   
2. Click View All next to the notification to open a new search view. Example of a search:
   
3. (Optional) If there are multiple appliances experiencing performance degradation, change the Display drop down from Default (Normalized) to Source IP to identify the appliances experiencing the error. Each unique Source IP found here represents a different host affected by the performance degradation:
   • To narrow the search for routing to storage at device parsing (DSM Filter), use this AQL search:

     Select sourceip as 'IP', COUNT(*) from events where qid='38750088' and utf8(payload) ILIKE '%DSMFilter%'GROUP BY IP last 24 HOURS;

     To narrow the search for routing to storage in the Custom Rule Engine (EPCRE), use this AQL search:

     Select sourceip as 'IP', COUNT(*) from events where qid='38750088' and utf8(payload) ILIKE '%Custom Rule Engine%'GROUP BY IP last 24 HOURS;
4. Determine which service in the pipeline is causing the issue by viewing the payloads of the events.
   Note: It is possible to see performance degradation at both ecs-ec and ecs-ep on the same system at the same time.
   
   Result
   If the payload refers to Device Parsing, then the issue is occurring at the ecs-ec portion of the pipeline:

   [ecs-ec.ecs-ec] [[type=com.eventgnosis.system.ThreadedEventProcessor][parent=hostname:ecs-ec/EC/Parsing/DSM_Normalize]] com.ibm.si.ec.filters.normalize.DSMFilter: [WARN] [NOT:0080004101][IP/- -] [-/- -]Device Parsing has sent a total of 465454 event(s) directly to storage. 32655 event(s) have been sent in the last 60 seconds.  Queue is at 0 percent capacity.

   If the payload refers to Custom Rule Engine, then the issue is occurring at the ecs-ep portion of the pipeline:

   [ecs-ep.ecs-ep] [[type=com.eventgnosis.system.ThreadedEventProcessor][parent=hostname:ecs-ep/EP/Processor2]] com.q1labs.semsources.cre.CRE: [WARN] [NOT:0080004101][IP/- -] [-/- -]Custom Rule Engine has sent a total of 37231337 event(s) directly to storage. 58327 event(s) were sent in the last 60 seconds.  Queue is at 100 percent capacity.