QRadar: Identifying which Managed Host or Hosts are experiencing problems

Diagnosing The Problem

Before you begin: If the particular problem you are investigating is an IO Error that is seen while searching for events, a similar, more specific article, titled QRadar: Understanding IO Errors while searching might be more useful.

QRadar, like any complex, enterprise level software, is an interplay of number of software processes that work in harmony on multiple hosts. Due to this variety in interactive parts, any article that is focused on generic troubleshooting methods is bound to be incomplete. Still, there are several avenues of investigation available to identify which host in your environment might be having a problem.

1. Review Messages Notifications
   
   Many common issues are reported by the QRadar UI and reviewing the messages can often yield a wealth of information about what kinds of issues you are facing. This is also the first place to look when you are trying to identify any problems.
   
   
2. Verify that the System and License management page is displaying the expected status for all hosts.
   
   System and License Management window is accessible from the Admin tab of the UI in the System Configuration section.
   
   
   While certain problems with a managed host may not be displayed in this page, any unexpected values on the Status column can be an indication of a problem that requires further investigation:
   
   Even in instances where the values in the Status column are correct, it can be a good idea to review the 'View and Manage System' window for any suspected host. This window is accessed by right clicking the hostname for the suspected host.
3. Filtering by Event Processor
   
   When you are experiencing problems with your Event or Flow real time stream or historical searches, filtering by Event Processor will help you identify the problem. If you are not receiving any real time events when you filter by an Event Processor, or if you are receiving an IO Error with your historical searches, that can indicate a problem with the Event Processor or a collector that's connected to it. To set add such a filter, click the Add Filter button and select Event Processor parameter.
   
4. Performing a Deploy Full Configuration action
   
   Even when there are no changes to your configuration, performing a "Deploy Full Configuration" action can be extremely useful. This action forces all hosts to load the latest configuration files and this can resolve various problems. Even if the problems you are experiencing is not resolved, a "Deploy Full Configuration" action can point you to the managed host that requires troubleshooting.
   
   To perform a full deploy, click Admin tab > Advanced > Deploy Full Configuration.
   
   
   
   Note: A "Deploy Full Configuration" action results in a restart of processes and thus can result in a brief interruption of event collection.
   
   If you are facing a time out or an error against managed hosts when performing a "Deploy Full Configuration" action, the host experiencing the problem must be examined further.
   
   
5. Examining QRadar logs in the back end
   
   The most effective way of identifying which hosts are having problems is by examining QRadar logs. Interpreting various error messages that appear in QRadar logs can sometimes require a degree of expertise and not all "ERROR" type messages indicate problems. In many instances, the host name or the IP address of the managed host that is having problems will be printed plainly enough. If other methods mentioned above do not yield satisfactory results, you can connect to your console by using SSH and review /var/log/qradar.error logs with a command line tool like less and look for additional clues like this example:
   
   Mar 10 00:56:23 ::ffff:198.51.100.2 [ariel.ariel_proxy_server] [aqw_remote_2:8ee83965-7cce-4640-8a5a-16db5af2edf1] java.net.NoRouteToHostException: No route to host