IBM Support

QRadar: Searching fails with error "There was a problem connecting to the query server. Please try again later"

Troubleshooting


Problem

By default, the Log Activity tab displays events in streaming mode, which allows to view events in real time. When this issue occurs, real-time streaming works as expected however administrators might find an error after a search is attempted by using filter criteria in the Log Activity despite which filter is used.

Symptom

The following are common symptoms when the issue occurs:
  • Events in streaming mode display correctly.
    image-20230327070837-1
  • Filtered-type searches report an error.
    image-20230324154549-1

Cause

The default behavior when the ariel_proxy_server service starts is to load all elements in the /store/transient/ariel_proxy.ariel_proxy_server/ directory including saved searches, then open the 32011 port for listening requests. When there are too many of these elements, the service can take several minutes to load until the port is set to LISTEN state. If the port is not in LISTEN state and users run searches, the error is prompted.

Diagnosing The Problem

The following steps guide administrators to determine when the ariel_proxy_server service is held loading the elements and not ready to receive new search requests. 

IMPORTANT: Restarting ariel_proxy_server might stop all searches from succeeding. Administrators with strict outage policies are advised to complete the next step during a scheduled maintenance window for their organization.
  1. SSH to the QRadar Console command line as the root user.
  2. Restart the ariel_proxy_server service. 
    systemctl restart ariel_proxy_server
  3. Check /var/log/qradar.log and review the elements being loaded. 
    tail -f /var/log/qradar.log | grep ariel_proxy
    Output example: 
    [ariel_proxy.ariel_proxy_server] [main] com.q1labs.ariel.searches.Loader: [INFO][-/- -]Quarantine file /store/transient/ariel_proxy.ariel_proxy_server/data/offense-33236-events-categories-1000-1662367166.0082064~offense-33236-events-categories-1000-1662367166.0082064.alias
[ariel_proxy.ariel_proxy_server] [main] com.q1labs.ariel.searches.Loader: [INFO][-/- -]Quarantine file /store/transient/ariel_proxy.ariel_proxy_server/data/offense-58385-events-destinationip-1000-1669150879.7965891~offense-58385-events-destinationip-1000-1669150879.7965891.alias
[ariel_proxy.ariel_proxy_server] [main] com.q1labs.ariel.searches.Loader: [INFO][-/- -]Quarantine file /store/transient/ariel_proxy.ariel_proxy_server/data/offense-37814-events-flows-1000-1663665759.4204779~offense-37814-events-flows-1000-1663665759.4204779.alias
  4. Verify that port 32011 is not in LISTEN state. 
    Note:  When port 32011 is not yet opened by the service the output of the following command is blank. 
    ss -nap | grep $(systemctl status ariel_proxy_server | grep 'Main PID' | awk '{print $3}') | grep LISTEN

    Result
    Administrators know now when the service fully loads and port 32011 is opened. A fully loaded ariel_proxy_server output looks like the following: 
    tcp    LISTEN     0      50     [::]:7782               [::]:*                   users:(("java",pid=9039,fd=666))
tcp    LISTEN     0      50     [::]:32011              [::]:*                   users:(("java",pid=9039,fd=663))

Resolving The Problem

To resolve the problem, administrators must reduce the number of elements to be loaded and tune the search criteria for faster results.
IMPORTANT: Restarting ariel_proxy_server might stop all searches from succeeding. Administrators with strict outage policies are advised to complete the next step during a scheduled maintenance window for their organization.
  1. SSH to the QRadar Console command line as the root user.
  2. Stop the ariel_proxy_server service. 
    systemctl stop ariel_proxy_server
  3. Remove the largest search files that slow down the service. 
    IMPORTANT: The following steps remove all the files to be reviewed later and faster recovery. Administrators must decide whether faster recovery or specific files to be removed is prefered. For more information about determining specific files to be removed, see How to identify and remove large search data files from /transient/ariel_proxy.ariel_proxy_server/data/ directory.
    1. Back up the current data: 
      mkdir -pv /store/IBM_Support
cp -fv /store/transient/ariel_proxy.ariel_proxy_server/data/* /store/IBM_Support/
    2. Remove the files. 
      rm -fv /store/transient/ariel_proxy.ariel_proxy_server/data/*
  4. Restart the ariel_proxy_server service. 
    systemctl restart ariel_proxy_server
  5. Verify that port 32011 is in a LISTEN state.  
    ss -nap | grep $(systemctl status ariel_proxy_server | grep 'Main PID' | awk '{print $3}') | grep LISTEN
  6. The fully loaded ariel_proxy_server output looks similar to: 
    tcp    LISTEN     0      50     [::]:7782               [::]:*                   users:(("java",pid=9039,fd=666))
tcp    LISTEN     0      50     [::]:32011              [::]:*                   users:(("java",pid=9039,fd=663))

    Results
    Administrators can now search by using the filter criteria as expected and use the backup files to start improving the search criteria to be more precise by using the following guidelines:

    Start - Searching Your QRadar Data Efficiently
    Part 1 - Utilize Quick Filters to Search Data
    Part 2 - Leveraging Indexed Properties in Search Queries
    Part 3 - Tips on Searching Data in QRadar

    If the problem persists after the restarts, contact QRadar Support for assistance.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt8AAA","label":"Ariel"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
05 May 2023

UID

ibm16966054

Page Feedback