QRadar: Creating event and flow indexes after restoring data on a managed host appliance

After upgrading to QRadar 7.2.6, new event and flow data collected is automatically converted to the super index format. However, existing data collected for QRadar 7.2.5 is not converted and still uses the previous index format. Administrators can follow the procedure in this technical note to convert 7.2.5 indices to the new 7.2.6 super index format.



Why are super indexes important?
In QRadar 7.2.6, event and flow data written to disk is indexed for specific types of data. now to take advantage of the new super indexes feature. Super indexes can improve search performance up to 10x for indexed value searches over the previous format, as well as being more compact to save on appliance disk space. Administrators must be on QRadar 7.2.6 to use Super Indexes. All new data received by the system that is upgraded to QRadar 7.2.6 will be automatically indexed using the new index format.



Do I need to update my index format?
No, updating 7.2.5 indexes is not required. Indexes written in the 7.2.5 format are still usable and fully searchable, however, they do not have the same performance as a super index when searching indexed values. The value of converting standard indexes depends on how long administrators retain data on their system and if you have already upgraded to 7.2.6 or not.



How does this process work?
Customers who recently upgraded to QRadar 7.2.6 can convert existing indexes to the new super index format using the command-line. Index files are on each QRadar appliance that runs search queries, such as Event Processors, Flow Processors, Data Nodes, Consoles, and All-in-One appliances. The administrator will have to use ariel_offline_indexer.sh utility from the command-line on every EP/FP/DNode/console in deployment.

The purpose of this script is to reads 7.2.5 indexes in to memory and convert the existing index to the new super index format. After the script completes the index conversion, the old QRadar 7.2.5 index is removed to save disk space.

A sample command that creates super index on events for events will looks like this:

/opt/qradar/bin/ariel_offline_indexer.sh -n events -v -s -d TimeinMinutes

For example, this command looks back 1 week for index files:

/opt/qradar/bin/ariel_offline_indexer.sh -n events -v -s -d 10080

Can the index conversion be run while the appliances are collecting data?
Yes, this conversion process can run while data is being collected. This script looks back in time to locate QRadar 7.2.5 indexes and converts them to the new 7.2.6 super index format that provides a significant search performance boost.

The following appliances create indexes in QRadar:

• Event Processors (16xx)
• Data Nodes (14xx)
• Flow Processor (17xx)
• Combination Event and Flow Processors (18xx)
• Consoles and All-in-One appliances (31xx or 21xx)
  
  IMPORTANT: The conversion process can run on systems while normal event collection is occurring, however, appliances that are near hardware capacity might want to start this process during a maintenance window. An example of an appliance that is near capacity would be as an Event Processor handling close to 20,000 EPS on hardware validated with a maximum limit of 20,000 EPS. This is not license related, but a capacity issue of the hardware. On these systems, administrators might want to start convert indexes to super indexes during a maintenance window hours to prevent slowing down high-capacity systems.

How long does this process typically take (Testing your system)?
To convert QRadar indexes to super indexes, administrators must run this script on each appliance that processes event data, such as Event Processors, Flow Processors, Data Nodes, Consoles, or All-in-Ones. Estimates during tests show that it takes approximately 6-8 minutes to convert an hour of indexes.

To confirm an average conversion time for your system, administrators can run a test on a recently upgraded 7.2.6 appliance. It is recommended that administrators run the ariel_offline_indexer script looking back 1 day to get an estimate of how long the data files take to convert, then run the script for the last week to test the time it takes to convert one day's worth of 7.2.5 index to 7.2.6 super indexes. After you have converted the data, the users or administrator can attempt a search using one of the indexed values to see the change in performance. Then administrators can determine how far back 7.2.5 data exists (based on retention settings) and convert that data to the new super index format. Depending on how far back administrators typically search, they can create super indexes for those time frames. There is no need to create super indexes for the last year of event or flow data, unless there is a need to search back that far.



Is there a recommended procedure to follow?
Yes, after upgrading to QRadar 7.2.6, administrators should start by testing the conversation to determine how long it will take on average for their appliance.

 

• Administrators can view the script help from the command link by typing: ./ariel_offline_indexer.sh --help.
• If super indexes already exist on the appliance, the indexes are skipped and the script continues to search the time frame for indexes that need to be converted. The script will not touch existing super indexes and these files are bypassed.

Procedure

1. Using SSH, log in to the QRadar Console as the root user.
2. Open an SSH session to the QRadar appliance where you restored data.
3. Navigate to the following directory: /opt/qradar/bin
4. Optional. Run a test command to determine the time it takes to convert indexes for one day (1440 minutes).
   
   For example:
   ./ariel_offline_indexer.sh -n events -v -s -d 1440
   OR
   ./ariel_offline_indexer.sh -n flows -v -s -d 1440

   Common time intervals	Minutes
   1 hour	60
   6 hours	360
   12 hours	720
   1 day	1440
   1 week	10080
   2 weeks	20160
   3 weeks	30240
   1 month	43800
   2 months	87600
   6 months	262800
   1 year	525600

   
   As the script converts indexes, a list of newly created indexes is displayed, along with the conversion time. The administrator can test the speed at which an appliance creates super indexes to determine how long it will take to convert all of the required data.
   
   Note: For longer conversions, administrators might want to start a screen session. To start a screen session, from the command-line type: screen
5. Run the command and expand the time frame to create super indexes as far back as required.
   
   For example:
   ./ariel_offline_indexer.sh -n events -v -s -d 10080
   OR
   ./ariel_offline_indexer.sh -n flows -v -s -d 10080
6. Wait for the conversion to complete.
   
   Note: If your SSH session becomes disconnected, you can reattach to the existing screen session.
7. Repeat this procedure on other QRadar appliances or for other data types (events or flows) on your appliance. The script can convert indexes for either events or flows and each conversion must be run separately.