QRadar: Storage Performance Requirements

Answer

The QRadar platform allows for multiple methods of deployment, such as appliances, software installs on user procured hardware, and virtual machines. As part of these implementations, the storage subsystem in use will greatly impact  the performance of the system, for example, external storage, iSCSI, fiber channel, or virtual devices.

Many users that are designing their own environments often inquire about storage details regarding size, capacity, input/output operations per second rates, and disk latency. For the most part, the higher the level of storage performance, the faster QRadar will be able to write, index and query data; however the performance costs need to be measured against your performance requirements when designing your system.

When looking at external storage performance requirements, you need to consider how you plan on using QRadar, as this will dictate the level of performance you need. Things to take into consideration are:

The Rate of Data Being Collected:

Affected By: Events Per Second (eps), Flows Per Minute (fpm), number of Log Sources and Flow Sources.

As your rate of data collection increases, so does size of files written to disk each 1 minute interval, as well the size of indexes. This is when a faster hard drive should be taken into consideration. Writing data to disk, Input/output operations per second and seek time are less important than the data throughput capacity. Approximately 20 files are created every minute to store data to disk (1 for normalized records, 1 for data payloads & 1 for each enabled index).

The Number of Users Using QRadar:

Affected By: Searching data, running queries, and other tasks.

Once data is written out to disk, QRadar users, reports, offenses, and other tasks will eventually need to query the data. It's this data querying that is most impacted by the Input/output operations per second and latency metrics of your storage. If your plan is to basically use QRadar in a "logging" role, then you can probably afford a bit higher latency and reduced input/output operations per second capacity, to keep costs down. However keep in mind, that if your user/data query activity changes in the future, you're going to be stuck with your storage platform until you can consider upgrading it.

Size/Capacity of Storage:

The size of your storage requirements really depends on how much data you need to keep, and for how long. We are continually making changes to the product to improve performance. For example, in QRadar 7.2.6 and above, at the end of each hour, the system now consolidates indexes created on a minute by minute basis. This allows for far fewer seek requests when searching data, and larger reads to be performed. This takes advantage of higher throughput capacity's of today's storage devices, while reducing the impact on input/output operations per second and latency when querying storage systems.

So How Fast and How Large Does My Storage Need to Be?:

As mentioned above, multiple factors will influence your storage design. As a baseline for comparison, the following are metrics of existing QRadar appliances, and their supported data rates. If you are targeting comparable storage performance, your storage design should reflect these values or better.

(Appliance "M4" models, as of March 2016)

xx28 appliances, 40TB storage, up to 40,000 eps:

• 1.6GB/s sequential read
• Approximately 1200 input/output operations per second at queue depth of 1
• Latency/seek times: 2-4ms sequential , ~8ms random

xx05 appliances, 6TB storage, up to 20,000 eps:

• 1.1GB/s sequential reads
• Approximately 700 input/output operations per second at queue depth of 1
• Latency/seek times: 2-4ms sequential, ~8ms random

Based on these appliance specifications, most users will try to provide external storage with performance metrics of around 2000 input/output operations per second and 2ms seek times. The slightly higher values being used to mitigate the potential impact of accessing shared storage environments.

Related Information:

QRadar: Techniques to Reduce Storage Used
QRadar: The Impacts of Storage Hardware Speed

Where do you find more information?

---