QRadar: How to calculate the storage used by events and flows per day on a QRadar host

How To

Summary

This article serves as a guide to calculate how much storage is being used daily for both events and flows in the present month by using the command-line interface.

Environment

The commands in this article can be executed on QRadar hosts with local storage, which includes:

All-in-One consoles.
Event and Flow Processors.
Flow Processors.
Event Processors.
Data Nodes.

Before you begin

Some of these commands can be long, make sure to copy the entire line.
Depending on the amount of data stored on the host, the output might take longer to be displayed.

Steps

The commands to calculate the storage use per month varies depending on the tenant configuration. Follow the next steps per environment type to get the needed information.

Single-tenant environments:

Log in to QRadar by using the CLI.
The command needs to be executed on the device that is storing the events, so if the events are on a device different than the console, ssh to that device to execute the command.

Execute the next command, these commands generate a text file with the needed information.
For events:

for i in $(eval echo {1..$(date +%-d)}); do echo $(date +%Y)"/"$(date +%-m)"/"$i && du -shc /store/ariel/events/records/$(date +%Y)/$(date +%-m)/$i /store/ariel/events/payloads/$(date +%Y)/$(date +%-m)/$i 2>/dev/null ; echo  ;done > dailyStorageUsedByEvents-$(date +%B)$(date +%Y).txt

For Flows:

for i in $(eval echo {1..$(date +%-d)}); do echo $(date +%Y)"/"$(date +%-m)"/"$i && du -shc /store/ariel/flows/records/$(date +%Y)/$(date +%-m)/$i 2>/dev/null ; echo  ;done > dailyStorageUsedByFlows-$(date +%B)$(date +%Y).txt

Result:
The commands generate a .txt file, for example dailyStorageUsedByEvents-July2022.txt with the amount of stored data per day:
image-20220719140413-1

Multi-tenant environments:

When multi-tenancy is enabled and retention buckets are configured, each tenant has its own directory in the file system, they are located in the next path:

/store/ariel/events/records/aux/<tenantID#>/Year/Month/Day

These directories contain the information for those specific tenants.

The default path /store/ariel/events/records|payloads/ still exists for any nontenant tagged data.

Follow the next steps to get the information for multi-tenant environments:

In order to check the storage used by a specific tenant, we need to identify the ID of that tenant. The following command can be used to get the name associated with the tenant ID:
```
psql -U qradar -c "SELECT id, name FROM tenant WHERE deleted='f';"
```

Once the ID of the tenant is identified, the following commands can be executed.

For events:
Make sure to replace <Tenant_ID> with the corresponding value. It has to be replaced three times in this command.

for i in $(eval echo {1..$(date +%-d)}); do echo $(date +%Y)"/"$(date +%-m)"/"$i && du -shc /store/ariel/events/records/aux/<Tenant_ID>/$(date +%Y)/$(date +%-m)/$i /store/ariel/events/payloads/aux/<Tenant_ID>/$(date +%Y)/$(date +%-m)/$i 2>/dev/null ; echo  ;done > dailyStorageUsedByEventsTenant<Tenant_ID>-$(date +%B)$(date +%Y).txt

For flows:

The <Tenant_ID> needs to be replaced two times in this command.

for i in $(eval echo {1..$(date +%-d)}); do echo $(date +%Y)"/"$(date +%-m)"/"$i && du -shc /store/ariel/flows/records/aux/<Tenant_ID>/$(date +%Y)/$(date +%-m)/$i 2>/dev/null ; echo  ;done > dailyStorageUsedByFlowsTenant<Tenant_ID>-$(date +%B)$(date +%Y).txt

Optional: You can use also use the following command to collect data from all tenants:

for x in $(ls /store/ariel/events/payloads/aux/);do echo "*****************************" TENANT $x "*****************************"; for i in $(eval echo {1..$(date +%-d)}); do echo $(date +%Y)"/"$(date +%-m)"/"$i && du -shc /store/ariel/events/records/aux/$x/$(date +%Y)/$(date +%-m)/$i /store/ariel/events/payloads/aux/$x/$(date +%Y)/$(date +%-m)/$i /store/ariel/flows/records/aux/$x/$(date +%Y)/$(date +%-m)/$i  2>/dev/null ; echo  ;done done > dataSizeUsedByTenant-$(date +%B)$(date +%Y).txt

Result:
The commands generate a .txt file, for example dailyStorageUsedByEventsTenant1July2022.txt with the amount of stored data per day:
image-20220719143012-2

Additional Information

Creating a report for previous month in single tenant environments:

The procedure to make a report for the previous month is the same but there are some changes in the command used. Command automatically infers last month and how many days it has. It can be used on any day of the following month, but works only one month prior.

To generate a report detailing storage used by events, run the following command on a console:
for i in $(eval echo {1..$(date -d "$(date +%Y-%m-01) -1 day" +%d)}); do echo $(date +%Y)"/"$(date -d 'last month' +%m | sed 's/^0//')"/"$i && du -shc /store/ariel/events/records/$(date +%Y)/$(date -d 'last month' +%m | sed 's/^0//')/$i /store/ariel/events/payloads/$(date +%Y)/$(date -d 'last month' +%m | sed 's/^0//')/$i 2>/dev/null ; echo ;done > dailyStorageUsedByEvents-$(date -d 'last month' +%B)$(date +%Y).txt
To generate a report detailing storage used by flows, run the following command on a console:
for i in $(eval echo {1..$(date -d "$(date +%Y-%m-01) -1 day" +%d)}); do echo $(date +%Y)"/"$(date -d 'last month' +%m | sed 's/^0//')"/"$i && du -shc /store/ariel/flows/records/$(date +%Y)/$(date -d 'last month' +%m | sed 's/^0//')/$i 2>/dev/null ; echo ;done > dailyStorageUsedByFlows-$(date -d 'last month' +%B)$(date +%Y).txt

Creating a report for last month in multi-tenant environments:

<TenantID > must be replaced with proper value as taken from psql query outlined in paragraph detailing use case for current month.

To generate a report detailing storage used by events for a particular tenant, run the following command on a console:
for i in $(eval echo {1..$(date -d "$(date +%Y-%m-01) -1 day" +%d)}); do echo $(date +%Y)"/"$(date -d 'last month' +%m | sed 's/^0//')"/"$i && du -shc /store/ariel/events/records/aux/<Tenant_ID>/$(date +%Y)/$(date -d 'last month' +%m | sed 's/^0//')/$i /store/ariel/events/payloads/aux/<Tenant_ID>/$(date +%Y)/$(date -d 'last month' +%m | sed 's/^0//')/$i 2>/dev/null ; echo ;done > dailyStorageUsedByEventsTenant-<Tenant_ID>-$(date -d 'last month' +%B)$(date +%Y).txt
To generate a report detailing storage used by flows for a particular tenant, run the following command on a console:
for i in $(eval echo {1..$(date -d "$(date +%Y-%m-01) -1 day" +%d)}); do echo $(date +%Y)"/"$(date -d 'last month' +%m | sed 's/^0//')"/"$i && du -shc /store/ariel/flows/records/aux/<Tenant_ID>/$(date +%Y)/$(date -d 'last month' +%m | sed 's/^0//')/$i /store/ariel/flows/payloads/aux/<Tenant_ID>/$(date +%Y)/$(date -d 'last month' +%m | sed 's/^0//')/$i 2>/dev/null ; echo ;done > dailyStorageUsedByFlowsTenant-<Tenant_ID>-$(date -d 'last month' +%B)$(date +%Y).txt

Related Information

QRadar: How to determine average event payload and record size (in bytes) (Upda…

Disk Space 101

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt8AAA","label":"Ariel"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Tips