IBM Support

QRadar: How to resolve disk space usage problems for /store partition

Question & Answer


Question

What troubleshooting steps can be used to help resolve high disk usage situations on the /store partition?

Cause

The /store partition holds all the events and flow data on the system, as well as the default location for backups.

The partition size and type varies based on the appliance type (Console, Event Processor, etc.), model (newer Console model has larger storage), hardware, software installation (customer appliance) or VM, and QRadar version.

By default, the QRadar disk sentry check runs every 60 seconds and looks for high disk usage across the /store partition. If the /store partition fills up above 95%, it stops the QRadar critical services.

To find out what files or directories are filling up the /store partition, see the Troubleshooting Disk Space Problems technote: 

   

Answer

Quick Links

    

1. Troubleshooting /store space issues

These are the most commonly encountered issues that cause /store to fill out. For specific information about troubleshooting /store space issues, see below tech docs:

This article provides a list, and a brief description of each directory contains within /store/ariel/.


This article addresses two possible approaches to resolve concerns around data storage space usage.


In a distributed environment, an Event Processor (EP) cannot send logs to the Console if the ecs-ep process is down. The EP can disable processes if disk usages grow too high.

2. Defects around /store partition

This is a summary list of defects encountered on /store partition:

It has been identified that /var/log/ on High Availability Secondary appliances can fill due to the /var/log/systemStabMon directories not being rotated.


It has been identified in instances where the file /store/persistent_queue/ecs-ec.ecs-ec is not present on a QRadar processor appliance, that all event processing and storage on that appliance fails to occur until corrected.


It has been identified that during a High Availability (HA) failover configured with /store on ISCSI, the ISCSI mount automatically mounts the /store partition on both the Primary and Secondary. When this occurs, /store partition corruption is possible on the ISCSI environment.


It has been identified that the QRadar upgrade pretest "30-checkpartitions.sh" fails when /store is mounted on a multipath device on a standalone server.

  

3. General Information about the sizing of /store partition

Partition requirements and recommendations when upgrading:

During a software upgrade software installations only, partition requirements and recommendations are generated and stored in the /root/partition_instructions.txt file. This file is deleted during QRadar® setup on the new operating system. If you choose not to use the partitions recommendations, make sure that you meet these partition requirements.

Note: This document is only available in versions 7.3.0/7.3.1. You will need to download and uncompress the file for review.


Linux operating system partition properties for QRadar installations on your own hardware:

If you use your own appliance hardware, you can delete and re-create partitions on your Red Hat Enterprise Linux operating system rather than modify the default partitions.

Note:  Once on the IBM Knowledge Center, use the drop-down to select additional versions of the document. Only supported versions are provided.


Upgrading an appliance to ensure correct /store partition size before adding to a high-availability (HA) cluster:

Before you add an appliance to a high-availability (HA) cluster, you must confirm that the combined size of the /store and /transient partitions on the secondary HA host is the same size or larger than the /store partition on the primary HA host.

Note:  Once on the IBM Knowledge Center, use the drop-down to select additional versions of the document. Only supported versions are provided.


Offboard Storage Guide:

This guide provides information about how to move the /store or /store/ariel file systems to an external storage device for IBM® Security QRadar® products.


Reduce disk usage on /store documentation:

File system partitions reach 95% when the data retention period settings are too high, or the available storage is insufficient for the rate at which IBM® Security QRadar®receives data. If you reconfigure your retention bucket storage settings, the storage across your entire QRadar deployment is affected.


How is event and flow retention data handled when tenants are assigned in QRadar:

This technical note explains how event/flow retention data is handled when tenants are assigned in QRadar. This technical note is written in an FAQ-style and answers common questions from users who leverage tenants in their QRadar environment.

  

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"Component":"","Platform":[{"code":"PF043","label":"Red Hat"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
25 April 2022

UID

ibm10882066