IBM Support

QRadar: Delete files or directories to gain space in / partition

Troubleshooting


Problem

When the root "/" partition in QRadar does not have enough space, it can affect the regular functioning of QRadar® SIEM. The purpose of this article is to help the administrator with the removal of files and directories when the "/" partition has not enough available disk space.

Symptom

Lack of free space in the "/" partition can cause the following issues:
  • Alerts about "Process monitor application failed to start multiple times".
  • Searches reporting I/O errors.
  • Services not starting.
  • Configuration deployment changes due to critical disk space.
      
    [tomcat.tomcat] /console/JSON-RPC/QRadar.scheduleDeployment QRadar.scheduleDeployment] com.q1labs.configservices.util.ConfigServicesUtil: 
[INFO] [-/--] Deployment is blocked due to critical disk space issue
  • Failed disk space checks when a software update runs.
      
    =-= Disk Space Report Complete for '/'

[ERROR](testmode) - Mountpoint: / has 924140 Kb available and requires 1396312.8 Kb
[ERROR](testmode) Pretest had 1 failed checks for free space;
 - Mountpoint: / has 924140 Kb available and requires 1396312.8 Kb

Patch Report for <QRadar host IP>, appliance type: <QRadar appliance type>
=-= DiskSpace Report for Mountpoint '/' =-=
=-= Available: 924140 Kb,  Required: 1396312.8 KB =-=
=-= Total RPM Files: 87436 Kb =-=

=-= Disk Space Report Complete for '/'
<Hostname> :  patch test failed.

Cause

By default, the QRadar disk sentry check runs every 60 seconds and looks for high disk usage across the partitions. When a partition goes beyond the critical warning threshold, an alert is triggered for administrators to investigate.

Diagnosing The Problem

Follow both diagnosis sections and complete the Resolving the Problem steps for the issues you confirm your appliance has.
Appliances with undersized disks
QRadar installed on appliances with less than 256GB (minimum disk storage) can cause some partitions to default to the "/" partition. When the disk requirements are not met, the following directories can fill up the "/" partition on an undersized appliance:
  • /store/backup/
  • /store/persistent_queue/
  • /store/transient/
  • /transient/spillover/
  • /transient/ariel_proxy.ariel_proxy_server/
Use the lsblk command to find out whether the disk size is less than 256GB and /store and /transient partition exists on the system.
Figure01
Identify largest directories and files in the / partition.
To investigate the contents of the underlying directory, the root "/" partition needs to be mounted again to a different mount point with the --bind option.
IMPORTANT: This procedure can be run without the need to stop services or unmount any partitions.
  1. SSH into the QRadar console.
  2. Create a directory to be used as a temporary mount point. 
    mkdir -p /root/root_tmp/
  3. Mount the root partition to the newly created temporary directory.
     
    For QRadar 7.3 and later versions: 
    mount -o --bind /dev/mapper/rootrhel-root /root/root_tmp
  4. Investigate the largest directories and files and make note of the absolute (full) path.
    1. Navigate to the directory. 
      cd /root/root_tmp/
    2. Identify the largest directories and files by following the steps in: Troubleshooting disk space usage problems. 
       du -xch /root/root_tmp/ | sort -h
  5. Once the directories and files are identified, unmount the bind mount. 
    cd; umount /root/root_tmp/

Resolving The Problem

Follow the steps in Diagnosing the Problem to determine whether you must complete the instructions under Appliances with undersized disks or Identify and delete large directories and files in the "/" partition. If both issues appear on your appliance, follow both sections.
Appliances with undersized disks
The administrators can contact QRadar Support for temporary assistance. However, for a permanent solution, QRadar needs to be reinstalled by meeting the disk space requirements.
  1. Remove the managed host from the deployment.
  2. Stop the virtual machine and increase the hard disk space allocation. To do this action, administrators must refer to their respective hypervisor documentation.
  3. Reinstall QRadar.
    1. Gather the ISO file from the QRadar Software List 101 by using the ISO keyword in the Search Bar.
      Figure02
    2. Mount the ISO to the virtual machine. To do this action, administrators must refer to their respective hypervisor documentation.
    3. Power on the virtual machine and select the same appliance type as before. For step by step instructions, see the QRadar Installation Guide.
  4. Do a software update to match the Console's version.
    1. Gather the SFS file from the QRadar Software List 101.
      Figure03
    2. Run the software update by following the steps in the "Release Note" article of the SFS.
  5. Add the managed host to the deployment.
Result
The /store and /transient partitions are now in a separate partition, which prevents the directories and files inside these partitions to fill up the "/" partition.
Safe-to-delete files in the / partition.
If the /root or /home directories are the largest directory, it is likely that user-forgotten files are filling up the partition. Administrators can move or remove the files inside the directory. For other possible files or scenarios, refer to the About / partition article.
 
  1. Move or Remove a file.
    • To move a file. 
      mkdir -p /store/IBM_Support/
mv -v /full/path/to/<file> /store/IBM_Support/
      Note: The /store/IBM_Support/ cannot be used on QRadar HA clusters when the "/" partition is full on a host in STANDBY state. On these appliances, use the /storetmp/ partition instead.

      Output Example 
      ‘/root/scripts/test_file.zip’ -> ‘/store/IBM_Support/test_file.zip’
removed ‘/root/scripts/test_file.zip’
    • To remove a file.
      rm -fv /full/path/to/<file>
      Output Example: 
      removed ‘/root/scripts/test_file.zip’
  2. Verify the partition now has more space. 
    df -Th /
    Output Example: 
    Filesystem                Type  Size  Used Avail Use% Mounted on
/dev/mapper/rootrhel-root xfs    13G  4.1G  8.5G  33% /
Result
The "/" partition no longer has disk space constraints. If the partition reached the point of critical services stop, administrators must restart the services in the proper order and wait 5 mins with the following commands:
IMPORTANT: When the QRadar core service restart, the QRadar UI, event processing, and database are not available to all users. Administrators with strict outage policies are advised to complete the next step during a scheduled maintenance window for their organization.
  
systemctl stop hostcontext
systemctl stop tomcat
systemctl restart hostservices
systemctl start tomcat
systemctl start hostcontext
If the partition does not decrease its usage or the services does not start properly, administrators can contact QRadar Support for assistance.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
30 September 2022

UID

ibm16598681

Page Feedback