IBM Support

QRadar: Unable to complete a nightly configuration backup with NFS

Troubleshooting


Problem

Backups fail as a result of insufficient space with a "Partition containing directory: '/nfs' above warning threshold, disallowing backup" error in logs and a "Disk Sentry: Disk Usage exceeded warning threshold" warning in the dashboard.

Symptom

Dashboard notifications report a "Disk Sentry: Disk Usage exceeded warning threshold" warning. Errors similar to the following appear in /var/log/qradar.error: 
[hostcontext.hostcontext] [Scheduled Backup] com.q1labs.hostcontext.backup.BackupRecoveryEngine:
 [ERROR] [NOT:0150023103][x.x.x.x/- -] [-/- -]Partition containing directory: '/nfs' above warning threshold, 
  disallowing backup
[hostcontext.hostcontext] [Scheduled Backup] com.q1labs.hostcontext.backup.BackupRecoveryEngine:
 [WARN] [NOT:0000004000][x.x.x.x/- -] [-/- -]Unable to complete backup: config
[hostcontext.hostcontext] [Scheduled Backup] com.q1labs.hostcontext.backup.BackupRecoveryEngine:
 [WARN] [NOT:0000004000][x.x.x.x/- -] [-/- -]Unable to complete backup: data

Cause

Free disk space is greater than two times the size of the last backup, but backups are going into failure. By default, the threshold to stop backups from working is 90% (0.90) of the partition.

Diagnosing The Problem

Confirm connectivity to the NFS store, then its disk space usage.
  1. Use SSH to log in to your QRadar Console as root user.
  2. Mount the NFS stores to test your connection. Do not unmount your NFS share yet.
  3. Verify the disk space usage of the NFS share. The offboard storage guide uses /store/backup as a directory to mount NFS to.  
    du /store/backup -sh

    Results
    If the usage is 90% or greater, proceed to Resolving the Problem for strategies to reduce it. If you plan to delete files from the share, keep the stores mounted. Otherwise, unmount them.

Resolving The Problem

Use any of the following strategies to free up space.
  • Remove any unnecessary files.
  • Clean any old backup from the NFS share by coping it to another storage device or removing per your required retention policy.
  • Manage the backups by using the QRadar UI.
    1. Log in to QRadar as an admin user.
    2. Open the Admin settings.
    3. Click Back and Recovery icon.
    4. Change Backup Retention.
      1. Click Configure.
      2. Change the Backup Retention period days to a shorter interval.
    5. Delete backups.
      1. Select the backups to delete.
      2. Click Delete.

    Result
    After you use one or more of the strategies, verify the space on the NFS share is under 90%. If you require assistance to determine what files can be removed, Contact Support.

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
30 November 2022

UID

swg22010750

Page Feedback