IBM Support

QRadar: Scheduled backups are timing out and fail to complete

Troubleshooting


Problem

Scheduled backups are running for a long time and fail to complete successfully. A system notification is generated when a backup fails to complete for "Backup: The last scheduled backup exceeded execution threshold.".

Symptom

A system notification is generated when a backup fails to complete for "Backup: The last scheduled backup exceeded execution threshold.".
The following messages are also displayed in the /var/log/qradar.log file:
[Scheduled Backup]com.q1labs.hostcontext.backup.BackupRecoveryEngine: [WARN] [NOT:0150024101][<Console IP>/- -] [-/- -]Backup has exceeded its time limit of 10800s, canceling...

[Scheduled Backup]com.q1labs.hostcontext.backup.BackupRecoveryEngine: [ERROR] [NOT:0150023105][<Console IP>/- -] [-/- -]Backup Failed: TIMED_OUT

Cause

Some causes for this issue are:
  1. One or more disks are under heavy load and cause the creation of the backup file to take more time than the limit configured.
  2. CPU resources are used by more critical processes during the backup file creation.
  3. The backup file is huge.

Resolving The Problem

The following actions can be combined to alleviate this issue.
  1. Ensure the disk performance.
    1. Virtual deployments.

      The administrator must first confirm that the storage requirements (IOPS and Data Transfer) are delivered to the hypervisor's disks. This requirement must be verified by the Virtualization Platform Admin hosting the QRadar® virtual machine.
       
      To know what the recommended values are, check the Storage Requirement section in the QRadar® documentation based on how QRadar® is installed:
       
      For QRadar® Software installations requirements, check: Prerequisites for installing QRadar® on your hardware.
      For QRadar® Virtual Appliances requirements, check: System requirements for virtual appliances.
       
    2. Physical deployments.
       
      This issue can indicate a RAID or disk performance degradation.
       
      If QRadar® is installed on QRadar® Appliances purchased from IBM®, check for the disk notifications in the disks for further steps to investigate hardware-related matters.
      If QRadar® is installed on other hardware, the administrator must refer to the specific vendor troubleshooting guide.
       
  2. Increase the timeout and priority values.

    By default, configuration backups in QRadar are set to time out in 180 minutes. In some cases, administrators might need to adjust this value or change the default priority setting if backups fail to complete in the allotted time. Administrators can investigate this issue by reviewing the duration of any successful backups and then they can adjust the backup time limit when required.
    1. Log in to the QRadar Console as an administrator.
    2. Click the Admin tab.
    3. Click the Backup and Recovery icon. This icon is found under System Configuration.
    4. From the navigation bar, click Configure.
    5. The default for configuration backups is to set a Backup Time Limit (min) of 180 minutes, change this value to double.  Do not extend the time limit over 24 hours.
      Note:
      The administrators can repeat this procedure to increase the value until backups don't time out.
       
    6. Verify that the Backup Priority field is configured to High.
      Note: Increasing the priority makes the OS allocate more resources to the backup process. Other processes such as searches and applications queries can be affected by resource competition.
       
    7. Click Save.
      From the navigation bar on the Admin tab, click Deploy Changes.

      Results
      The administrator can monitor the status to ensure that a backup completes successfully. It is typically suggested that administrators set time limits on configuration backups to be 45 - 60 minutes longer than the longest backup.
       
  3. Configure the backups to be stored on an NFS Server.

    To avoid the disk I/O requests affecting the server's disks, the administrator can configure an NFS Server and configure QRadar to store the events in it.

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
31 March 2022

UID

swg21694955