IBM Support

QRadar: Migration from GlusterFS to Distibuted Replication Block Device on Event Collector terminates due to insufficient space

Troubleshooting


Problem

The QRadar upgrade to version 7.4.2 requires you to run a migration script on the console. This script migrates the High Availability file system from GlusterFS to Distributed Replication Block Device on all Event Collectors in your deployment:
/opt/qradar/ha/bin/glusterfs_migration_manager-<script_version>.bin
In some scenarios, the script terminates due to insufficient space.

Symptom

The glusterfs_migration_manager script displays an error:
Migration process did not start successfully for <event collector hostname>. Received a return code of 255.
Failed to run backup check. Check logs for more details.

Cause

Before the migration, the glusterfs_migration_manager script takes a backup of the /store on each Event Collector and stores that backup on /storetmp/backup by default. This error occurs due to one or more Event Collector(s) not having enough space in /storetmp.
More details of this script can be found in the product documentation.

Environment

QRadar Event Collectors (stand-alone or in a High Availability setup).

Diagnosing The Problem

On each of the affected Event Collectors, check the file /var/log/remove_glusterfs.log. If the migration ran into space related issues, the following messages are observed:

[WARNING] DRBD Metadata space check failed. Preparing space for metadata on the host
[INFO] Getting the backup directory
[INFO] The backup directory is /storetmp/backup
[ERROR] Store used space: 16663. Backup directory available space : 14294
[ERROR] Not enough space in /storetmp/backup. Clean up directory or provide another directory with --m to backup /store

Note: The space usage in the error above, is in megabytes. With regards to the entry above, the backup is 16,663 MBs and the space available on the /storetmp partition is 14,294 MBs.
 

Resolving The Problem

Before you begin:
Refer to Fix central for the most current version of the glusterfs_migration_manage script.
 

Step 1: Confirm whether the migrate script failed on an Event Collector(s) due to space issues.

  • The output from the script mentions the host-name of the Event Collector.

    Migration process did not start successfully for <event collector hostname>. Received a return code of 255.
    Failed to run backup check. Check logs for more details.
  • Check the content of /var/log/remove_glusterfs.log for these entries:

    [ERROR] Store used space: 16663. Backup directory available space : 14294
    [ERROR] Not enough space in /storetmp/backup. Clean up directory or provide another directory with --m to backup /store

If the above entries are seen in the remove_glusterfs.log continue to Step 2.


Step 2: If the script failed on a certain Event Collector due to space issues, a different partition than /storetmp needs to used for the backup on ALL the Event Collectors in the deployment. This is because the migration script has no provision to specify different partitions for each Event Collector.

Use the combination of the below commands to find which partition can be used for the backup.

  • On individual Event Collectors run the following command: 
    df -Th
  • On the console run the following command and filter the data for the Event Collectors in the deployment. 
    /opt/qradar/support/all_servers.sh -k "df -Th"

Note: Usually, the /recovery and /transient partitions are the best options from a space usage point of view. Do not use the /store partition as it will interfere with the stability of the system.


Step 3: Once you have decided on the partition to use for the backup, re-run the migration script on the console by specifically mentioning the partition in the migration command (using the --migrate parameter):

/opt/qradar/ha/bin/glusterfs_migration_manager-<script_version>.bin --migrate /transient

Document Location

Worldwide

The words LINSTOR®, DRBD®, LINBIT®, and the logo LINSTOR®, DRBD®, and LINBIT® are trademarks or registered trademarks of LINBIT in Austria, the United States and other countries.

[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtrAAA","label":"Rules"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.4.2"}]

Document Information

Modified date:
01 April 2021

UID

ibm16413281

Page Feedback