Troubleshooting
Problem
/opt/qradar/ha/bin/glusterfs_migration_manager-<script_version>.bin
Symptom
Cause
Environment
Diagnosing The Problem
On each of the affected Event Collectors, check the file /var/log/remove_glusterfs.log. If the migration ran into space related issues, the following messages are observed:
[WARNING] DRBD Metadata space check failed. Preparing space for metadata on the host
[INFO] Getting the backup directory
[INFO] The backup directory is /storetmp/backup
[ERROR] Store used space: 16663. Backup directory available space : 14294
[ERROR] Not enough space in /storetmp/backup. Clean up directory or provide another directory with --m to backup /store
Note: The space usage in the error above, is in megabytes. With regards to the entry above, the backup is 16,663 MBs and the space available on the /storetmp partition is 14,294 MBs.
Resolving The Problem
Before you begin:
Refer to Fix central for the most current version of the glusterfs_migration_manage script.
Step 1: Confirm whether the migrate script failed on an Event Collector(s) due to space issues.
- The output from the script mentions the host-name of the Event Collector.
Migration process did not start successfully for <event collector hostname>. Received a return code of 255.
Failed to run backup check. Check logs for more details.
- Check the content of /var/log/remove_glusterfs.log for these entries:
[ERROR] Store used space: 16663. Backup directory available space : 14294
[ERROR] Not enough space in /storetmp/backup. Clean up directory or provide another directory with --m to backup /store
If the above entries are seen in the remove_glusterfs.log continue to Step 2.
Step 2: If the script failed on a certain Event Collector due to space issues, a different partition than /storetmp needs to used for the backup on ALL the Event Collectors in the deployment. This is because the migration script has no provision to specify different partitions for each Event Collector.
Use the combination of the below commands to find which partition can be used for the backup.
- On individual Event Collectors run the following command:
df -Th
- On the console run the following command and filter the data for the Event Collectors in the deployment.
/opt/qradar/support/all_servers.sh -k "df -Th"
Note: Usually, the /recovery and /transient partitions are the best options from a space usage point of view. Do not use the /store partition as it will interfere with the stability of the system.
Step 3: Once you have decided on the partition to use for the backup, re-run the migration script on the console by specifically mentioning the partition in the migration command (using the --migrate parameter):
/opt/qradar/ha/bin/glusterfs_migration_manager-<script_version>.bin --migrate /transient
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
01 April 2021
UID
ibm16413281