Troubleshooting
Problem
The QRadar® upgrade to version 7.4.2 requires you to run a migration script on the console. This script migrates the High Availability file system from GlusterFS to Distributed Replication Block Device on all Event Collectors in your deployment (irrespective of whether they are currently part of an HA setup or not).
In some rare scenarios, the script can fail on Event Collectors that were upgraded from versions prior to 7.3.x that used an ext4 partition for /store.
Symptom
When run, the migration script fails and displays this sequence of error messages:
Jul 29 16:34:52 [ERROR] Failed to mount store: mount: wrong fs type, bad option, bad superblock on /dev/sda8, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so.
Jul 29 16:34:52 [ERROR] Failed to resize /store on deployment. Check logs for more details
Jul 29 16:36:02 [ERROR] Unexpected error running run_prepare_ha: cannot concatenate 'str' and 'function' objects
Cause
The migration script creates a new xfs filesystem with space left for the Distributed Replication Block Device metadata.
Jul 29 16:34:44 [INFO] Resizing /store to make space for DRBD metadata
Jul 29 16:34:44 [INFO] Preparing /store resizing
Jul 29 16:34:44 [WARNING] Could not locate store on LVM. Upgraded system detected
Jul 29 16:34:44 [INFO] Found /store on /dev/sda8
Jul 29 16:34:50 [INFO] /store has been unmounted properly
Jul 29 16:34:52 [INFO] Running xfscmd mkfs.xfs -f -d size=51394048b /dev/sda8
Jul 29 16:34:52 [INFO] /store has been resized
However, for appliances that were migrated from older versions, the migration script checks for and finds an existing ext4 entry for /store in /etc/fstab and does not update the filesystem type in that entry.
When the script later attempts to mount /store by using the older /etc/fstab entry (with the ext4 filesystem), there is a filesystem mismatch (xfs vs ext4) and this error occurs:
Jul 29 16:34:52 [ERROR] Failed to mount store: mount: wrong fs type, bad option, bad superblock on /dev/sda8, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so.
Environment
QRadar® Event Collectors upgrading to 7.4.2
Diagnosing The Problem
When the script fails on one or more Event Collectors, check these points on each Event Collector:
- Check the log file for the message indicating /store cannot be mounted:
cat /var/log/remove_glusterfs.log | grep -i 'failed to mount store'
Jul 29 16:34:52 [ERROR] Failed to mount store: mount: wrong fs type, bad option, bad superblock on /dev/sda8, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. - Run these commands and compare the filesystem types:
cat /etc/fstab | grep -ivE "storetmp|transient" | grep -i store blkid /store
cat /etc/fstab | grep -ivE "storetmp|transient" | grep -i store UUID=882500c4-a465-4efa-9b5a-d001f0d58dbd /store ext4 defaults 1 2
blkid /store /dev/sda8: UUID="e0501d2d-201f-47a6-ac75-0778f4e86333" TYPE="xfs" PARTLABEL="/store" PARTUUID="0f1ee5d8-fd68-47bf-8a91-ec15e9d90d68"
Resolving The Problem
- Get the new UUID of the /store partition:
blkid /store /dev/sda8: UUID="e0501d2d-201f-47a6-ac75-0778f4e86333" TYPE="xfs" PARTLABEL="/store" PARTUUID="0f1ee5d8-fd68-47bf-8a91-ec15e9d90d68"
- Create backup file of /etc/fstab file:
mkdir /store/ibm_support cp -p /etc/fstab /store/ibm_support/fstab.bkp
- Edit and update the /etc/fstab file's entry for the /store partition so it uses the new UUID, the correct filesystem type, and the filesystem options, so the entry looks like this:
UUID=e0501d2d-201f-47a6-ac75-0778f4e86333 /store xfs defaults 0 0
- Mount the /store partition manually:
mount -a
- Copy back the content of /storetmp/backup/glusterbackup to /store:
cp -a /storetmp/backup/glusterbackup/. /store
- Run the migration script on the Event Collector:
/opt/qradar/ha/bin/glusterfs_migration_manager-<script_version>.bin --migrate
-
If the script is successful, then the output looks as follows:
[WARNING] Could not locate store on LVM. Upgraded system detected
[INFO] Found /store on /dev/sda8
[INFO] Running: updating_values
[INFO] Updated Configuration values on EC
- Once the script finishes, reboot the appliance
- Wait for 10 minutes and check whether all the services are running with the help of these commands:
systemctl status hostservices
/opt/qradar/upgrade/util/setup/upgrades/wait_for_start.sh
The words LINSTOR®, DRBD®, LINBIT®, and the logo LINSTOR®, DRBD®, and LINBIT® are trademarks or registered trademarks of LINBIT in Austria, the United States, and other countries.
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.4.2"}]
Was this topic helpful?
Document Information
Modified date:
04 August 2021
UID
ibm16476930