IBM Support

QRadar: GlusterFS migration script encounters a "Failed to mount store" error

Troubleshooting


Problem

The QRadar® upgrade to version 7.4.2 requires you to run a migration script on the console. This script migrates the High Availability file system from GlusterFS to Distributed Replication Block Device on all Event Collectors in your deployment (irrespective of whether they are currently part of an HA setup or not).
In some rare scenarios, the script can fail on Event Collectors that were upgraded from versions prior to 7.3.x that used an ext4 partition for /store.

Symptom

When run, the migration script fails and displays this sequence of error messages:


Jul 29 16:34:52 [ERROR] Failed to mount store: mount: wrong fs type, bad option, bad superblock on /dev/sda8, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so.
Jul 29 16:34:52 [ERROR] Failed to resize /store on deployment. Check logs for more details
Jul 29 16:36:02 [ERROR] Unexpected error running run_prepare_ha: cannot concatenate 'str' and 'function' objects

Cause

The migration script creates a new xfs filesystem with space left for the Distributed Replication Block Device metadata.
Jul 29 16:34:44 [INFO] Resizing /store to make space for DRBD metadata
Jul 29 16:34:44 [INFO] Preparing /store resizing
Jul 29 16:34:44 [WARNING] Could not locate store on LVM. Upgraded system detected
Jul 29 16:34:44 [INFO] Found /store on /dev/sda8
Jul 29 16:34:50 [INFO] /store has been unmounted properly
Jul 29 16:34:52 [INFO] Running xfscmd mkfs.xfs -f -d size=51394048b /dev/sda8
Jul 29 16:34:52 [INFO] /store has been resized
However, for appliances that were migrated from older versions, the migration script checks for and finds an existing ext4 entry for /store in /etc/fstab and does not update the filesystem type in that entry.
When the script later attempts to mount /store by using the older /etc/fstab entry (with the ext4 filesystem), there is a filesystem mismatch (xfs vs ext4) and this error occurs:
Jul 29 16:34:52 [ERROR] Failed to mount store: mount: wrong fs type, bad option, bad superblock on /dev/sda8, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so.

Environment

QRadar® Event Collectors upgrading to 7.4.2

Diagnosing The Problem

When the script fails on one or more Event Collectors, check these points on each Event Collector:
  1. Check the log file for the message indicating /store cannot be mounted:
      
    cat /var/log/remove_glusterfs.log | grep -i 'failed to mount store'
    The log file has a message similar to:

    Jul 29 16:34:52 [ERROR] Failed to mount store: mount: wrong fs type, bad option, bad superblock on /dev/sda8, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so.
  2. Run these commands and compare the filesystem types:
      
    cat /etc/fstab |  grep -ivE "storetmp|transient" |  grep -i store
blkid /store
    The output of the cat command shows the filesystem type to be ext4 but the output of the blkid command shows the filesystem type to be xfs (notice the different UUIDs as well):
      
    cat /etc/fstab |  grep -ivE "storetmp|transient" |  grep -i store
UUID=882500c4-a465-4efa-9b5a-d001f0d58dbd     /store      ext4     defaults   1    2

    blkid /store
/dev/sda8: UUID="e0501d2d-201f-47a6-ac75-0778f4e86333" TYPE="xfs" PARTLABEL="/store" PARTUUID="0f1ee5d8-fd68-47bf-8a91-ec15e9d90d68"

Resolving The Problem

  1. Get the new UUID of the /store partition:
      
    blkid /store
/dev/sda8: UUID="e0501d2d-201f-47a6-ac75-0778f4e86333" TYPE="xfs" PARTLABEL="/store" PARTUUID="0f1ee5d8-fd68-47bf-8a91-ec15e9d90d68"
  2. Create backup file of /etc/fstab file:
      
    mkdir /store/ibm_support
cp -p /etc/fstab  /store/ibm_support/fstab.bkp
  3. Edit and update the /etc/fstab file's entry for the /store partition so it uses the new UUID, the correct filesystem type, and the filesystem options, so the entry looks like this:
      
    UUID=e0501d2d-201f-47a6-ac75-0778f4e86333  /store   xfs   defaults  0  0
  4. Mount the /store partition manually:
      
    mount -a
  5. Copy back the content of /storetmp/backup/glusterbackup to /store:
      
    cp -a /storetmp/backup/glusterbackup/.  /store
  6. Run the migration script on the Event Collector:
      
    /opt/qradar/ha/bin/glusterfs_migration_manager-<script_version>.bin  --migrate
  7. If the script is successful, then the output looks as follows:
     
    [WARNING] Could not locate store on LVM. Upgraded system detected
    [INFO] Found /store on /dev/sda8
    [INFO] Running: updating_values
    [INFO] Updated Configuration values on EC
     
  8. Once the script finishes, reboot the appliance
     
  9. Wait for 10 minutes and check whether all the services are running with the help of these commands:
      
    systemctl status hostservices
    /opt/qradar/upgrade/util/setup/upgrades/wait_for_start.sh
    The words LINSTOR®, DRBD®, LINBIT®, and the logo LINSTOR®, DRBD®, and LINBIT® are trademarks or registered trademarks of LINBIT in Austria, the United States, and other countries.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.4.2"}]

Document Information

Modified date:
04 August 2021

UID

ibm16476930

Page Feedback