Troubleshooting
Problem
Administrators who upgrade to QRadar® 7.4.0 Patch 3 can experience a Red Hat kernel issue where appliances are unable to mount the filesystem or properly boot as documented in APAR IJ25612. Administrators can experience this issue on a per appliance basis. To assist users in identifying this issue, QRadar development has created an identification utility that can be run on appliances to identify potential issues.
Symptom
Administrators who install QRadar® 7.4.0 Fix Pack 3 can install the fix pack, but the appliance reboots fails with a number of XFS errors where the filesystem cannot be mounted. This issue is identified as APAR IJ25612 by QRadar Support.
Error message:
![image 4420](/support/pages/system/files/inline-images/image_4420.png)
Figure 1: After the fix pack installation, an appliance reboot can fail the XFS filesystem check due to Red Hat kernel issues.
Error message:
XFS (dm-0): SB strip unit sanity check failed XFS (dm-0): SB validate failed with error -117.
Screen capture:![image 4420](/support/pages/system/files/inline-images/image_4420.png)
Figure 1: After the fix pack installation, an appliance reboot can fail the XFS filesystem check due to Red Hat kernel issues.
Cause
This issue is identified as a problem in Red Hat kernel 3.10.0-1127.el7.x84_64 as identified in Red Hat Bug Advisory: RHBA-2020:2355 - xfs: catch bad stripe alignment configurations.
Environment
QRadar® versions that contain kernel 3.10.0-1127.el7.x86_64:
- QRadar 7.4.0 Fix Pack 3
- QRadar 7.4.0 Fix Pack 2
- QRadar 7.3.3 Fix Pack 3
Diagnosing The Problem
Before administrators upgrade to QRadar® 7.4.0 Fix Pack 3, they can run the IJ25612.tgz utility to identify appliances that can experience this kernel issue. The utility must be run on each QRadar appliance to determine if the host passes or fails the test. The utility identifies whether the minimum IO size exceeds the optimal IO size for devices with an optimal IO size greater than zero.
Procedure
Procedure
- Download the support utility: IJ25612.tgz
- Copy the file to the QRadar Console.
- Confirm the MD5 / 256 sum for the file:
MD5 - 248ed6814e9675947e1d5843d25607da IJ25612.tgz
SHA256 - 825fee9b8f871574bf73e809255ca44d55436678f3616a45498e9c8ec518b024 IJ25612.tgz - To extract the file, type:
tar zxvf IJ25612.tgz
- Set permissions on the file with the following command:
chmod +x IJ25612.sh
- To copy this file to all QRadar appliances, type:
/opt/qradar/support/all_servers.sh -kp /root/IJ25612.sh
- Use SSH to open a connection to each host in the deployment.
- Navigate to the /storetmp directory.
- Run IJ25612.sh.
Note: You must run the utility on each QRadar appliance to diagnose any impacted hosts. - Review the output to determine whether your appliance is impacted by IJ25612.
Affected appliance output
Failed checks indicate the appliance is impacted by the Red Hat kernel issue identified in IJ25612. If you upgraded to QRadar 7.4.0 Fix Pack 3, you can reboot the appliance and select the previous kernel version in the grub menu. If an appliance reports as affected, administrators can wait for a software resolution in the next fix pack to ensure you do not experience this issue.[root@qradar_examplehost tmp]# ./IJ25612.sh / Failed check /var/log Failed check /opt/qradar/bin/ca_jail/lib64 Failed check /opt/qradar/bin/ca_jail/lib Failed check /opt/qradar/bin/ca_jail/usr/share Failed check /opt/qradar/bin/ca_jail/usr/lib64 Failed check /opt/qradar/bin/ca_jail/usr/bin Failed check /opt/qradar/bin/ca_jail/bin Failed check /opt/qradar/bin/ca_jail/usr/lib Failed check ***************************************************************** ** IJ25612 ** ** xfs filesystems found that will result in failure to mount ** ** and cause the QRadar appliance to fail to boot ** ** This is due to the use of kernel 3.10.0-1127.el7.x86_64 ** ** as identified in the following note: ** ** https://access.redhat.com/solutions/5075561 ** ** ** ** Don't apply QRadar 7.4.0 FixPack 3 ** ** ** *****************************************************************
Not affected by IJ25612 output
Administrators who see the following output message can proceed with the installation of QRadar 7.4.0 Fix Pack 3. Administrators must confirm that each appliance reports this error message before the upgrade to QRadar 7.4.0 Fix Pack 3.No xfs filesystems found that will hit IJ25612 a bug in kernel 3.10.0-1127.el7.x86_64 You can proceed with the patch to QRadar 7.4.0 Fix Pack 3
Resolving The Problem
If the QRadar® appliance is booted into a previous version of the Red Hat kernel, the XFS file system can be mounted. If you upgraded your appliance and the XFS filesystem fails to mount after an upgrade to QRadar 7.4.0 Fix Pack 3, complete the following steps.
Procedure
Administrators must confirm that all appliances in the deployment mount the XFS filesystem after you upgrade and select the previous kernel version from the grub select menu.
- Reboot the QRadar® appliance.
- On the GRUB 2 boot screen, use the cursor and select Red Hat Enterprise Linux 3.10.0-1062.1.1.
- Press Enter to boot in to the previous kernel.
- Verify the appliance reboots and successfully mounts the XFS filesystem.
- Repeat this procedure for each appliance that experiences issue IJ25612.
Results
If you experience any issues, administrators can contact IBM QRadar Support for assistance.
Document Location
Worldwide
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000Gnc6AAC","label":"QRadar->Upgrade"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.3.3;7.4.0","Line of Business":{"code":"LOB24","label":"Security Software"}}]
Was this topic helpful?
Document Information
Modified date:
22 June 2020
UID
ibm16235774