IBM Support

QRadar SOAR: Boot mount point is running out of available disk

Troubleshooting


Problem

There are times when the /boot mount point gets close to using the majority of its available disk. This might cause monitoring software to determine there is a problem.

Symptom

The mount point might grow in size getting close to 90%.
Filesystem   1K-blocks     Used  Available  Use% Mounted on
/dev/sda1       404128   353944      50184   88% /boot

Cause

When appliance security updates are installed on a RHEL SOAR appliance, any kernel updates are added to the /boot directory. This allows you to use a different kernel version at OS boot.
The OS, by default, keeps three versions of the kernel.
sudo grep "installonly_limit" /etc/yum.conf
installonly_limit=3
When a new appliance security update is installed, the new .img file is added and the oldest is removed. This keeps the number consistent at three.

Environment

RHEL 8 appliances are more likely to observe this behavior because the kernel files are larger than RHEL 7.

Diagnosing The Problem

Commands such a du and df allows you to identify the size of the directory and mount point.
The command ls shows you the files in that directory.
$ sudo ls -alh /boot

total 317M
dr-xr-xr-x. 5 root root 4.0K Jan 29 16:25 .
dr-xr-xr-x. 18 root root 237 Aug 21 11:34 ..
-rw-r--r--. 1 root root 196K Jul 20 2023 config-4.18.0-477.21.1.el8_8.x86_64
-rw-r--r--. 1 root root 198K Sep 29 18:26 config-4.18.0-513.5.1.el8_9.x86_64
-rw-r--r--. 1 root root 198K Nov 17 01:14 config-4.18.0-513.9.1.el8_9.x86_64
drwxr-xr-x. 3 root root  17 Aug 21 11:34 efi
drwx------. 4 root root  83 Jan 30 17:00 grub2
-rw-------. 1 root root 89M Aug 21 11:37 initramfs-0-rescue-800d6401762342afac74bee216142dde.img
-rw-------. 1 root root 31M Jan 30 00:53 initramfs-4.18.0-477.21.1.el8_8.x86_64.img
-rw-------. 1 root root 28M Jan 29 22:21 initramfs-4.18.0-477.21.1.el8_8.x86_64kdump.img
-rw-------. 1 root root 31M Jan 30 00:52 initramfs-4.18.0-513.5.1.el8_9.x86_64.img
-rw-------. 1 root root 28M Jan 29 16:13 initramfs-4.18.0-513.5.1.el8_9.x86_64kdump.img
-rw-------. 1 root root 31M Jan 29 16:22 initramfs-4.18.0-513.9.1.el8_9.x86_64.img
-rw-------. 1 root root 28M Jan 29 16:25 initramfs-4.18.0-513.9.1.el8_9.x86_64kdump.img
drwxr-xr-x. 3 root root  21 Aug 21 11:35 loader
lrwxrwxrwx. 1 root root  52 Aug 21 11:48 symvers-4.18.0-477.21.1.el8_8.x86_64.gz -> /lib/modules/4.18.0-477.21.1.el8_8.x86_64/symvers.gz
lrwxrwxrwx. 1 root root  51 Jan 30 00:51 symvers-4.18.0-513.5.1.el8_9.x86_64.gz -> /lib/modules/4.18.0-513.5.1.el8_9.x86_64/symvers.gz
lrwxrwxrwx. 1 root root  51 Jan 29 16:21 symvers-4.18.0-513.9.1.el8_9.x86_64.gz -> /lib/modules/4.18.0-513.9.1.el8_9.x86_64/symvers.gz
-rw-------. 1 root root 4.3M Jul 20 2023 System.map-4.18.0-477.21.1.el8_8.x86_64
-rw-------. 1 root root 4.3M Sep 29 18:26 System.map-4.18.0-513.5.1.el8_9.x86_64
-rw-------. 1 root root 4.3M Nov 17 01:14 System.map-4.18.0-513.9.1.el8_9.x86_64
-rwxr-xr-x. 1 root root 11M Aug 21 11:36 vmlinuz-0-rescue-800d6401762342afac74bee216142dde
-rwxr-xr-x. 1 root root 11M Jul 20 2023 vmlinuz-4.18.0-477.21.1.el8_8.x86_64
-rw-r--r--. 1 root root 173 Jul 20 2023 .vmlinuz-4.18.0-477.21.1.el8_8.x86_64.hmac
-rwxr-xr-x. 1 root root 11M Sep 29 18:26 vmlinuz-4.18.0-513.5.1.el8_9.x86_64
-rw-r--r--. 1 root root 172 Sep 29 18:26 .vmlinuz-4.18.0-513.5.1.el8_9.x86_64.hmac
-rwxr-xr-x. 1 root root 11M Nov 17 01:14 vmlinuz-4.18.0-513.9.1.el8_9.x86_64
-rw-r--r--. 1 root root 172 Nov 17 01:13 .vmlinuz-4.18.0-513.9.1.el8_9.x86_64.hmac

Resolving The Problem

The following extract from /boot shows that there are three kernel versions. All of which will use disk space but the amount of disk will not grow further because the oldest files will be replaced once a new kernel is installed.
-rw-------. 1 root root 31M Jan 30 00:53 initramfs-4.18.0-477.21.1.el8_8.x86_64.img
-rw-------. 1 root root 28M Jan 29 22:21 initramfs-4.18.0-477.21.1.el8_8.x86_64kdump.img
-rw-------. 1 root root 31M Jan 30 00:52 initramfs-4.18.0-513.5.1.el8_9.x86_64.img
-rw-------. 1 root root 28M Jan 29 16:13 initramfs-4.18.0-513.5.1.el8_9.x86_64kdump.img
-rw-------. 1 root root 31M Jan 29 16:22 initramfs-4.18.0-513.9.1.el8_9.x86_64.img
-rw-------. 1 root root 28M Jan 29 16:25 initramfs-4.18.0-513.9.1.el8_9.x86_64kdump.img
Action can be taken to remove the kdump.img files so that disk is returned to the OS. This process involves masking the kdump service.
sudo systemctl mask --now kdump
Each time an appliance security update is installed:
  1. The oldest kernel files, including kdump.img files are removed
  2. New kdump.img files are not created
All kdump.img files will be removed after masking the service and installing three appliance security updates. Approximately 90MB will be returned to the OS.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSA230","label":"IBM Security QRadar SOAR"},"ARM Category":[{"code":"a8m0z0000001h0WAAQ","label":"Installation \/ Upgrade-\u003EOperating System"}],"ARM Case Number":"TS015930969","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
09 April 2024

UID

ibm17147765

Page Feedback