Troubleshooting
Problem
There are times when the /boot mount point gets close to using the majority of its available disk. This might cause monitoring software to determine there is a problem.
Symptom
The mount point might grow in size getting close to 90%.
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 404128 353944 50184 88% /boot
Cause
When appliance security updates are installed on a RHEL SOAR appliance, any kernel updates are added to the /boot directory. This allows you to use a different kernel version at OS boot.
The OS, by default, keeps three versions of the kernel.
sudo grep "installonly_limit" /etc/yum.conf
installonly_limit=3
When a new appliance security update is installed, the new .img file is added and the oldest is removed. This keeps the number consistent at three.
Environment
RHEL 8 appliances are more likely to observe this behavior because the kernel files are larger than RHEL 7.
Diagnosing The Problem
Commands such a du and df allows you to identify the size of the directory and mount point.
The command ls shows you the files in that directory.
$ sudo ls -alh /boot
total 317M
dr-xr-xr-x. 5 root root 4.0K Jan 29 16:25 .
dr-xr-xr-x. 18 root root 237 Aug 21 11:34 ..
-rw-r--r--. 1 root root 196K Jul 20 2023 config-4.18.0-477.21.1.el8_8.x86_64
-rw-r--r--. 1 root root 198K Sep 29 18:26 config-4.18.0-513.5.1.el8_9.x86_64
-rw-r--r--. 1 root root 198K Nov 17 01:14 config-4.18.0-513.9.1.el8_9.x86_64
drwxr-xr-x. 3 root root 17 Aug 21 11:34 efi
drwx------. 4 root root 83 Jan 30 17:00 grub2
-rw-------. 1 root root 89M Aug 21 11:37 initramfs-0-rescue-800d6401762342afac74bee216142dde.img
-rw-------. 1 root root 31M Jan 30 00:53 initramfs-4.18.0-477.21.1.el8_8.x86_64.img
-rw-------. 1 root root 28M Jan 29 22:21 initramfs-4.18.0-477.21.1.el8_8.x86_64kdump.img
-rw-------. 1 root root 31M Jan 30 00:52 initramfs-4.18.0-513.5.1.el8_9.x86_64.img
-rw-------. 1 root root 28M Jan 29 16:13 initramfs-4.18.0-513.5.1.el8_9.x86_64kdump.img
-rw-------. 1 root root 31M Jan 29 16:22 initramfs-4.18.0-513.9.1.el8_9.x86_64.img
-rw-------. 1 root root 28M Jan 29 16:25 initramfs-4.18.0-513.9.1.el8_9.x86_64kdump.img
drwxr-xr-x. 3 root root 21 Aug 21 11:35 loader
lrwxrwxrwx. 1 root root 52 Aug 21 11:48 symvers-4.18.0-477.21.1.el8_8.x86_64.gz -> /lib/modules/4.18.0-477.21.1.el8_8.x86_64/symvers.gz
lrwxrwxrwx. 1 root root 51 Jan 30 00:51 symvers-4.18.0-513.5.1.el8_9.x86_64.gz -> /lib/modules/4.18.0-513.5.1.el8_9.x86_64/symvers.gz
lrwxrwxrwx. 1 root root 51 Jan 29 16:21 symvers-4.18.0-513.9.1.el8_9.x86_64.gz -> /lib/modules/4.18.0-513.9.1.el8_9.x86_64/symvers.gz
-rw-------. 1 root root 4.3M Jul 20 2023 System.map-4.18.0-477.21.1.el8_8.x86_64
-rw-------. 1 root root 4.3M Sep 29 18:26 System.map-4.18.0-513.5.1.el8_9.x86_64
-rw-------. 1 root root 4.3M Nov 17 01:14 System.map-4.18.0-513.9.1.el8_9.x86_64
-rwxr-xr-x. 1 root root 11M Aug 21 11:36 vmlinuz-0-rescue-800d6401762342afac74bee216142dde
-rwxr-xr-x. 1 root root 11M Jul 20 2023 vmlinuz-4.18.0-477.21.1.el8_8.x86_64
-rw-r--r--. 1 root root 173 Jul 20 2023 .vmlinuz-4.18.0-477.21.1.el8_8.x86_64.hmac
-rwxr-xr-x. 1 root root 11M Sep 29 18:26 vmlinuz-4.18.0-513.5.1.el8_9.x86_64
-rw-r--r--. 1 root root 172 Sep 29 18:26 .vmlinuz-4.18.0-513.5.1.el8_9.x86_64.hmac
-rwxr-xr-x. 1 root root 11M Nov 17 01:14 vmlinuz-4.18.0-513.9.1.el8_9.x86_64
-rw-r--r--. 1 root root 172 Nov 17 01:13 .vmlinuz-4.18.0-513.9.1.el8_9.x86_64.hmac
Resolving The Problem
The following extract from /boot shows that there are three kernel versions. All of which will use disk space but the amount of disk will not grow further because the oldest files will be replaced once a new kernel is installed.
-rw-------. 1 root root 31M Jan 30 00:53 initramfs-4.18.0-477.21.1.el8_8.x86_64.img
-rw-------. 1 root root 28M Jan 29 22:21 initramfs-4.18.0-477.21.1.el8_8.x86_64kdump.img
-rw-------. 1 root root 31M Jan 30 00:52 initramfs-4.18.0-513.5.1.el8_9.x86_64.img
-rw-------. 1 root root 28M Jan 29 16:13 initramfs-4.18.0-513.5.1.el8_9.x86_64kdump.img
-rw-------. 1 root root 31M Jan 29 16:22 initramfs-4.18.0-513.9.1.el8_9.x86_64.img
-rw-------. 1 root root 28M Jan 29 16:25 initramfs-4.18.0-513.9.1.el8_9.x86_64kdump.img
Action can be taken to remove the kdump.img files so that disk is returned to the OS. This process involves masking the kdump service.
sudo systemctl mask --now kdump
Each time an appliance security update is installed:
- The oldest kernel files, including kdump.img files are removed
- New kdump.img files are not created
All kdump.img files will be removed after masking the service and installing three appliance security updates. Approximately 90MB will be returned to the OS.
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSA230","label":"IBM Security QRadar SOAR"},"ARM Category":[{"code":"a8m0z0000001h0WAAQ","label":"Installation \/ Upgrade-\u003EOperating System"}],"ARM Case Number":"TS015930969","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
09 April 2024
UID
ibm17147765