Security Bulletin: Publicly disclosed vulnerabilities from Kernel affect IBM Netezza Host Management

Workarounds and Mitigations

Mitigation of the reported CVEs : CVE-2019-19448, CVE-2019-19816, CVE-2019-19377, CVE-2019-19378, CVE-2019-19965 is blocklisting kernel modules btrfs, libsas to prevent them from loading automatically on PureData System for Analytics N200x and N3001 is as follows:

  1. Change to user nz:
      [root@nzhost1 ~]# su – nz

  2. Check to see if Call Home is enabled:
      [nz@nzhost1 ~]$ nzcallhome -status
      If enabled, disable it:
      [nz@nzhost1 ~]$ nzcallhome –off
      Note: Ensure that nzcallhome returns status as disabled. If there are errors in the callHome.txt configuration file, errors             are listed in the output, and call-Home is disabled.

  3. Check the state of the Netezza system:
      [nz@nzhost1 ~]$ nzstate

  4. If the system state is online, stop the system using the command:
      [nz@nzhost1 ~]$ nzstop

  5. Wait for the system to stop, using the command:
      [nz@nzhos1t ~]$ nzstate
      System state is 'Stopped'.

  6. Exit from the nz session to return to user root:
      [nz@nzhost1 ~]$ exit

  7. Logged into the active host as root, type the following commands to stop the heartbeat processes:
      [root@nzhost1 ~]# ssh ha2 /sbin/service heartbeat stop
      [root@nzhost1 ~]# /sbin/service heartbeat stop

  8. Run below commands as a root user to disable heartbeat from startup:
      [root@nzhost1 ~]# ssh ha2 /sbin/chkconfig heartbeat off
      [root@nzhost1 ~]# /sbin/chkconfig heartbeat off

  9. Type the following commands to stop the DRBD processes:
      [root@nzhost1 ~]# ssh ha2 /sbin/service drbd stop
      [root@nzhost1 ~]# /sbin/service drbd stop

  10. Run below commands as a root user to disable drbd from startup:
      [root@nzhost1 ~]# ssh ha2 /sbin/chkconfig drbd off
      [root@nzhost1 ~]# /sbin/chkconfig drbd off

Execute below steps using "root" user on both ha1/ha2 hosts

Step 1: Check if kernel modules btrfs, libsas are loaded in the hosts

lsmod | grep btrfs
lsmod | grep libsas

example:
[root@ nzhost1 ~]# lsmod | grep btrfs
btrfs 787404 0
zlib_deflate 21661 1 btrfs
lzo_decompress 2343 1 btrfs
lzo_compress 2368 1 btrfs
libcrc32c 1246 1 btrfs
[root@ nzhost1 ~]# lsmod | grep libsas
libsas 74610 0
scsi_transport_sas 35620 1 libsas

Note: No output on Step 1 for any module indicates, that module is not loaded hence skip Step 2 for that module, and proceed with Step 3

Step 2: Unload kernel modules are btrfs, libsas if they are loaded

modprobe -rv btrfs
modprobe -rv libsas

example:
[root@nzhost1 ~]# modprobe -rv btrfs
rmmod /lib/modules/2.6.32-754.35.1.el6.x86_64/kernel/fs/btrfs/btrfs.ko
rmmod /lib/modules/2.6.32-754.35.1.el6.x86_64/kernel/lib/zlib_deflate/zlib_deflate.ko
rmmod /lib/modules/2.6.32-754.35.1.el6.x86_64/kernel/lib/lzo/lzo_decompress.ko
rmmod /lib/modules/2.6.32-754.35.1.el6.x86_64/kernel/lib/lzo/lzo_compress.ko
rmmod /lib/modules/2.6.32-754.35.1.el6.x86_64/kernel/lib/libcrc32c.ko
[root@nzhost1 ~]# modprobe -rv libsas
rmmod /lib/modules/2.6.32-754.35.1.el6.x86_64/kernel/drivers/scsi/libsas/libsas.ko
rmmod /lib/modules/2.6.32-754.35.1.el6.x86_64/kernel/drivers/scsi/scsi_transport_sas.ko

Kernel modules and their dependent modules will be unloaded in the reverse order that they are loaded, given that no processes depend on any of the modules being unloaded.

Step 3: To prevent modules from being loaded directly you add the blocklist line to a configuration file specific to the system configuration.

echo "blocklist btrfs" >> /etc/modprobe.d/local-blocklist.conf
echo "blocklist libsas" >> /etc/modprobe.d/local-blocklist.conf

example :
[root@nzhost1 ~]# echo "blocklist btrfs" >> /etc/modprobe.d/local-blocklist.conf
[root@nzhost1 ~]# echo "blocklist libsas" >> /etc/modprobe.d/local-blocklist.conf
[root@nzhost1 ~]# cat /etc/modprobe.d/local-blocklist.conf | grep btrfs
blocklist btrfs
[root@nzhost1 ~]# cat /etc/modprobe.d/local-blocklist.conf | grep libsas
blocklist libsas

Step 4: Kernel modules can be loaded directly or loaded as a dependency from another module
To prevent installation as a dependency from another module follow below step:

echo "install btrfs /bin/false" >> /etc/modprobe.d/local-blocklist.conf
echo "install libsas /bin/false" >> /etc/modprobe.d/local-blocklist.conf

example:
[root@nzhost1 ~]# echo "install btrfs /bin/false" >> /etc/modprobe.d/local-blocklist.conf
[root@nzhost1 ~]# echo "install libsas /bin/false" >> /etc/modprobe.d/local-blocklist.conf
[root@nzhost1 ~]# cat /etc/modprobe.d/local-blocklist.conf | grep btrfs
blocklist btrfs
install btrfs /bin/false
[root@nzhost1 ~]# cat /etc/modprobe.d/local-blocklist.conf | grep libsas
blocklist libsas
install libsas /bin/false

The install line simply causes /bin/false to be run instead of installing a module.

Step 5: Make a backup copy of your initramfs.

cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.$(date +%m-%d-%H%M%S).bak

Example:
[root@nzhost1 ~]# cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.$(date +%m-%d-%H%M%S).bak
[root@nzhost1 ~]# uname -r
2.6.32-754.35.1.el6.x86_64
[root@nzhost1 ~]# ll /boot/initramfs-2.6.32-754.35.1.el6.x86_64.img.11-12-051237.bak
-rw------- 1 root root 22554174 Nov 12 05:12 /boot/initramfs-2.6.32-754.35.1.el6.x86_64.img.11-12-051237.bak

Step 6: If the kernel module is part of the initramfs (boot configuration), rebuild your initial ramdisk image, omitting the module to be avoided

dracut --omit-drivers btrfs -f
dracut --omit-drivers libsas -f

example:
[root@nzhost1 ~]# dracut --omit-drivers btrfs -f
[root@nzhost1 ~]# dracut --omit-drivers libsas -f
[root@nzhost1 ~]# lsinitrd /boot/initramfs-2.6.32-754.35.1.el6.x86_64.img | grep btrfs
[root@nzhost1 ~]# lsinitrd /boot/initramfs-2.6.32-754.35.1.el6.x86_64.img | grep libsas

Step 7: Append module_name.blocklist to the kernel cmdline. We give it an invalid parameter of blocklist and set it to 1 as a way to preclude the kernel from loading it.

sed --follow-symlinks -i '/\s*kernel \/vmlinuz/s/$/ btrfs.blocklist=1/' /etc/grub.conf
sed --follow-symlinks -i '/\s*kernel \/vmlinuz/s/$/ libsas.blocklist=1/' /etc/grub.conf

example :
[root@nzhost1 ~]# sed --follow-symlinks -i '/\s*kernel \/vmlinuz/s/$/ btrfs.blocklist=1/' /etc/grub.conf
[root@nzhost1 ~]# sed --follow-symlinks -i '/\s*kernel \/vmlinuz/s/$/ libsas.blocklist=1/' /etc/grub.conf

Step 8: blocklist the kernel module in kdump's configuration file.

echo "blocklist btrfs" >> /etc/kdump.conf
echo "blocklist libsas" >> /etc/kdump.conf

example:
[root@nzhost1 ~]# echo "blocklist btrfs" >> /etc/kdump.conf
[root@nzhost1 ~]# echo "blocklist libsas" >> /etc/kdump.conf
[root@nzhost1 ~]# cat /etc/kdump.conf | grep btrfs
blocklist btrfs
[root@nzhost1 ~]# cat /etc/kdump.conf | grep libsas
blocklist libsas

Note: Perform Step 9 if kexec-tools is installed and kdump is configured else continue with Step 10.
Perform below commands to check if kexec-tools is installed and Kdump is operational
[root@nzhost1 ~]# rpm -qa | grep kexec-tools
[root@nzhost1 ~]# service kdump status

Step 9: Restart the kdump service to pick up the changes to kdump's initrd.

service kdump restart

example:
[root@nzhost1 ~]# service kdump restart
Stopping kdump: [ OK ]
Detected change(s) the following file(s):

/etc/kdump.conf
Rebuilding /boot/initrd-2.6.32-754.31.1.el6.x86_64kdump.img
Starting kdump: [ OK ]

Step 10: Reboot the system at a convenient time to have the changes take effect.
Make sure the secondary host is up by pinging or logging in before rebooting the primary host.

/sbin/shutdown -r now

example:
[root@nzhost1 ~]# /sbin/shutdown -r now
Make sure the primary server comes up and is reachable before performing Mitigation steps on the secondary server.

  After applying the mitigation:

      1. Start the services using following:
          [root@nzhost1 ~]# service heartbeat start
          [root@nzhost1 ~]# ssh ha2 service heartbeat start
          [root@nzhost1 ~]# service drbd start
          [root@nzhost1 ~]# ssh ha2 service drbd start

      2. Check the stat of the system. Type:
          [root@nzhost1 ~]# crm_mon -i5

          Result: When the cluster manager comes up and is ready, status appears as follows.
          Make sure that nzinit has started before you proceed. (This could take a few minutes.)
          Node: nps61074 (e890696b-ab7b-42c0-9e91-4c1cdacbe3f9): online
          Node: nps61068 (72043b2e-9217-4666-be6f-79923aef2958): online
          Resource Group: nps
          drbd_exphome_device(heartbeat:drbddisk): Started nps61074
          drbd_nz_device(heartbeat:drbddisk): Started nps61074
          exphome_filesystem(heartbeat::ocf:Filesystem): Started nps61074
          nz_filesystem (heartbeat::ocf:Filesystem): Started nps61074
          fabric_ip (heartbeat::ocf:IPaddr): Started nps61074
          wall_ip (heartbeat::ocf:IPaddr): Started nps61074
          nzinit (lsb:nzinit): Started nps61074
          fencing_route_to_ha1(stonith:apcmaster): Started nps61074
          fencing_route_to_ha2(stonith:apcmaster): Started nps61068

      3. From host 1 (ha1), press Ctrl+C to break out of crm_mon.

      4. Turn on heartbeat and DRBD using the chkconfig:
          ssh ha2 /sbin/chkconfig drbd on
          /sbin/chkconfig drbd on
          ssh ha2 /sbin/chkconfig heartbeat on
          /sbin/chkconfig heartbeat on