IT36391: VSNAP CAN FAIL TO START AND BOOT INTO LINUX EMERGENCY MODE

APAR status

• Closed as program error.

Error description

• The primary symptom is that the vSnap server fails to boot and
  drops into emergency mode. The issue occurs in two possible
  situations:
  (1) If the vSnap was recently upgraded, the new
  kernel image did not correctly generate the boot image because
  of insufficient space in the /var/tmp partition. During the
  next reboot, the system cannot load the boot image because it
  is missing.
  (2) Even if the vSnap was not recently upgraded,
  the problem can occur if the system takes a long time to scan
  for devices during boot. If the scan takes too long, the boot
  process times out and drops into emergency mode.
  Both problems
  occur when the file /etc/modprobe.d/zfs.conf is very large.
  The
  problem can be confirmed by temporarily booting from an
  alternate kernel or by logging into the system as root in
  emergency mode and then checking the size of the file:
  
  du -sh
  /etc/modprobe.d/zfs.conf
  
  Under normal circumstances this file
  should be no more than a few kilobytes in size. If it is very
  large (few hundred MB or few GB), it can cause this
  issue.
  During every restart, the vSnap service regenerates this
  file by merging the user-specified parameters with some other
  dynamically-generated parameters. In the process of
  regenerating the file, vSnap inserts extra empty lines in the
  file which causes the size to grow. Over a long period of time
  as more reboots occur, the file can grow to be very large in
  size.
  
  Versions affected: 10.1.x
  
  Keywords: TS004786227 crash
  

Local fix

• -Delete the file /etc/modprode.d/zfs.conf. vSnap will
  regenerate a new copy upon next startup.
  -Regenerate the boot
  image using: dracut -f /boot/initramfs-$(uname -r).img $(uname
  -r)
  -Reboot
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus levels 10.1.5, 10.1.6 and 10.1.7.  *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply the fixing level when available. This problem was      *
  * fixed in IBM Spectrum Protect Plus level 10.1.8. Note that   *
  * this is subject to change at the discretion of IBM>          *
  ****************************************************************
  

Problem conclusion

• A code fix has been implemented to ensure extra newlines are
  removed whenever the file zfs.conf is regenerated. Additionally
  during vSnap upgrade, if the existing file is found to be very
  large in size, it is deleted completely and regenerated after
  upgrade. Some user-defined settings in the file may be lost if
  the file is deleted, but this step is necessary to prevent
  upgrade failures. Once the upgrade is complete and the fixed
  version is in place, the file will never be deleted again.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels