IT31792: CTGGA3054 - SQL BACKUP FAILS TO MOUNT BACKUP TARGET VOLUME

APAR status

• Closed as program error.

Error description

• The SQL backups are failing with following messages at the
  Spectrum Protect Plus job log:
  CTGGA2244,Mapping lun on volume spp_1010_2110_123456abcde to
  host (name:IBM-VM-SQL
  initiator:abc.234.com.microsoft:IBM-VM-SQL.xx.yyyy.com)
  CTGGA2245,SPP log dir:
  /data/log/guestdeployer/2019-12-07/1234009900000/1020304050607/I
  BM-VM-SQL.xx.yyyy.com
  CTGGA3052,[IBM-VM-SQL.xx.yyyy.com] Start mount command.
  CTGGA3054,[IBM-VM-SQL.xx.yyyy.com] Failed to mount for
  volume[spp_1010_2110_123456abcde]  error Volume id/name is not
  found.
  CTGGA3053,[IBM-VM-SQL.xx.yyyy.com] Mount command complete
  CTGGA1987,[IBM-VM-SQL.xx.yyyy.com] Unmapping lun on volume
  spp_1010_2110_123456abcde from host for
  initiator:iqn.1991-05.com.microsoft:IBM-VM-SQL.xx.yyyy.com
  CTGGA2073,Failed to perform backup of db instance IBM-VM-SQL on
  application server IBM-VM-SQL.xx.yyyy.com : Failed to mount
  backup target volume
  .
  At the vSnap server used for the SQL backup, the message log
  indicates following:
  Dec 18 03.45:00 VSNAP kernel: WARNING: CPU: 0 PID: 8 at
  kernel/workqueue.c:2911 __flush_work.isra.30+0x1be/0x1d0
  [... truncated ...]
  Dec 18 03.45:00 VSNAP kernel: Call Trace:
  Dec 18 03.45:00 VSNAP kernel: ? target_show_cmd+0x23d/0x280
  [target_core_mod]
  Dec 18 03.45:00 VSNAP kernel: __cancel_work_timer+0x103/0x190
  Dec 18 03.45:00 VSNAP kernel: ? __switch_to_asm+0x41/0x70
  Dec 18 03.45:00 VSNAP kernel: ? __switch_to_asm+0x35/0x70
  Dec 18 03.45:00 VSNAP kernel: ? __switch_to_asm+0x41/0x70
  Dec 18 03.45:00 VSNAP kernel: ? __switch_to_asm+0x35/0x70
  Dec 18 03.45:00 VSNAP kernel: core_tmr_lun_reset+0x551/0x7a0
  [target_core_mod]
  Dec 18 03.45:00 VSNAP kernel: ? __switch_to_asm+0x35/0x70
  Dec 18 03.45:00 VSNAP kernel: target_tmr_work+0x106/0x150
  [target_core_mod]
  Dec 18 03.45:00 VSNAP kernel: process_one_work+0x171/0x370
  Dec 18 03.45:00 VSNAP kernel: worker_thread+0x49/0x3f0
  Dec 18 03.45:00 VSNAP kernel: kthread+0xf8/0x130
  Dec 18 03.45:00 VSNAP kernel: ? max_active_store+0x80/0x80
  Dec 18 03.45:00 VSNAP kernel: ? kthread_bind+0x10/0x10
  Dec 18 03.45:00 VSNAP kernel: ret_from_fork+0x35/0x40
  .
  That kernel issue is causing the LIO (Linux-IO) subsystem on the
  vSnap server to get hang. Since LIO is handing the iSCSI
  traffic,
  the SQL backup were failing because the iSCSI mount from the
  vSnap was failing.
  
  IBM Spectrum Protect Versions Affected:
  IBM Spectrum Protect Plus 10.1.4.x
  Initial Impact: Medium
  
  Additional Keywords: SPP, SPPLUS, TS002978359
  

Local fix

• Reboot of the vSnap server used for the SQL backup.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus level 10.1.4 and 10.1.5.           *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in IBM Spectrum Protect Plus levels    *
  * 10.1.5 patch1 and 10.1.6. Note that this is subject to       *
  * change at the discretion of IBM.                             *
  ****************************************************************
  

Problem conclusion

• The newer Linux kernel 4.19 incorporated into vSnap contained
  some bugs introduced by upstream Linux kernel developers. Under
  certain conditions these bugs caused hangs of the LIO (Linux-IO)
  subsystem which is responsible for handling vSnap iSCSI traffic.
  
  The bugs were subsequently fixed in a later version of the
  upstream Linux kernel. The problem has been resolved by
  incorporating this newer kernel into vSnap.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels