IT44377: FAIL TO UPLOAD SYSTEM LOGS (SNAP) FROM STORAGE INSIGHTS DUE TO CLI COMMAND OUTPUT

APAR status

• Closed as program error.

Error description

• When using Storage Insights to collect and upload system logs,
  the following error can occur.  It was observed with Storage
  Virtualize.
  
  Uploading system logs for system <device_name> to ticket
  <ticket_#>
  The log package for the device with serial number <serial_#> for
  ticket <ticket_#> was not created, because the CLI Command
  issued for the storage system failed.
  
  When running the svc_snap command to collect the snap from the
  storage CLI, the following errors were observed although the
  snap was successfully created on the system.
  
  IBM_FlashSystem:device:user>svc_snap -gui1
  Collecting data
  df: /run/do_usb_16489: Input/output error
  df: /run/do_usb_16489: Input/output error
  df: /run/do_usb_16489: Input/output error
  df: /run/do_usb_16489: Input/output error
  df: /run/do_usb_16489: Input/output error
  ...
  df: /run/do_usb_16489: Input/output error
  df: /run/do_usb_16489: Input/output error
  df: /run/do_usb_16489: Input/output error
  df: /run/do_usb_16489: Input/output error
  Packaging files
  Snap data collected in /dumps/snap.xxxxxxxxxxxx.tgz
  IBM_FlashSystem:device:user>
  
  The unexpected error output from the svc_snap command causes the
  log upload to fail.  The errors should be resolved on the
  storage side.  See local fix.
  
  A subsequent attempt to upload the system logs from Storage
  Insights with "Add existing package" selected should allow you
  to select the snap that was created and upload it to the case.
  

Local fix

• The input/output error is related to the USB encryption keys.
  Remove keys from the nodes, one at a time, wait 2 minutes and
  reinsert.
  
  
  Note: Keys only have to be inserted during system boot for
  encryption.  Removing them will not affect system operation.
  
  Run the CLI svc_snap again, and check for "df:
  /run/do_usb_xxxxx: Input/output error" messages.
  
  If the error messages reoccur, reboot each node, one at at time,
  starting with non-config node, wait 30 minutes, verify node back
  online and then reboot config node.
  
  Then run CLI snap again and check for messages.  If the errors
  are all gone, retry log collection from Storage Insights.
  
  If you have concern about doing this action on your storage
  device, open a support case for the storage for advisement on
  the errors you see from the CLI.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Storage Insights users uploading support data for        *
  * storage device                                               *
  *                                                              *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * When using Storage Insights to collect and upload storage    *
  * device system logs, the following error can occur.  It was   *
  * observed with Storage Virtualize.                            *
  *                                                              *
  * Uploading system logs for system <device_name> to            *
  * ticket <ticket_#>                                            *
  * The log package for the device with serial number            *
  * <serial_#> for ticket <ticket_#> was not created, because    *
  * the CLI Command issued for the storage system failed.        *
  *                                                              *
  * When running the svc_snap command to collect the snap        *
  * from the storage CLI, the following errors were observed,    *
  * although the snap was successfully created on the system.    *
  *                                                              *
  * IBM_FlashSystem:device:user>svc_snap -gui1                   *
  * Collecting data                                              *
  * df: /run/do_usb_16489: Input/output error                    *
  * df: /run/do_usb_16489: Input/output error                    *
  * df: /run/do_usb_16489: Input/output error                    *
  * df: /run/do_usb_16489: Input/output error                    *
  * df: /run/do_usb_16489: Input/output error                    *
  * ...                                                          *
  * df: /run/do_usb_16489: Input/output error                    *
  * df: /run/do_usb_16489: Input/output error                    *
  * df: /run/do_usb_16489: Input/output error                    *
  * df: /run/do_usb_16489: Input/output error                    *
  * Packaging files                                              *
  * Snap data collected in /dumps/snap.xxxxxxxxxxxx.tgz          *
  * IBM_FlashSystem:device:user>                                 *
  *                                                              *
  * The unexpected error output from the svc_snap command        *
  * causes the log upload to fail.                               *
  * The errors should be resolved on the storage side.           *
  *                                                              *
  * A subsequent attempt to upload the system logs from          *
  * Storage Insights with "Add existing package" selected        *
  * should allow you to select the snap that was created and     *
  *  upload it to the case.                                      *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  

Problem conclusion

• The fix for this APAR is targeted for the following release:
  
  IBM Storage Insights 4Q23   [ 54X-IBM-SI ]
  ( release target 4Q 2023 / October )
  
  The target dates for future releases do not represent a formal
  commitment by IBM. The dates are subject to change without
  notice.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  STORAGE INSIGHT

• Fixed component ID

  5608TPCSI

Applicable component levels