Troubleshooting
Problem
Customers running SVC target mode of NPIV enabled might experience storage path loss, or read-only file system events on SUSE Linux Enterprise Server (SLES) and Red Hat Enterprise Linux (RHEL), when using the ibmvfc driver on IBM Power.
Symptom
During an SVC node update or restart, the VIO client might lose access to a all of the SVC storage node paths, resulting in a read-only file system event.
Customers might also see in some errors in /var/log/messages such as
Sep 4 04:44:41 hostname1 multipathd: 360060e801236380050403***************: remaining active paths: 1
Sep 4 04:44:41 hostname1 multipathd: 66:80: mark as failed
Sep 4 04:44:41 hostname1 multipathd: 360060e801236380050403***************: remaining active paths: 0
At more recent kernel levels, it seems to be more common to just see a gap in the /var/log/messages before the reset.
Multipath.conf settings can also affect how quickly we lose our paths entirely.
Cause
There are bugs within the ibmvfc driver code, and in the VIOS code. The result is timing problems when the WWPNs log out and back in, during an SVC node failover or restart, if NPIV target mode is enabled.
See the following SVC link for more information on NPIV target mode enablement:
Environment
- Linux distributions: SLES 12, SLES 15, RHEL 7, RHEL 8
- SVC/Storwize/FlashSystem storage with target mode set to NPIV
- IBM PowerVM
Diagnosing The Problem
* Check if SVC target mode is set to enabled. This is the default setting on SVC code version 8.2.0 and higher.
* Check the level of kernel you are using. This problem is seen on SLES 12 SP3, SP4, or SP5, SLES 15 SP0, SP1, and SP2, and on RHEL 7.5 through 8.1.
Here is a history of the different bugs and kernel versions where the bugs were fixed.
ibmvfc: do not send implicit logouts prior to NPIV login (bsc#1169625 ltc#184611).
SLES 12 SP3
SLES 12 SP4
SLES 12 SP5
SLES 15 LTSS
SLES 15 SP1
For Red Hat, the fixes have been added in these kernel versions:
Red Hat:
scsi: ibmvfc: Avoid loss of all paths during SVC node reboot (BZ#1830889) and (BZ#1866371)
RHEL 7.7
RHEL 8.1
scsi: ibmvfc: Avoid link down on FS9100 canister reboot
(bsc#1176962 ltc#188304).
(bsc#1176962 ltc#188304).
This bug is resolved in the following versions.
SLES 12 SP4 (LTSS) at kernel-default-4.12.14-95.68.1
SLES 12 SP5 at kernel-default-4.12.14-122.41.1
SLES 15 SP1 at kernel-default-4.12.14-197.61.1
Currently this bug is fixed in RHEL 8.4 (resolved in the 4.18.0-305 kernel)
There have been some more recent bug fixes relating to similar problems (as of June 2021).
- ibmvfc: Avoid move login if fast fail is enabled (bsc#1185938 ltc#192043).
- ibmvfc: Handle move login failure (bsc#1185938 ltc#192043).
- ibmvfc: Reinit target retries (bsc#1185938 ltc#192043).
These bugs are fixed in the following levels on SLES 12 and SLES 15 at:
SLES 12 SP4 LTSS (June 2021)
SLES 12 SP5 (June 2021)
SLES 15 SP1 (June 2021)
SLES 15 SP2 (June 2021)
RHEL 8.3 (Nov 2021)
https://access.redhat.com/errata/RHSA-2021:4056
Resolving The Problem
Update to the appropriate kernel and use the correct multipath configuration options, provided in the IBM Documentation:
It is also recommended to update VIOS to 3.1.2.21 plus interim fixes as of 14 June 2021:
3.1.2.21 + ifixes listed (ifix list updated)
+ new fix added: o add IJ32369 - LINUX CLIENT VIRTUAL NPIV DISKS MAY SEE I/O ERRORS
(required combined iFix for IJ32369 and IJ31936/IJ31191 (order via SW support)
+ added headsup for IJ32841: VIOSUPGRADE WITH 3.1.2.21 IMAGE MAY NOT RESTORE NPIV/VSCSI
Document Location
Worldwide
[{"Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SGMV168","label":"SUSE Linux Enterprise Server"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)"}]
Product Synonym
SVC storage node - NPIV ibmvfc - lost paths - readonly filesystem
Was this topic helpful?
Document Information
Modified date:
05 January 2022
UID
ibm16396140