NPIV Fibre Channel-NVMe over Fabrics Protocol Support

Learn about the support of Fibre Channel (FC)-Nonvolatile Memory express (NVMe) over Fabrics (NVMeoF) protocol in the PowerVM N_Port ID Virtualization (NPIV) stack. The NPIV-NVMeoF is a virtualization of physical NVMeoF protocol support. The NPIV stack supports the Small Computer Systems Interface (SCSI) and the NVMeoF protocols on a single virtual adapter if the following conditions are met:

The physical FC adapter port supports both SCSI and NVMeoF protocols.
The connectivity to SCSI and NVMeoF protocols is configured.
The software and hardware requirements that are mentioned in Table 1 are met.

By default, the NVMeoF protocol support is enabled in the physical Fibre Channel but disabled in the Virtual Fibre Channel (VFC) on Virtual I/O Server (VIOS). You can run ioscli vfcctrl command on the VIOS to enable the NPIV-NVMeoF protocol support.

A single physical port on VIOS exports both the SCSI and FC-NVMeoF protocols to the client partition. Each protocol has its own end-to-end I/O queues. The client partition can identify disks from both protocols and can perform the I/O operations on both types of disks at the same time. NPIV-NVMeoF enablement for boot support is also added in firmware version FW 1030, or later.

The FC-NVMe devices are fast and supported by multiple-queue environments only. All the requirements that are applicable to the multiple-queue feature for NPIV are also applicable for the NVMeoF protocol.

Hardware and software requirements to enable NVMeoF protocol for the NPIV stack

Table 1. Requirements for NPIV-NVMeoF protocol support
Operating system or PFW	Supported versions
AIX®	Version 7.3 Technology Level 01, or later
Virtual I/O Server	Version 3.1.4.0, or later
POWER® firmware	Version 1030, or later
Fibre Channel (FC) adapter	Emulex FC 32 Gb or 64 Gb FC adapters or any high-bandwidth FC adapters that support NVMeoF protocol in AIX physical stack.
IBM® i	Not supported
Linux® Enterprise Server (SUSE, Red Hat®)	Not supported

Considerations for NPIV-NVMeoF configuration

The virtual adapter worldwide port names (WWPN) must be used for zoning, similar to FC SCSI.
Different types of storage can have different ways to connect a host initiator to target ports. You must follow the storage-specific documentation to configure storage side connectivity. For certain types of storage, the host NVMe Qualified Name (NQN) is required to configure the storage device. The PowerVM NVMeoF protocol capable host uses a known approach to form the host NQN. All the virtual adapters on a logical partition share same host NQN. The format of the host NQN is as follows:
nqn.2014-08.org.nvmexpress:uuid:<CLIENT_LPAR_UUID>, where
- The nqn.2014-08.org.nvmexpress:uuid string is static and defined in the NVMe express specification.
- The <CLIENT_LPAR_UUID> string is the Universal Unique Identifier (UUID) of the client partition. The data is represented as a string of hexadecimal characters, with hyphens added for readability. Hexadecimal values “a” through “f” are lowercase. For example, in nqn.2014-08.org.nvmexpress:uuid:2f772722-9ff6-4311-b58c-b736941041de NQN, the string 2f772722-9ff6-4311-b58c-b736941041de is the UUID of the client partition.
  Note:
  You can run lssyscfg command, which is a Hardware Management Console (HMC) command, to find the UUID of the client partition.

Enabling NVMeoF protocol in PowerVM NPIV stack

By default, the NVMeoF protocol support is disabled on VIOS. To enable the NVMeoF protocol in the NPIV stack, complete the following steps:

Configure switch and storage so that the host can access NVMe devices.
Decide whether you want to enable NVMe for all current and future adapters. If you want to enable NVMe for all adapters, complete the following steps:
- Change the dflt_enabl_nvme attribute value of viosnpiv0 pseudo device to yes.
- Set the enable_nvme attribute value to yes for all the VFC host devices.
If you want to enable NVMeoF protocol only on a few adapters, then you can change the enable_nvme value of VFC host device attribute to yes.

Note: You can run ioscli vfcctrl command to change the attribute values at the VFC host adapter level. You can change the value of viosnpiv0 attribute only by using the chdev command. For VFC host device, you can run ioscli vfcctrl command. You can also run the vfcctrl command to change the value for all VFC host devices for a specific client partition.
Unconfigure and reconfigure the virtual adapter on an AIX client partition so that it can discover NVMeoF devices. You do not have to reboot the partition to activate the NVMe devices if you are using multiple Virtual I/O Servers.

Live Partition Mobility with NPIV-NVMeoF support

The NPIV-NVMeoF protocol enablement requires support from the operating system that is running on a client partition, power hypervisor, and VIOS. The Live Partition Mobility (LPM) operation might fail if the client partition has access to the NPIV-NVMeoF protocol on the source side and if either the hypervisor or the VIOS on the destination system does not support NPIV-NVMeoF protocol.

Currently, the NPIV-NVMeoF protocol is supported only on AIX clients running with Version 7.3 Technology Level 1, or later and VIOS Version 3.1.4.0, or later. If the other client operating systems (such as Linux Enterprise Server and IBM i) support this protocol, LPM support extends to those clients as well.

Considerations for NPIV configuration and LPM validation

During the initial configuration, when you connect the NPIV client to the VIOS, the VIOS reports whether the NPIV-NVMeoF protocol is supported. Both the client and VIOS exchange their NVMeoF support and if the client is interested, the NVMeoF protocol is enabled for the client adapter. If the protocol is enabled for a client during an LPM operation, the VIOS validates whether such a client can migrate to the destination managed system. This validation ensures that the client continues to have access to the NPIV-NVMeoF storage on the destination managed system and does not allow the client partition to migrate to a destination managed system that does not have support or access to the same NVMeoF storage.
The current LPM functions for the FC-SCSI protocol support the target port level validation and the logical unit number (LUN) level validation. The LUN level validation is required because access to the target port does not always guarantee LUN access, as some LUNs might be masked. For the NVMe over FC specifications, the namespace (analogous to LUN in SCSI) access inherits the access of the namespace controller. Therefore, the validation of the access of the NPIV-NVMeoF protocol namespace controller ensures the accessibility of all the namespaces under it.
During the LPM validation, it is important to find any misconfiguration and to avoid data corruptions. Hence, you need to consider cases where the client partition that supports NPIV-NVMeoF protocol might be running on the source managed system, where the hardware or VIOS does not support NPIV-NVMeoF protocol. Therefore, the client partition might not have access to NVMeoF storage on the source managed system. However, the client partition can migrate to the destination managed system where the NPIV-NVMeoF is supported or configured. To avoid the client partition from gaining accidental access to the new protocol and to allow the client to migrate back to the original source (if it needs to), blocks may be placed for the NVMe or SCSI protocol by the LPM operation. In such cases, if you want to start using the NVMeoF protocol on the destination managed system, you need to unblock the protocol, if it is blocked. You can run the ioscli vfcctrl command to unblock the protocols that are blocked.
Note: Some storage target ports support both SCSI and NVMe protocols, hence the destination adapter might discover some controllers even in the absence of backing devices. As a result, the client partition cannot migrate back to the original source managed system but a block can protect from this type of undesirable configuration change.

VIOS/ AIX Tunable Attributes

The new VIOS tunable attributes are available in VIOS version 3.1.4.0, or later as part of the NPIV FC-NVMe support to provide flexibility with the number of NVMe FC adapter queues (physical queues) that each VFC host adapter uses. The NPIV multiple-queue feature also provides QoS type features and tunable attributes that are applicable to all the VFC host adapters. The num_per_nvme attribute can be set at the VIOS partition level and can be overridden at the individual VFC host adapter level.

The dflt_enabl_nvme attribute can be set at the VIOS partition level to decide the default value for the enable_nvme attribute for the VFC host device. The change in the attribute value affects only newly created VFC host devices after the change.

The partition-wide tunable attributes are provided by the viosnpiv0 device. The local tunable attributes are provided by the VFC host adapter device. The following tables describe the new NVMe-related attributes:

Table 2. viosnpiv0 device attributes
Attribute	Min value	Max value	Default value	Description
`num_per_nvme`	4	64	8	A VIOS-level tunable attribute that indicates the number of FC-NVMeoF queues that each VFC host uses.
`dflt_enabl_nvme`	-	-	No (Possible values are Yes or No)	A VIOS-level tunable attribute that controls the default value for the newly created VFC adapter. It is a global attribute and can be overridden by the adapter-level attribute.

Table 3. vfchost attributes
Attribute name	Min value	Max value	Default value	Description
`num_per_nvme`	4	64	0	If this tunable attribute is set to a nonzero value, it overrides the partition-wide attribute `num_per_nvme` of the viosnpiv0 device. If the tunable attribute value is 0, this attribute is not in effect.
`enable_nvme`	-	-	Empty string (Possible values are Yes or No)	This tunable attribute specifies whether NPIV-NVMeoF protocol is enabled or disabled by default. The local tunable attribute gets the default value from the `dflt_enabl_nvme` attribute of the viosnpi0 device. You can change this value by running the `ioscli vfcctrl` command. Note: This attribute is private to vfchost adapter and its value is decided based on the `dflt_enabl_nvme` attribute of the viosnpiv0 device hence it cannot be tracked under VIOS rules.

The local limit_intr attribute has the highest precedence. If the limit_intr attribute is set to false, the local num_per_range attribute is effective. When the local num_per_range attribute is not set, the partition wide num_per_range attribute is effective.

Note: The limit_intr attribute is applicable to both SCSI and NVME protocols.

The number of queues that a client uses depends on the FC adapter, FW level, client capabilities, and also on the VIOS level and the tunable attributes of the VFC host adapter. After a successful LPM operation, if the client is using multiple-queues, the local num_per_range attribute or the limit_intr attribute of the VFC host adapter is set on the destination managed system based on the value that is used at the source managed system.

Table 4. New AIX VFC client tunable attributes
Attribute name	Min value	Max value	Default value	Description
`num_nvme_queues`	1	16	4	Determines the number of I/O queues that are used in the FC NVMe I/O communication.

Note:

The number of queues that the client NPIV-NVMeoF protocol uses depends on several factors such as the tunable attribute of the client virtual FC adapter queues (num_nvme_queues), the tunable attribute of FC adapter queues (num_nvme_queues), and the tunable attributes of the VFC host adapter queues (num_per_nvme). During the initial configuration, the VFC client negotiates the number of queues with the VFC host and configures the minimum value of num_nvme_queues attribute and the number of queues that are reported by the VFC host.
The current physical adapters support a total of 32 queues (16 SCSI queues and 16 NVMe queues) however a virtual adapter cannot support all 32 queues at the same time. Only 18 total queues are supported in the virtual mode. You can choose any combinations of queues.
In the initial releases of AIX, Live Kernel Update (LKU) feature was not supported for partitions that were configured with NPIV-FC-NVMe disks.
On AIX, the VFC resources are divided between SCSI and NVMe protocol. The default setting favors SCSI protocol over NVMe. If you want to improve the performance of the NVMe protocol, you might have to disable the SCSI protocol.
On AIX, the nchan attribute at the NVMe controller level controls the number of queues that are allocated per path, which might improve performance due to the additional parallelization of I/Os. The higher the value of the nchan attribute, the more resources that are used. Hence, increasing the nchan attribute value might limit the number of paths discovered due to the lack of resources.