SAS Adapter Performance Boost with VIOS

SAS Adapter Performance Boost with VIOS

The SAS Adapter Performance Boost with VIOS is designed to enable 520-byte sector virtual disks in a VIOS configuration.

Overview of Using IBM i 520-byte sector disk through VIOS

Ever since the virtual SCSI support was introduced with IBM i 6.1, the virtual SCSI disk devices presented to client operating systems have been formatted to use a 512-byte disk sector size.  This was true whether the server partition was a Virtual I/O Server (VIOS) or an IBM i partition acting as a virtual storage server.  Because of the unique requirements to implement its Single Level Storage architecture, for every 512 bytes of data written to disk, an IBM i partition must also store 8 bytes of additional data.  Some storage devices, such as the DS8000 family of storage servers attached directly to an IBM i partition, provide a 520-byte sector format to allow space for that additional data.  However, when using other types of storage devices that have a 512-byte sector format, the IBM i operating system instead must use an additional sector for every 8 sectors written to disk in order to accommodate the extra bytes.  This results in 4608 bytes of disk space being consumed for every 4096 bytes of user data written.  The “9^th sector” is used to store the extra data that would otherwise have been written in each 520- byte sector by devices capable of that support.

The latest generation of IBM Serial-Attached SCSI (SAS) adapters is optimized for disk operations with data aligned on 4096-byte boundaries. This design produces a noticeable degradation of the performance of disk requests with data that is not aligned on such a boundary.  Specifically, virtual disk units provided to an IBM i client partition from a VIOS partition that is using SAS hdisks as the backing storage devices experience this performance degradation.  In this type of configuration the 4608-byte-aligned I/O requests initiated by the IBM i partition are passed to the SAS adapter with no change in alignment, resulting in less than optimal performance.

With IBM i 7.1 Technology Refresh 7 IBM has addressed this performance issue by enabling the VIOS to provide a 520-byte sector format virtual disk when backed by SAS hardware that is capable of supporting that format.  This enables the IBM i operating system to format and use these virtual disks the same as other 520-byte sector disks.  The SAS adapter optimizes the I/O as if it were attached directly to the IBM i partition instead of being in a VIOS configuration.

Requirements

When adding or changing hardware, always refer to the IBM Prerequisite website for the up-to-date specifics for all types of code levels needed for support of a feature.  Generally speaking, the requirements are as follows:

Hardware requirements

• Supported SAS adapter cards:
  • PCIe2 1.8 GB Cache RAID SAS adapter Tri-port 6 Gb (CCIN 57B5)
  • PCIe2 1.8 GB Cache RAID SAS adapter Tri-port 6 Gb CR (CCIN 57BB)
  • PCIe2 RAID SAS Adapter Dual-port 6 Gb (CCIN 57C4)
• All future SAS adapter cards introduced after January 1^st, 2014
• SAS physical disk media must be attached to a card from the list above, and must be configured in RAID sets
• Power system must be from the POWER7 family or later.  See the IBM Pre-req website for specific models supported.
• If using 2 VIOSes, each VIOS needs it's own storage.  There is no Multi-Path I/O support for IBM i with 2 VIOSes seeing the same SAS attached storage.

Software requirements

• VIOS 2.2.3.0, or later
• IBM i 7.1 Technology Refresh PTF Group Level 7, or later

Firmware requirements

• Adapter firmware:
  • PCIe2 SAS RAID cards:  53495350.03500041, or later

• System firmware FW780, or later – required only for 520-byte sector load source disk

VIOS Configuration

Creating RAID Arrays

For an IBM i configuration with VIOS, a virtual disk that supports 520-byte sectors can be created only from a physical volume (“hdisk”, in VIOS terminology) that represents a SAS RAID array.  Each eligible hdisk must be created from one or more physical SAS disk devices (“pdisks”) that have been grouped together into a RAID array.  Prior to being added to a RAID array, each pdisk must be formatted to use a RAID block size to make it capable of storing the additional 8 bytes of data per sector needed by the IBM i client partition. Virtual target devices backed by other types of disk storage -- such as logical volumes, logical devices from a shared storage pool, or other physical volume types -- do not support 520-byte sectors.

In the simplest configuration, one physical pdisk may be used to create a RAID 0 array which produces a single hdisk that will then be capable of supporting 520-byte sector I/O.  Multiple pdisks may also be grouped together to produce an hdisk with higher levels of protection, if desired.

A pdisk that is not added to a RAID array, but is instead formatted as a standalone SAS hdisk, will not support the IBM i 520-byte sector format.

See the Knowledge Center topic on SAS RAID Controllers for AIX for more information on managing SAS RAID arrays in the VIOS.

Creating Virtual Disks

Once a SAS RAID array hdisk has been created, a virtual disk can be constructed from that device and assigned to a server vhost device using the mkvdev command.  For example, the following command creates a virtual disk named “vtscsi0” from a physical volume named “hdisk0” and attaches the virtual device to virtual adapter “vhost0”:

          $ mkvdev –vdev hdisk0 –vadapter vhost

          vtscsi0 available

See the Knowledge Center topic on VIOS and IVM Commands for more information on the mkvdev command.

Only virtual disk devices created from a RAID array on a supported SAS adapter with the appropriate microcode and software levels will be seen by the IBM i client partition as 520-byte sector capable disk units.  All others will remain 512-byte sector disks.

Preventing virtual disks from using 520-byte sectors when otherwise capable

The VIOS provides a global configuration mechanism that instructs the IBM i partition to ignore the hardware capability of 520-byte sectors.  With this mechanism, IBM i will  continue to format disks to use 512 bytes per sector.  The configuration setting is the "allow520blocks" attribute of the vios0 device.  The value of the attribute is set permanently until changed, and is then applied to all virtual SCSI disk devices created after the setting is changed.  To change the setting, use the following command on the VIOS:

              chdev -dev vios0 -attr allow520blocks=[yes | no] -perm

The current value of the allow520blocks setting can be determined using the lsdev command:

      $ lsdev -dev vios0 -attr

      attribute         value    description                               user_settable

      allow520blocks    no       Allow 520-byte blocks where applicable    True

The default value when the VIOS is first installed is "yes", which allows 520-byte sector capable devices to be used in that format.

It is important to note that virtual disk devices created prior to a change to the global allow520blocks attribute will not automatically change to use the new value.  In order to change the attribute for individual virtual disks, each virtual SCSI device must be deleted and recreated as in the following example:

          $ chdev -dev vios0 -attr allow520blocks=no -perm

          $ rmvdev -vtd vtscsi0

          vtscsi0 deleted

          $ mkvdev -vdev hdisk0 -vadapter vhost0

          vtscsi0 Available

The current setting for individual virtual disks can be determined by using the lsdev command on the virtual SCSI device:

    $ lsdev -dev vtscsi0 -attr

    attribute       value              description                         user_settable

    LogicalUnitAddr 0x8100000000000000 Logical Unit Address                False

    aix_tdev        hdisk0             Target Device Name                  False

    allow520blocks  no                 Allow 520-byte blocks, if supported False

    client_reserve  no                 Client Reserve                      True

    mig_name                           N/A                                 True

    mirrored        false              Metro/Global Mirror                 True

Note that the "user_settable" value for the allow520blocks attribute is "False", indicating that it cannot be changed directly.  It can only be modified by changing the vios0 device setting to the desired value, then deleting and recreating the virtual device.

Microcode updates

Updating the SAS adapter microcode level to a different version has the potential for changing the adapter's ability to support 520-byte sector disk devices. The most likely occurrence of this change would be when upgrading the adapter microcode from an older level which does not support 520-byte sectors to a level that does. While this type of upgrade should not have a detrimental effect on existing configured IBM i disk units, the reverse is not true.  If configured disks are formatted to use 520-byte sectors and the adapter microcode is downgraded to an older level which does not support the 520-byte format, the IBM i client will no longer be able to perform I/O to those disk units until either the microcode is upgraded again, or the disks are reformatted by IBM i to use them as 512-byte sector devices.  Be aware that reformatting destroys all existing data on a disk.

It is also important to note that if the 520-byte sector capability of the SAS adapter microcode is modified during a microcode update, this capability change is not made visible to the client partition by the VIOS until the child hdisk SAS devices (and associated virtual disk devices) are unconfigured and reconfigured after the change. This can be accomplished manually using the following process for each child hdisk:

1. Unconfigure the virtual disk device, e.g. rmdev -dev vtscsi0 -ucfg
2. Unconfigure the hdisk device, e.g. rmdev -dev hdisk0 -ucfg
3. Reconfigure the hdisk device, e.g. cfgdev -dev hdisk0
4. Reconfigure the virtual disk device, e.g. cfgdev -dev vtscsi0

For a large number of disks this may be more easily accomplished by simply rebooting the VIOS partition after completing the microcode update(s), though this action will affect all client partitions until the VIOS is operational again.

IBM i disk unit configuration

Prior to IBM i 7.1 TR7, all virtual SCSI disk units available to an IBM i client partition were represented with a device type of “6B22” and model “050”.  This device type and model will continue to be used to identify virtual disk units formatted with 512-byte sectors; however, a new model “055” is introduced in TR7 to identify virtual disk units that are formatted to use 520-byte sectors.

Only disks units backed by SAS hdisks that are capable of using the new 520-byte sector format can be formatted as 6B22-055 devices.  Additionally, disk units that are capable of using 520-byte sectors, but were previously formatted to use 512-byte sectors, will still be 6B22-050 devices, and therefore not use 520-byte sectors, until they are manually reformatted.

Previously unformatted disk units will appear to be DPHxxx disk resources with a model number that reflects the capability of the device.  For example, a newly created virtual disk that is capable of supporting the 520-byte format will appear as a nonconfigured 6B22-055 DPHxxx resource, while unformatted disk units that are not capable of the 520-byte format will continue to be displayed as 6B22-050 devices.

Reformatting disk units

A 6B22-050 disk unit can be converted to a 6B22-055 device if the VIOS backing hdisk device is capable of being formatted to use 520-byte sectors. This action will typically be necessary after installing the required software upgrades to enable this support, since any disk units previously formatted with 512-byte sectors will still be 6B22-050 devices until they are reformatted.

As previously stated, reformatting is a destructive operation.  The disk unit is initialized to all zeroes during this process, so any existing data on the disk unit will be lost.

A 6B22-055 disk unit may also be reformatted as a 6B22-050 device if the VIOS virtual disk is changed to set the “allow520blocks” attribute to “no”, or if any of the software or hardware components required to support 520-byte sectors are removed before reformatting the device.

In order to change a disk unit from a model 050 to model 055 or vice versa, the disk unit must first be removed from the IBM i disk configuration so that it becomes a nonconfigured unit.  The disk unit can then be manually reformatted using the “Initialize and format” disk recovery option in Dedicated Service Tools (DST):

1. From the DST main menu, select option 4 (Work with disk units)
2. Select option 2 (Work with disk unit recovery)
3. Select option 6 (Disk unit problem recovery procedures)
4. Select option 1 (Initialize and format disk unit)
5. Select the disk unit(s) you wish to reformat
6. Press F10 to confirm.

Once the initialization operation completes, the selected disk units will have the model number and capacity appropriate for their format capabilities.  Disk units whose format capabilities have not changed will remain the same type/model that they were when the initialization operation began.

A disk unit's sector size, capacity, and model number must be up to date before it is added to an Auxiliary Storage Pool (ASP).  Adding a disk unit with incorrect sector size to an ASP will generate an error message, indicating that the selected unit requires manual initialization as described above.

Disk Capacity

Similar to other disk units which support the IBM i 520-byte sector format, the full storage capacity of the backing physical device is available for user data on 6B22-055 disks.  This allows for more effective use of the SAS disk storage resources.

To illustrate, two identical SAS device RAID sets each providing 69.7GB of storage space can be created with the IBM SAS Disk Array Manager in VIOS:

    hdisk21   FC0200FF  Optimal     RAID 0 Array            69.7GB

     pdisk1   000E09FF  Active      Array Member            69.7GB

    hdisk22   FC0300FF  Optimal     RAID 0 Array            69.7GB

     pdisk2   000E0DFF  Active      Array Member            69.7GB

When these devices are virtualized to an IBM i 7.1 TR7 partition with one of the hdisks allowing 520-byte sectors and the other restricted to 512-byte sectors, the resulting IBM i disk units have the following characteristics:

                                                   ----Protected--- ---Unprotected--

     ASP  Unit   Type Model  Threshold  Overflow       Size  %Used      Size  %Used

       1                         90%    No                0  0.00%    131831  3.34%

             1   6B22  055                                0  0.00%     69793  6.13%

             2   6B22  050                                0  0.00%     62038  0.19%

The 6B22-050 device displays less available storage capacity because only 8/9ths of the capacity is available for user data. The 6B22-055 device shows that the full capacity of the backing RAID array can be used for data.

Load Source Disk Support

6B22-055 disk units are supported as load source devices by the IBM i operating system; however, system firmware must also be upgraded to a version that supports these devices as the load source.  Otherwise the IBM i partition will not IPL after the load source disk is formatted to use 520-byte sectors.

If system firmware does not support 6B22-055 devices as the load source unit, the load source must remain formatted as a 6B22-050 device, even if it is otherwise capable of supporting 520-byte sectors.  One way that this can be guaranteed is to change the “allow520capable” setting to “no” in the VIOS virtual disk configuration for the load source disk.

All other disk units may be formatted as 6B22-055 devices, even if the load source device must remain a 6B22-050 device.  The system firmware level requirement applies only to the load source disk unit.

Recommended Practices

• Adapter microcode updates:  All hdisks attached through the SAS adapter need to be reconfigured after any microcode update which changes the adapter’s ability to support 520-byte sector disks.  Therefore, it is recommended that the adapter microcode update be followed by restarting the VIOS partition.  When initially setting up the system to support this new function, this can be accomplished by updating the adapter microcode level, if necessary, before updating the VIOS level.  If it is necessary to downgrade the adapter microcode level to a version that does not support 520-byte sector disks, this should be scheduled around other VIOS maintenance activity so that the number of VIOS reboots can be minimized.  Otherwise, the child hdisks will need to be unconfigured and reconfigured manually, and that manual configuration should be preceded by powering off all client partitions using those disks before they are reconfigured.
• Changing the “allow520blocks” attribute:  Changing the “allow520blocks” attribute of an existing virtual disk device requires that the VIOS virtual target device be deleted and recreated.  To ensure that the client system operating system is always synchronized with the current attribute setting, this is best performed while the client operating system is powered off.
• Solid State Disk configuration recommendations:  To maximize performance, only a single solid state disk (SSD) should be assigned to an individual vhost server device.  When assigning multiple SSDs to a client partition, each SSD should be assigned to a unique client/server adapter pair so that each server adapter attaches only a single SSD device.  This configuration helps to avoid VIOS parallelism and overhead issues when performing I/O to storage devices that have very fast response times.