Setting up cryptographic adapter resources for VFIO pass-through

6.6 LPAR mode KVM guest

Free cryptographic adapter resources from host control and assign them to a VFIO AP mediated device.

AP queues

Cryptographic adapter resources are managed as AP queues (see Cryptographic domains). An AP queue corresponds to a specific cryptographic domain on a specific cryptographic adapter.

AP queues are assigned indirectly through assignments of domains and adapters. You can represent AP queues as the cells of a table with a column for each adapter and a row for each domain. In this representation, assigned AP queues are the cells where assigned adapters intersect with assigned domains.

Table with adapters and domains where the assigned AP queues correspond to the intersection of assigned adapters and assigned domains — Figure 1. Table of assigned AP queues

In Figure 1, the grey table cells indicate the implicitly assigned AP queues, which are 00.0000, 00.0002, 00.0003, 02.0000, 02.0002, and 02.0003.

Multi-dimensional arrays, like the two-dimensional table of Figure 1, are often called matrices. Therefore, the term matrix is widely used for representations of tables of AP queues, for example, in sysfs attributes.

You work with matrices of AP queues at different levels.

LPAR level

AP queues that are available to the KVM host.

For hosts that run in LPAR mode, these are the AP queues that result from the AP configuration for the LPAR. On a running host, this matrix can change through dynamic changes of the adapter and domain configurations for the LPAR.

Host level

AP queues that are controlled by the host's zcrypt device drivers and so can be accessed by applications on the host.

By default, all queues that are available to the host are controlled by the host's zcrypt device drivers. You can change this default with the ap.apmask= and ap.aqmask= kernel parameters, see Kernel parameters. You can dynamically change this matrix by adding or removing adapters and domains, for example, with the chzdev command.

If the vfio_ap device driver is loaded, it takes control of the AP queues that are not controlled by the host's zcrypt device drivers. These AP queues can then be passed on to KVM guests through AP mediated devices.

Mediated device level

AP queues that are assigned to a VFIO AP mediated device.

You make AP queues available to a KVM guest by attaching a VFIO AP mediated device to the KVM guest.

A mediated device cannot include queues that, at the host level, are assigned to the host's zcrypt device drivers. Still, it can include AP queues that are unavailable because they are associated with adapters that are not physically plugged into the hardware. Other queues might not be available because the associated adapter, domain, or both are not configured for the LPAR. As a consequence, attaching a mediated device to a KVM guest might not make all queues of the mediated device matrix available to the KVM guest.

Read the /sys/devices/vfio_ap/matrix/<uuid>/matrix attribute of a mediated device to list all AP queues that are assigned to it. Read the /sys/devices/vfio_ap/matrix/<uuid>/guest_matrix to list only queues that can be used by KVM guests. In the paths, <uuid> is the UUID that identifies the mediated device.

You can dynamically change the adapter and domain assignment through a mediated device's sysfs attributes, see Table 1.

Figure 2 is a schematic illustration of the three matrix layers as a succession of filters, where grey cells indicate assigned AP queues. An AP queue can be passed on to a KVM guest only if it is suitably assigned at each level.

Visualization of the three levels as described in the preceding text — Figure 2. Matrices as a succession of filters

Hotplug and hot unplug

On a running KVM guest, attaching a mediated device results in hotplug events for all those AP queues of the mediated device that are controlled by the vfio_ap device driver on the host, which implies that both their adapter and domain is configured for the LPAR.

With a mediated device attached to a KVM guest, hotplug or hot unplug events for queues can be triggered at the LPAR or mediated device configuration level:

Dynamic changes to the AP configuration for the LPAR on the SE or HMC
Dynamic changes of the mediated device

Detaching a mediated device results in hot unplug events for all AP queues of the mediated device that are used by the KVM guest.

Persistence across host reboots

You can set up your cryptographic resources such that, after a host reboot, your mediated devices are ready to be attached to KVM guests, without further configuration steps. This setup requires a persistent assignment of AP queues to the vfio_ap device driver and persistent mediated devices with their assignment of AP queues.

Use the chzdev command to persistently assign AP queues to the vfio_ap device driver. Use the nodedev-define command to make a mediated device and its assignment of AP queues persistent.

Procedure

Ensure that the vfio_ap device driver is loaded. If it is compiled as a separate module, you might have to load it yourself. The following command loads vfio_ap and any additional modules it requires.
```
# modprobe vfio_ap
```
Assign AP queues to the vfio_ap device driver.
Two 256-bit masks, one for adapters and the other for domains, rule which AP queues are controlled by the zcrypt device driver. Unless you change the initial setting with the ap.apmask= and ap.aqmask= kernel parameters, both masks have all bits on by default. With all bits set in the masks, all AP queues that are available to the host can be accessed only by the host itself.

Setting the bit for a particular adapter to 0 frees all queues of that adapter from the host and makes them available for use by vfio_ap. Similarly, setting the bit for a particular domain to 0 frees all queues of that domain, across all adapters, from the host and makes them available for use by vfio_ap.

Set bits for adapters or domains or both to 0 to assign queues to the vfio_ap device driver.

Using the chzdev command

The preferred method for changing the masks is the chzdev command with the --type ap option. With the chzdev command, you can change the masks persistently, for the active configuration, or both. For details, see KVM Virtual Server Management, SC34-2752.

Writing to the masks in sysfs

As a non-persistent fallback method, you can directly write to the masks in sysfs, see Freeing AP queues for KVM guests.

Create and configure a mediated device.

Using libvirt

The preferred method for creating and configuring a mediated device is using the virsh nodedev-define command and a node-device XML description. With this command, you can create a persistent mediated device, see KVM Virtual Server Management, SC34-2752.

Using general Linux commands and sysfs

As a non-persistent fallback method, you can create VFIO mediated device for the vfio_ap device driver by writing a universally unique identifier (UUID) to /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create.

You then configure the mediated device by writing to the attributes within the sysfs representations of VFIO AP mediated device. VFIO AP mediated devices are represented as subdirectories of /sys/devices/vfio_ap/matrix. The directory names match the UUIDs that identify the mediated devices.

Use the attributes in a device directory to configure the device and to obtain information about the device.

Table 1. sysfs attributes of VFIO mediated devices for cryptographic adapter resources
Attribute	Explanation
assign_adapter	Write an adapter ID to this attribute to assign the adapter to the mediated device. Specify the adapter ID in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x. Example: `# echo 0x0a > assign_adapter`
assign_control_domain	Write a domain ID to this attribute to assign the domain as a control domain to the mediated device. Assign a control domain for each usage domain that you assign to the mediated device, so that you can manage your domains from the guest that uses the mediated device. Specify the domain ID in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x. Example: `# echo 0x001b > assign_control_domain`
assign_domain	Write a domain ID to this attribute to assign a usage domain to the mediated device. Specify the domain ID in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x. Example: `# echo 0x001b > assign_domain`
control_domains	Read this attribute to list the assigned control domains. Example: `# cat control_domains 001b`
guest_matrix	Read this attribute to list the subset of assigned AP queues that are eligible for KVM guests.
matrix	Read this attribute to list the assigned AP queues that result from the adapter and domain assignments. Example: `# cat matrix 0a.001b`
mdev_type	Symbolic link that points to the vfio_ap-passthrough directory.
remove	Write `1` to this attribute to remove the mediated device. Example: `# echo 1 > remove`
subsystem	Symbolic link that points to the matrix bus.
unassign_adapter	Write an adapter ID to this attribute to remove the adapter from the mediated device. Specify the adapter ID in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x. Example: `# echo 0x0a > unassign_adapter`
unassign_control_domain	Write a domain ID to this attribute to remove the domain from the control domains of the mediated device. Specify the domain ID in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x. Example: `# echo 0x001b > unassign_control_domain`
unassign_domain	Write a domain ID to this attribute to remove a the domain from the usage domains of the mediated device. Specify the domain ID in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x. Example: `# echo 0x001b > unassign_domain`

Scenario: Fallback method for creating and configuring a mediated device

This scenario assumes a KVM host on which 8 AP queues have been freed from control of the zcrypt device driver. A matrix of two of these queues are to be assigned to a mediated device.

Note: The steps that follow describe a fallback method that uses common Linux commands to directly write to sysfs. This method does not create a persistent mediated device. The preferred method for creating and configuring mediated devices is an XML description of the mediated device and the virsh nodedev-define command, see KVM Virtual Server Management, SC34-2752.

Load the required modules.
```
# modprobe vfio_ap
```

List the eligible AP queues. AP queues are eligible only if they are controlled by the vfio_ap device driver.

# lszcrypt -V | grep vfio
00.0001     CEX7A   Accelerator  online         0        0     13     08 -MC-A-NF-  vfio_ap
00.0002     CEX7A   Accelerator  online         0        0     13     08 -MC-A-NF-  vfio_ap
00.0004     CEX7A   Accelerator  online         0        0     13     08 -MC-A-NF-  vfio_ap
00.001b     CEX7A   Accelerator  online         0        0     13     08 -MC-A-NF-  vfio_ap
0a.0001     CEX7P   EP11-Coproc  online         0        0     13     08 -----XNF-  vfio_ap
0a.0002     CEX7P   EP11-Coproc  online         0        0     13     08 -----XNF-  vfio_ap
0a.0004     CEX7P   EP11-Coproc  online         0        0     13     08 -----XNF-  vfio_ap
0a.001b     CEX7P   EP11-Coproc  online         0        0     13     08 -----XNF-  vfio_ap

The output shows that 8 AP queues are eligible. The eight queues correspond to a matrix of two adapters, 0x00 and 0x0a and four domains, 0x0001, 0x0002, 0x0004, and 0x001b. These adapters and domains are the only ones that you can assign to the mediated device.

Create a mediated device.

# uuidgen
4b0518fd-9237-493f-93c8-c5597f8006a3
# echo 4b0518fd-9237-493f-93c8-c5597f8006a3 \
> /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create

Add adapters 0x00 and 0x0a to the device.

# echo 0x00 > /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/assign_adapter
# echo 0x0a > /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/assign_adapter
# cat /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/matrix
00.
0a.

Reading the matrix attribute does not yield any AP queues. To assign AP queues both adapters and domains must be added.

Add domain 0x001b.
```
# echo 0x001b > /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/assign_domain
# cat /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/matrix
00.001b
0a.001b
# cat /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/control_domains
```
The content of the matrix attribute shows that two AP queues are assigned to the mediated device. The two queues correspond to a matrix of two adapters, 0x00 and 0x0a and one domain, 0x001b. Reading the control_domains attribute shows that no control domain is configured.

Add domain 0x001b to the control domains.

# echo 0x001b > /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/assign_control_domain
# cat /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/control_domains
001b