Assigning cryptographic adapter resources to vfio_ap

Red Hat Enterprise Linux 8.6 LPAR mode KVM guest

Create a VFIO AP mediated device and assign cryptographic adapter resources to the device.

Cryptographic adapter resources are managed as AP queues (see Cryptographic domains). An AP queue corresponds to a specific cryptographic domain on a specific cryptographic adapter. AP queues are assigned indirectly through assignments of domains and adapters. As a consequence, AP queues are not assigned independently, but in the form of a matrix of queues. A queue is in the matrix if both its adapter and its domain are specified.

Load the modules

Ensure that the vfio_ap and vfio_mdev device drivers are loaded.

# modprobe vfio_ap
# modprobe vfio_mdev

Create a mediated device

Before you begin: Mediated devices that you create with the commands that follow do not persist across reboots. For information about creating a persistent or transient mediated device with libvirt, see KVM Virtual Server Management, SC34-2752.

Associate a new VFIO mediated device with the vfio_ap device driver by writing a universally unique identifier (UUID) to /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create.

Tip: Use the uuidgen command to generate a UUID.

Example: This example creates a VFIO AP mediated device.

# uuidgen
4b0518fd-9237-493f-93c8-c5597f8006a3
# echo 4b0518fd-9237-493f-93c8-c5597f8006a3 \
> /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create

Configure the mediated device

The sysfs representations of VFIO AP mediated devices are subdirectories of /sys/devices/vfio_ap/matrix. The directory names match the UUIDs that identify the mediated devices. Use the attributes in a device directory to configure the device and to obtain information about the device.

Table 1. sysfs attributes of VFIO mediated devices for cryptographic adapter resources
Attribute	Explanation
assign_adapter	Write adapter IDs to this attribute to assign adapters to the mediated device. Specify the adapter IDs in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x. Example: # echo 0x0a > assign_adapter
assign_control_domain	Write domain IDs to this attribute to assign control domains to the mediated device. Assign a control domain for each domain that you assign to the mediated device, so that you can manage your domains from the guest that uses the mediated device. Specify the domain IDs in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x. Example: # echo 0x001b > assign_control_domain For information about control domains, see Cryptographic devices on LPARs.
assign_domain	Write domain IDs to this attribute to assign usage domains to the mediated device. Specify the domain IDs in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x. Example: # echo 0x001b > assign_domain For information about usage domains, see Cryptographic devices on LPARs.
control_domains	Read this attribute to list the assigned control domains. Example: # cat control_domains 001b
matrix	Read this attribute to list the assigned AP queues that result from the adapter and domain assignments. Example: # cat matrix 0a.001b
mdev_type	Symbolic link that points to the vfio_ap-passthrough directory.
remove	Write `1` to this attribute to remove the mediated device. Example: # echo 1 > remove
subsystem	Symbolic link that points to the matrix bus.
unassign_adapter	Write adapter IDs to this attribute to remove adapters from the mediated device. Specify the adapter IDs in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x. Example: # echo 0x0a > unassign_adapter
unassign_control_domain	Write domain IDs to this attribute to remove control domains from the mediated device. Specify the domain IDs in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x. Example: # echo 0x001b > unassign_control_domain
unassign_domain	Write domain IDs to this attribute to remove usage domains from the mediated device. Specify the domain IDs in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x. Example: # echo 0x001b > unassign_domain

Scenario

This scenario assumes a KVM host on which 8 AP queues have been freed from control of the zcrypt device driver. A matrix of two of these queues are to be assigned to a mediated device.

Load the required modules.
```
# modprobe vfio_ap
# modprobe vfio_mdev
```

List the eligible AP queues. AP queues are eligible only if they are controlled by the vfio_ap device driver.

# lszcrypt -V | grep vfio
00.0001     CEX6A   Accelerator  online         0        0     12     08 -MC-A-NF-  vfio_ap
00.0002     CEX6A   Accelerator  online         0        0     12     08 -MC-A-NF-  vfio_ap
00.0004     CEX6A   Accelerator  online         0        0     12     08 -MC-A-NF-  vfio_ap
00.001b     CEX6A   Accelerator  online         0        0     12     08 -MC-A-NF-  vfio_ap
0a.0001     CEX6P   EP11-Coproc  online         0        0     12     08 -----XNF-  vfio_ap
0a.0002     CEX6P   EP11-Coproc  online         0        0     12     08 -----XNF-  vfio_ap
0a.0004     CEX6P   EP11-Coproc  online         0        0     12     08 -----XNF-  vfio_ap
0a.001b     CEX6P   EP11-Coproc  online         0        0     12     08 -----XNF-  vfio_ap

The output shows that 8 AP queues are eligible. The eight queues correspond to a matrix of two adapters, 0x00 and 0x0a and four domains, 0x0001, 0x0002, 0x0004, and 0x001b. These adapters and domains are the only ones that you can assign to a mediated device.

Create a mediated device.

# uuidgen
4b0518fd-9237-493f-93c8-c5597f8006a3
# echo 4b0518fd-9237-493f-93c8-c5597f8006a3 \
> /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create

Add adapters 0x00 and 0x0a to the device.

# echo 0x00 > /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/assign_adapter
# echo 0x0a > /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/assign_adapter
# cat /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/matrix
00.
0a.

Reading the matrix attribute does not yield any AP queues. To assign AP queues both adapters and domains must be added.

Add domain 0x001b.
```
# echo 0x001b > /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/assign_domain
# cat /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/matrix
00.001b
0a.001b
# cat /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/control_domains
```
The content of the matrix attribute shows that two AP queues are assigned to the mediated device. The two queues correspond to a matrix of two adapters, 0x00 and 0x0a and one domain, 0x001b. Reading the control_domains attribute shows that no control domain is configured.

Add domain 0x001b to the control domains.

# echo 0x001b > /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/assign_control_domain
# cat /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/control_domains
001b