Managing VFIO AP mediated devices with libvirt

Use libvirt commands to manage the lifecycle of mediated devices for VFIO pass-through of cryptographic resources.

Before you begin

Cryptographic adapter resources are managed as AP queues. To make an AP queue eligible for VFIO pass-through, you must bring it under control of the vfio_ap device driver, see Free AP queues for use by KVM guests.

Procedure

List the available cryptographic resources by issuing the following command:

# lszcrypt -V
card.domain type  mode           status ... functions driver
-------------------------------------------------------------
08          cex5c cca-coproc     online ... s--d--nf- cex4card
08.0001     cex5c cca-coproc     -      ... s--d--nf- vfio_ap
08.0002     cex5c cca-coproc     -      ... s--d--nf- vfio_ap
09          cex5c cca-coproc     online ... s--d--nf- cex4card
09.0001     cex5c cca-coproc     -      ... s--d--nf- vfio_ap
09.0002     cex5c cca-coproc     -      ... s--d--nf- vfio_ap

The AP queues with driver vfio_ap are eligible for a mediated device. In the sample output, these AP queues are 08.0001, 08.0002, 09.0001, and 09.0002, as shown in the card.domain column. These AP queues correspond to a matrix of adapters 08 and 09 with domains 0001 and 0002.

If the AP queues of interest are not controlled by vfio_ap, you have to free them from control of the crypto device driver, see Free AP queues for use by KVM guests.

You can also use the virsh nodedev-list command to list the adapters and queues.

# virsh nodedev-list --cap ap_card
ap_card08
ap_card09
# virsh nodedev-list --cap ap_queue
ap_08_0001
ap_08_0002
ap_09_0001
ap_09_0002

Use the virsh nodedev-dumpxml command to confirm that the AP queues are controlled by the vfio_ap device driver.

# virsh nodedev-dumpxml ap_08_0001
<device>
    <name>ap_08_0001</name>
    <path>/sys/devices/ap/card08/08.0001</path>
    <parent>ap_card08</parent>
    <driver>
        <name>vfio_ap</name>
    </driver>
    <capability type="ap_queue">
        <ap-adapter>0x08</ap-adapter>
        <ap-domain>0x0001</ap-domain>
    </capability>
</device>

Use the following template to create a file with a node-device XML description of the mediated device.
```
<device>
    <parent>ap_matrix</parent>
    <capability type="mdev">
        <type id="vfio_ap-passthrough"/>
    </capability>
</device>
```
As child elements of the capabilities element, add attr elements for adapters and domains to specify a subset of the matrix of available AP queues. The values are adapter IDs and domain IDs in hexadecimal notation with 0x prefixes.
For example, for the matrix of domain 0002 on both of the available adapters 08 and 09, add three attr elements, one for each adapter and one for the domain.
```
<device>
    <parent>ap_matrix</parent>
    <capability type="mdev">
        <type id="vfio_ap-passthrough"/>
        <attr name="assign_adapter" value="0x08"/>
        <attr name="assign_adapter" value="0x09"/>
        <attr name="assign_domain" value="0x0002"/>
    </capability>
</device>
```
As a child element of the capabilities element, add a uuid element that specifies the UUID that you want to use for the mediated device.
You can use the uuidgen command to obtain a UUID. For transient mediated devices, you can omit this specification. A UUID is then generated for you with the command that creates the transient mediated device.
For example, the following specification configures a stable UUID bfccf00d-21f1-448c-9979-b7341129d985 for the mediated device.
```
<device>
    <parent>ap_matrix</parent>
    <capability type="mdev">
        <type id="vfio_ap-passthrough"/>
        <attr name="assign_adapter" value="0x08"/>
        <attr name="assign_adapter" value="0x09"/>
        <attr name="assign_domain" value="0x0002"/>
        <uuid>bfccf00d-21f1-448c-9979-b7341129d985</uuid>
    </capability>
</device>
```
Define the persistent mediated device by issuing a virsh nodedev-define command.
The following example assumes that the node-device XML file of the previous step is stored at ~/apmatrices/08-09_0002.xml.
```
# virsh nodedev-define ~/apmatrices/08-09_0002.xml
Node device mdev_bfccf00d_21f1_448c_9979_b7341129d985_matrix defined from ~/apmatrices/08-09_0002.xml
```
For transient devices, use the virsh nodedev-create command instead. This command creates and activates a mediated device and the activation step that follows for persistent devices does not apply.
The mediated device of the example has a UUID bfccf00d-21f1-448c-9979-b7341129d985. Within libvirt, the device is represented as mdev_bfccf00d_21f1_448c_9979_b7341129d985_matrix.
You can list the device with the virsh nodedev-list command.
```
# virsh nodedev-list --cap mdev
mdev_bfccf00d_21f1_448c_9979_b7341129d985_matrix
```
The tree view of the virsh nodedev-list command shows that the mediated device corresponds to a matrix of AP queues. Because the device is not yet activated, you need the --all option to include inactive devices in the command output.
```
$ virsh nodedev-list --tree --all
computer
  |
  +- ap_card08
  |
  | +- ap_08_0001
  | +- ap_08_0002
  |
  +- ap_card09
  |   |
  |   +- ap_09_0001
  |   +- ap_09_0002
  |
  +- ap_matrix
  |   |
  |   +- mdev_bfccf00d_21f1_448c_9979_b7341129d985_matrix
  ...
```
Use the virsh nodedev-dumpxml command to display the properties of the mediated device in node-device XML format.
```
# virsh nodedev-dumpxml mdev_bfccf00d_21f1_448c_9979_b7341129d985_matrix
<device>
    <name>mdev_bfccf00d_21f1_448c_9979_b7341129d985_matrix</name>
    <path>/sys/devices/vfio_ap/matrix/bfccf00d-21f1-448c-9979-b7341129d985</path>
    <parent>ap_matrix</parent>
    <driver>
        <name>vfio_mdev</name>
    </driver>
    <capability type="mdev">
        <type id="vfio_ap-passthrough"/>
        <iommuGroup number="4"/>
    </capability>
</device>
```
The path element contains the sysfs path of the mediated device. Read the matrix attribute to display the matrix of AP queues.
```
# cat /sys/devices/vfio_ap/matrix/bfccf00d-21f1-448c-9979-b7341129d985/matrix
08.0002
09.0002
```

For persistent mediated devices: Activate the mediated device by issuing a virsh nodedev-start command.

# virsh nodedev-start mdev_bfccf00d_21f1_448c_9979_b7341129d985_matrix
Device mdev_bfccf00d_21f1_448c_9979_b7341129d985_matrix started

What to do next

You can now use the mediated device to configure the AP queues for a KVM guest. See Configuring cryptographic adapter resources.

When you no longer need a mediated device, you can stop it with the virsh nodedev-destroy command.

# virsh nodedev-destroy mdev_bfccf00d_21f1_448c_9979_b7341129d985_matrix
Destroyed node device 'mdev_bfccf00d_21f1_448c_9979_b7341129d985_matrix'

A transient device ceases to exist as a result of the virsh nodedev-destroy command. A persistent device is deactivated, and you must use the virsh nodedev-undefine command to remove the inactive device from libvirt.

# virsh nodedev-undefine mdev_bfccf00d_21f1_448c_9979_b7341129d985_matrix
Undefined node device 'mdev_bfccf00d_21f1_448c_9979_b7341129d985_matrix'