Setting up the SMC support

How to associate the TCP/IP network interfaces and SMC-capable devices that can reach a communication peer is different for SMC-Dv2 and SMC-Rv2.

SMC-Dv2

Other than for SMC-Dv1, SMC-Dv2 does not require PNET IDs to explicitly associate the interfaces, but PNET IDs must also not contradict the association. If set for both interfaces, the PNET ID must be the same, thus enabling the fallback to SMC-Dv1. This fallback would otherwise not be available, and is required when connecting to peers that support SMC-Dv1 only.

SMC-Rv2

Like SMC-Rv1, SMC-Rv2 requires PNET IDs to explicitly associate the interfaces.

An EID can be pre-defined in the hardware system or it can be user-defined.

System-defined EID

The unique system-defined EIDs of IBM Z® and IBM LinuxONE hardware systems are relevant to SMC-Dv2. Operating system instances with the same system-defined EID run on the same hardware system and are eligible to exchange SMC-Dv2 traffic.

By default, Linux instances use the system-defined EID. With the smcd seid command, you can disable or enable the system-defined EID (see smcd - Display information about SMC-D link groups and devices).

In contrast, z/OS disables the system-defined EID by default. The system-defined EID is enabled or disabled through a configuration parameter, see z/OS Communications Server: IP Configuration Guide.

With user-defined EIDs you can restrict SMC traffic to groups of operating system instances.

User-defined EIDs

User-defined EIDs are relevant to both SMC-Dv2 and SMC-Rv2, and the same user-defined EIDs apply to both SMC variants.

Assign user-defined EIDs to set up groups of operating system instances that are eligible for SMC traffic within the groups. For SMC-Rv2, user-defined EIDs can span multiple hardware systems.

If EIDs are used to group operating system instances that are geographically close, guests of the same z/VM system can all share an EID. Similarly, for SMC-Rv2 traffic, KVM guests on the same KVM host often have the same EID.

A Linux instance can have up to four EIDs, and so be a member of up to four groups. It is then eligible for SMC traffic with operating system instances in each group.

You can use the smcd ueid command or the smcr ueid command to manage user-defined EIDs (see smcr - Display information about SMC-R and smcd - Display information about SMC-D link groups and devices).

Instances of Linux on IBM Z or IBM LinuxONE have at least one active EID.

• You cannot disable the system-defined EID unless at least one user-defined EID is assigned.
• Deleting the last user-defined EID automatically enables the system-defined EID.

Figure 2 shows an example with three Linux instances on an IBM Z system. For all instances, the system-defined EID is enabled. With IP connectivity and eligible ISM devices in place, all instances can exchange SMC-Dv2 traffic, across IP subnets.

In Figure 3, two of the Linux instances disabled their system-defined EID and use a matching user-defined EID instead. With this setup, only the instances with matching user-defined EIDs can exchange SMC-Dv2 traffic, Linux 1 and Linux 3 in the example.

SMC-R connections can span both IP subnets and hardware systems, as illustrated in Figure 4.

For example, if eno3 is the network device that is associated with the wanted PCI function:

# ethtool -a eno3
Pause parameters for eno3:
Autonegotiate: off
RX: on
TX: on

RoCE Express PCI functions provide both, interfaces for SMC-R RDMA traffic and Ethernet interfaces for TCP traffic. To use a PCI function as a failover device for RDMA, the Ethernet interface must be active but not permit any traffic. The following example shows how this condition can be attained. The example uses the ip command. For a persistent configuration, use the network manager of your distribution.

1. Set up a link mylnk_eth0 for an interface eth0

   # ip link add dev mylnk_eth0 link eth0

   To set up the link in the context of a VLAN, append the VLAN specifications to this command. For example, for a VLAN with ID 661, the command becomes:

   # ip link add dev mylnk_eth0 link eth0 type vlan id 661

2. Assign an IP address to the link.

   # ip addr add 10.2.1.1/16 dev mylnk_eth0

3. Activate the link.

   # ip link set mylnk_eth0 up

4. Remove all auto-generated routes for the new link.

   # ip route flush scope link dev mylnk_eth0

5. The network manager of your distribution might interpret this stale link setup as a configuration error. Prevent the network manager from reversing your settings to make the link functional. The example shows a NetworkManager command.

   # nmcli device set eth0 managed no

   Your distribution might use a different network manager, for example, wicked or netplan. Use a command according to your network manager.

SMC requires contiguous memory. The minimum is 16 KB, and the maximum is 512 MB. The SMC implementation selects a value as follows:

• Some socket applications define the socket send- and receive buffer sizes with a setsockopt call, whose upper limits are defined in net.core.wmem_max and net.core.rmem_max.
• If setsockopt SO_SNDBUF is not used, the socket send buffer size is taken from the value of net.ipv4.tcp_wmem.
• If setsockopt SO_RCVBUF is not used, the socket receive buffer is taken from the value of net.ipv4.tcp_rmem, rounded to the next higher power of 2.

As an alternative to smc_run, you can use the LC_PRELOAD environment variable to specify the preload library with the application's start command:

# LD_PRELOAD=libsmc-preload.so <application_start_cmd>

Alternatively, if you need to, you can convert an application. To convert an application from TCP/IP to SMC sockets, change the socket() function call from AF_INET to AF_SMC with protocol 0 and from AF_INET6 to AF_SMC with protocol 1. For example, change:

sd = socket(AF_INET, SOCK_STREAM, 0);

to:

sd = socket(AF_SMC, SOCK_STREAM, 0);

and

sd = socket(AF_INET6,SOCK_STREAM, 0);

to:

sd = socket(AF_SMC, SOCK_STREAM, 1);