RDMA over Converged Ethernet
The two nodes in a mirrored pair require a high-speed network connection to perform synchronous data replication.
The high-speed network protocol being used is Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE). Adapters that support RDMA must be installed in both nodes. For details about the hardware requirements, see Network adapters.
IP addresses are configured on the line description associated with each port of the RoCE adapters. A pair of IP addresses, one from each node, is used to identify each physical RDMA link between the two nodes.
The following RDMA protocols are supported:- RoCE v1: RoCE version 1 is a non-encrypted and non-routable RDMA protocol. The ports of RoCE v1 adapters can be cabled directly together or connected by two cables and a single switch. The maximum cable length varies by adapter type. Most adapters support up to 200 meters, but some can support up to 10 kilometers. This type of link is available only if non-encrypted RDMA is allowed in your environment.
- RoCE v2: RoCE version 2 is a non-encrypted and routable RDMA protocol. The ports of RoCE v2 adapters can be connected through multiple switches or routers, up to a maximum distance of 10 kilometers. This type of link is available only if non-encrypted RDMA is allowed in your environment.
- Encrypted RoCE v2: Encrypted RoCE version 2 is an encrypted and routable RDMA protocol. It uses IPsec protocols to provide data authentication, integrity, and confidentiality. The ports of RoCE v2 adapters can be connected through multiple switches or routers, up to a maximum distance of 10 kilometers. Adapters that support encryption are available on POWER9™ or newer hardware.
A link can be configured as one type of RDMA protocol only. Both ends of the link
must be configured to use the same RDMA protocol or the link will not be available. When a link is
shared between multiple NRGs, changing the RDMA protocol of a link in one NRG will also change the
RDMA protocol for that link in all NRGs. If the RDMA protocol of a link is changed while the link is
active, the new protocol will not be used until after the link is restarted.