Setting up the netmon.cf file on a RoCE network
(Linux)
On a remote direct memory access (RDMA) over Converged
Ethernet (RoCE) network, one or more pingable IP addresses must be manually set up in the
netmon.cf configuration file. The
netmon.cf file is required by Reliable Scalable Cluster Technology (RSCT) to
monitor the network and ensure that the interfaces are pingable or not.
Starting
from V11.1.4.4, the procedures documented in this page are no longer required as adapter port
liveliness test has been enhanced and automated. Some restrictions apply. Refer to technote#0733765 for
restrictions.
Determine which IP address should be entered into the members' netmon.cf configuration
file.
On the member host, to check the communication adapter
ports and the associated destination IP subnet, run the route command.
/sbin/route | grep -v link-local
For
example, based on the figure at the end of this topic:
Member 0
[root@host3]# route | grep -v link-local
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
192.168.2.0 * 255.255.255.0 U 0 0 0 eth1
9.26.92.0 * 255.255.254.0 U 0 0 0 eth2
default 9.26.92.1 0.0.0.0 UG 0 0 0 eth2
Member 2
[root@host5]# route | grep -v link-local
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
192.168.2.0 * 255.255.255.0 U 0 0 0 eth1
9.26.92.0 * 255.255.254.0 U 0 0 0 eth2
default 9.26.92.1 0.0.0.0 UG 0 0 0 eth2
The last column
(with column name "Iface") lists the adapters on the current host.
Choose the adapter that corresponds to the target communication adapter
port. In this example, "eth0" and "eth1" are
the target RoCE adapters. The corresponding IP addresses in the first
column shows the target IP subnet to be used in the next step. In
this case, the IP subnets are "192.168.1.0" and
"192.168.2.0".
With
the IP subnet, use the IP interfaces created on the switch 1 and switch
2 that the current host connects to with the same IP subnet. (The
IP interface should already be created as part of the RoCE network
configuration steps, for details see Setting up the IP interfaces on the switch on a RoCE network (Linux).)
In this example, assuming the IP interfaces on switch 1 have IP addresses
of 192.168.1.2 and 192.168.2.2, and switch 2 have IP addresses of
192.168.1.5 and 192.168.2.5, these entries are added to the members
configuration file/var/ct/cfg/netmon.cf.
token2 - eth0 and eth1 are the
RoCE adapter interface names on the local host
token3 - 192.168.1.2, 192.168.2.5,
192.168.1.5, and 192.168.2.2 are the external pingable IP addresses
assigned to the interface created on the switches
The following is an example of what the full configuration
file /var/ct/cfg/netmon.cf looks like for members:
Determine which IP address should be entered into the cluster caching facilities (CFs) netmon.cf configuration
file.
To check the communication
adapter port and the associated destination IP subnet, enter:
/sbin/route | grep -v link-local
For
example:
Host1> $ /sbin/route | grep -v link-local
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.4.0 * 255.255.255.0 U 0 0 0 eth3
192.168.3.0 * 255.255.255.0 U 0 0 0 eth1
192.168.2.0 * 255.255.255.0 U 0 0 0 eth2
192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
9.26.92.0 * 255.255.252.0 U 0 0 0 eth2
default rsb-v94-hsrp.to 0.0.0.0 UG 0 0 0 eth2
The
last column (Iface) indicates the adapter interface name. In this
case, eth0, eth1, eth2, and eth3 are the only communication adapter port interface
on this host. Four IP subnets are relevant to this host.
All
four IP addresses created on the switch (which covers all four IP
subnets) must be entered into this host's netmon.cf configuration
file. For example:
