CES IP aliasing to network adapters on protocol nodes

Cluster Export Services (CES) is a functionality in IBM Spectrum Scale that enables NFS, SMB, and Object protocols. Irrespective of which protocols you choose, all are accessible through a floating pool of IP addresses called CES IPs. This pool of CES IPs is considered floating because each IP can move independently among all protocol nodes. In the event of a protocol node failure, accessibility to all protocols is maintained as the CES IPs automatically move from the failed protocol node to a healthy protocol node. Use this information to understand how CES IPs are assigned and how they are aliased to adapters with or without VLAN tagging.

Virtual LANs (VLANs) are often associated with secure networks because they provide a means of separating network devices into independent networks. Although the physical network infrastructure is shared, unicast, multicast, and broadcast traffic from a network device in a VLAN is restricted to other devices within that same VLAN.

How are CES IPs assigned

CES IPs are automatically assigned and aliased to existing network adapters on protocol nodes during startup. The following example shows aliased CES IPs in a flat network environment or a single VLAN environment in which the switch ports are set to Access mode and thus, do not need VLAN tagging.

Example of aliased CES IPs using the ip addr command

eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.11.1.122/24 brd 10.11.1.255 scope global eth1
   valid_lft forever preferred_lft forever
inet 10.11.1.5/24 brd 10.11.1.255 scope global secondary eth1:0
   valid_lft forever preferred_lft forever
inet 10.11.1.7/24 brd 10.11.1.255 scope global secondary eth1:1
   valid_lft forever preferred_lft forever

Example of preexisting routes

Kernel IP routing table
Destination     Gateway        Genmask         Flags Metric Ref  Use  Iface
default          gateway        0.0.0.0         UG   100    0   0     eth1
10.11.1.0        0.0.0.0        255.255.255.0   U    100    0   0     eth1
172.31.128.0    0.0.0.0         255.255.128.0   U    300    0   0     data0

In the preceding example, eth1 preexists with an established route and IP: 10.11.1.122. This is manually assigned and must be accessible prior to any CES configuration. Once CES services are active, CES IPs are then automatically aliased to this base adapter, thus creating eth1:0 and eth1:1. The floating CES IPs assigned to the aliases are 10.11.1.5 and 10.11.1.7. Both CES IPs are allowed to move to other nodes in case of a failure. This automatic movement combined with the ability to manually move CES IPs, might cause a variance in the number of aliases and CES IPs among protocol nodes. The data0 interface illustrates how a network used for GPFS intra-cluster connectivity between nodes can be separate from the adapter used for CES IPs.

Example distribution of CES IPs among two protocol nodes after enablement of protocols

mmces address list

Address       Node                        Group      Attribute
-------------------------------------------------------------------------
10.11.1.5     protocol-node-1             none       none
10.11.1.6     protocol-node-2             none       object_database_node,object_singleton_node
10.11.1.7     protocol-node-1             none       none
10.11.1.8     protocol-node-2             none       none

CES IPs and VLAN tags

A network switch port can be considered a trunk port if it gives access to multiple VLANs. When this occurs, it is necessary for a VLAN tag to be added to each frame. This VLAN tag is an identification allowing switches to contain traffic within specific networks. If multiple networks must access data from IBM Spectrum Scale protocol nodes, then one possible option is to configure trunk ports on the switch directly connected to the IBM Spectrum Scale protocol nodes. Once a trunk port exists, VLAN tags are necessary on the connected network adapters. Note that CES IPs are automatically assigned and aliased to existing network adapters on protocol nodes during startup. Due to this, the existence of VLAN tags requires a preexisting network adapter with an established route and IP so that CES IPs can alias to it.

Example of aliased CES IPs using the ip addr command (with VLAN tag)

eth1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
    valid_lft forever preferred_lft forever
 
eth1.3016: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.30.16.122/24 brd 10.30.16.255 scope global eth1.3016
   valid_lft forever preferred_lft forever
inet 10.30.16.5/24 brd 10.30.16.255 scope global secondary eth1.3016:0
   valid_lft forever preferred_lft forever
inet 10.30.16.7/24 brd 10.30.16.255 scope global secondary eth1.3016:1
   valid_lft forever preferred_lft forever

Example of pre-existing routes (with VLAN tag)

Kernel IP routing table
Destination      Gateway     Genmask         Flags Metric Ref Use   Iface
default          gateway     0.0.0.0         UG    100    0   0    eth1.3016
10.30.16.0       0.0.0.0     255.255.255.0   U     100    0   0    eth1.3016
172.31.128.0     0.0.0.0     255.255.128.0   U     300    0   0    data0

As in the no VLAN tag example, an existing network adapter must be present so that CES\ IPs can alias to it. Note the non-VLAN base adapter eth1, has no IPs assigned. In this example, the preexisting network adapter with an established route and IP is eth1.3016. The IP for eth1.3016 is 10.30.16.122 and the VLAN tag is 3016. This preexisting IP can be used for network verification prior to CES IP configuration by pinging it from external to the cluster or pinging it from other protocol nodes. It is a good practice to make sure that all protocol node base adapter IPs are accessible before enabling protocols. The data0 interface shows how a network used for GPFS intra-cluster connectivity between nodes can be separate from the adapter used for CES IPs.

Example distribution of CES IPs among two protocol nodes after enablement of protocols (with VLAN tag)

mmces address list

Address        Node                         Group      Attribute
-------------------------------------------------------------------------
10.30.16.5     protocol-node-1              none       none
10.30.16.6     protocol-node-2              none       object_database_node,object_singleton_node
10.30.16.7     protocol-node-1              none       none
10.30.16.8     protocol-node-2              none       none

CES IPs and multiple VLAN tags

The following diagram shows a node with two network adapters devoted to CES protocols: eth1 and eth2. The eth1 interface has two VLANs associated: 3016 and 3017. The eth2 interface has one VLAN associated: 80.

Example of aliased CES IPs using the ip addr command (with multiple VLAN tag)

eth1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
    valid_lft forever preferred_lft forever
 
eth1.3016: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.30.16.122/24 brd 10.30.16.255 scope global eth1.3016
   valid_lft forever preferred_lft forever
inet 10.30.16.5/24 brd 10.30.16.255 scope global secondary eth1.3016:0
   valid_lft forever preferred_lft forever
inet 10.30.16.7/24 brd 10.30.16.255 scope global secondary eth1.3016:1
   valid_lft forever preferred_lft forever
 
eth1.3017: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.30.17.50/24 brd 10.30.17.255 scope global eth1.3017
   valid_lft forever preferred_lft forever
inet 10.30.17.100/24 brd 10.30.16.255 scope global secondary eth1.3017:0
   valid_lft forever preferred_lft forever
inet 10.30.17.103/24 brd 10.30.16.255 scope global secondary eth1.3017:1
   valid_lft forever preferred_lft forever
 
eth2: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
    valid_lft forever preferred_lft forever
 
eth2.80: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.11.80.50/24 brd 10.11.80.255 scope global eth1.80
   valid_lft forever preferred_lft forever
inet 10.11.80.55/24 brd 10.11.80.255 scope global secondary eth1.80:0
   valid_lft forever preferred_lft forever

Example of preexisting routes (with multiple VLAN tag)

Kernel IP routing table
Destination    Gateway     Genmask         Flags Metric Ref  Use  Iface
default        gateway     0.0.0.0         UG    100    0     0   eth1.3016
10.30.16.0     0.0.0.0     255.255.255.0   U     100    0     0   eth1.3016
10.30.17.0     0.0.0.0     255.255.255.0   U     400    0     0   eth1.3017
10.11.80.0     0.0.0.0     255.255.255.0   U     200    0     0   eth2.80
172.31.128.0   0.0.0.0     255.255.128.0   U     300    0     0   data0

Example distribution of CES IPs from multiple VLANs among two protocol nodes after enablement of protocols

mmces address list
Address         Node                           Group      Attribute
-------------------------------------------------------------------------
10.11.80.54     protocol-node-2                none       none
10.11.80.55     protocol-node-1                none       none
10.30.16.5      protocol-node-1                none       none
10.30.16.6      protocol-node-2                none       none
10.30.16.7      protocol-node-1                none       none
10.30.16.8      protocol-node-2                none       none
10.30.17.100    protocol-node-1                none       none
10.30.17.101    protocol-node-2                none       none
10.30.17.102    protocol-node-2                none       object_database_node,object_singleton_node
10.30.17.103    protocol-node-1                none       none