CES network configuration

Cluster Export Services (CES) IP addresses are used to export data via the NFS, SMB, and Object protocols. File and Object clients use the public IPs to access data on GPFS™ file systems.

CES IP addresses have the following characteristics:
  • Shared between all CES protocols
  • Organized in an address pool (there can be fewer or more CES addresses than nodes)
  • Hosted on the CES nodes (there can be CES nodes without CES addresses)
  • Can move between CES nodes (triggered manually via the command or as part of a CES failover)
  • Must not be used for GPFS communication at the same time
CES IP addresses have these restrictions:
  • The network on CES nodes must be configured so that all CES IPs can run on any CES node. Typically this configuration requires that all CES nodes have at least one NIC interface or VLAN-compatible interface with each CES IP network address.
  • CES IPs are created as aliases on each CES node. Do not include the primary address of an adapter in the CES IP address pool.
  • CES IPs must be resolvable by DNS or /etc/hosts.
  • CES does not manage the subnet or netmask configuration; it is the user's task.
To add CES IP addresses to the address pool, use the mmces command:
mmces address add --ces-ip Address[,Address...]
By default, addresses are distributed among the CES nodes, but a new address can be assigned to a particular node:
mmces address add --ces-ip Address[,Address...] --ces-node Node
After a CES IP address is added to the address pool, you can manually move the address to a particular node:
mmces address move --ces-ip Address[,Address...] --ces-node Node
You can remove a CES IP address from the address pool with the mmces command:
mmces address remove --ces-ip Address[,Address...]

Removing an address while there are clients connected causes the clients to lose those connections. Any reconnection to the removed IP results in a failure. If DNS is used to map a name entry to one or more IP addresses, update the DNS to ensure that a client is not presented an address that was already removed from the pool. This process might also include invalidation of any DNS caches.

CES addresses are virtual IP addresses

The CES addresses that are assigned to the CES nodes are implemented as IP aliases. Each network adapter that hosts CES addresses must already be configured (with different non-CES IPs) in /etc/sysconfig. CES uses the netmask to figure out which interfaces to use. For example, if eth1 is 10.1.1.1 and eth2 is 9.1.1.1, then the CES IP 10.1.1.100 maps to eth1 and the CES IP 9.1.1.100 maps to eth2.

Virtual IP addresses include the following advantages:
  • The node does not need to wait for the switch to accept the link when an IP address is failed back. Since the address is an alias, the interface on which it resides is already up.
  • IP address failover is much faster.
  • Administration is simplified by providing a clear distinction between the system IP and the CES IP. For example, you have a two-node cluster. One of the nodes in the two-node cluster has a problem that induces failover and someone logs in to the suspected node to reboot it. The surviving node might get rebooted by accident if the system address was migrated to the surviving node.

How to use an alias

To use an alias address for CES, you need to provide a static IP address that is not already defined as an alias in the /etc/sysconfig/network-scripts directory.

Before you enable the node as a CES node, configure the network adapters for each subnet that are represented in the CES address pool:
  1. Define a static IP address for the device:
    /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth1
BOOTPROTO=none
IPADDR=10.1.1.10
NETMASK=255.255.255.0
ONBOOT=yes
GATEWAY=10.1.1.1
TYPE=Ethernet
  2. Ensure that there are no aliases that are defined in the network-scripts directory for this interface:
    # ls -l /etc/sysconfig/network-scripts/ifcfg-eth1:*
ls: /etc/sysconfig/network-scripts/ifcfg-eth1:*: No such file or directory

After the node is enabled as a CES node, no further action is required. CES addresses are added as aliases to the already configured adapters.

Parent topic: CES features