Using public and private IP addresses for GPFS nodes

GPFS permits the system administrator to set up a cluster such that both public and private IP addresses are in use. For example, if a cluster has an internal network connecting some of its nodes, it is advantageous to use private IP addresses to communicate on this network, and public IP addresses to communicate to resources outside of this network.

Public IP addresses are those that can be used to access the node from any other location for which a connection exists. Private IP addresses might be used only for communications between nodes directly connected to each other with a communications adapter. Private IP addresses are assigned to each node at hardware setup time, and must be in a specific address range (IP addresses on a 10.0.0.0, 172.16.0.0, or 192.168.0.0 subnet). For more information on private IP addresses, refer to RFC 1597 - Address Allocation for Private Internets (www.ip-doc.com/rfc/rfc1597).

The subnets operand on the mmchconfig command specifies an ordered list of subnets available to GPFS for private TCP/IP communications. Each listed subnet might have a list of cluster names (allowing shell-style wild cards) that specifies other GPFS clusters that have direct access to the same subnet.

When the GPFS daemon starts on a node, it obtains a list of its own IP addresses and associated subnet masks from its local IP configuration. For each IP address, GPFS checks whether that address is on one of the subnets specified on the subnets configuration parameter. It records the list of its matching IP addresses and subnet masks, and then listens for connections on any of these addresses. If any IP address for the node (specified when the cluster was created or when the node was added to the cluster), is not specified with the subnets configuration parameter, GPFS automatically adds it to the end of the node's IP address list.

Therefore, when you use public IP addresses for a node, there is no need to explicitly list the public IP subnet with the subnets configuration parameter. For example, the normal way to configure a system would be to use host names that resolve to the external Ethernet IP address in the mmcrcluster command, and then, if the system administrator wants GPFS to use the High Performance Switch within the cluster, add one subnets configuration parameter for the HPS subnet. It is acceptable to add two subnets configuration parameters, one for the HPS and one for the external Ethernet, making sure that they are in that order. In this case, it does not matter which of each node's two addresses was specified when the cluster was created or when the node was added to the cluster. For example, to add remote access to an existing cluster that was using only switch addresses, it is sufficient to add two subnets configuration parameters.

When a node joins a cluster (its own cluster on startup, or another cluster when mounting a file system owned by another cluster), the node sends its list of IP addresses (ordered according to the order of subnets configuration parameters) to the cluster manager node, which forwards the list to all other nodes as part of the join protocol. No other additional information needs to be propagated.

When a node attempts to establish a connection to another node, GPFS determines the destination IP address to use according to this procedure:
  1. For each of its own IP addresses, it searches the other node's list of IP addresses for an address that is on the same subnet.
    • For normal public IP addresses this is done by comparing IP address values ANDed with the node's subnet mask for its IP address.
    • For private IP addresses GPFS assumes that two IP addresses are on the same subnet only if the two nodes are within the same cluster, or if the other node is in one of the clusters that are explicitly listed in the subnets configuration parameter.
  2. If the two nodes have more than one IP address pair on a common subnet, GPFS uses the first one found according to the order of subnets specified in the initiating node's configuration parameters.
  3. If a given pair of nodes do not share a common subnet defined by the subnets configuration parameter, IBM Spectrum Scale uses the network based on the daemon address of the node, which is automatically added as the last entry in each node's network address list.
Note: When you use subnets, both the interface corresponding to the daemon address and the interface that matches the subnet settings must be operational.

For more information and an example, see Using remote access with multiple network definitions.