Cluster Export Services overview

Cluster Export Services (CES) provides highly available file and object services to a GPFS cluster by using Network File System (NFS), Hadoop Distributed File System (HDFS), Object, or Server Message Block (SMB) protocols.

IBM Storage Scale includes Cluster Export Services (CES) infrastructure to support the integration of the NFS, HDFS, SMB, and Object servers.

  • The NFS server supports NFS v3 and the mandatory features in NFS v4.0 and NFS v4.1.
  • The SMB server supports SMB 2, SMB 2.1, and the mandatory features of SMB 3.0 and 3.11.
  • The Object server supports the Train release of OpenStack Swift along with Keystone v3.
  • CES supports HDFS protocols. HDFS Transparency is supported from IBM Storage Scale version 3.1.1. The IBM Storage Scale Big Data Analytics Integration Toolkit for HDFS Transparency or Toolkit for HDFS must be installed. The Toolkit for HDFS is supported from version 1.0.0.0. See Support Matrix under CES HDFS in Big data and analytics support documentation.

The CES infrastructure is responsible for:

  • Managing the setup for high-availability clustering that is used by the protocols.
  • Monitoring the health of these protocols on the protocol nodes and raising events or alerts during failures.
  • Managing the addresses that are used for accessing these protocols by including failover and failback of these addresses because of protocol node failures.

High availability

With GPFS, you can configure a subset of nodes in the cluster to provide a highly available solution for exporting GPFS file systems by using the Network File System (NFS), Server Message Block (SMB), Hadoop Distributed File System (HDFS), and Object protocols. The participating nodes are designated as Cluster Export Services (CES) nodes or protocol nodes. The set of CES nodes is frequently referred to as the CES cluster.

A set of IP addresses, the CES address pool, is defined and distributed among the CES nodes. As nodes enter and leave the GPFS cluster, the addresses in the pool can be redistributed among the CES nodes to provide high availability. GPFS nodes that are not CES nodes that are leaving or entering the GPFS cluster do not redistribute the IP addresses. It is possible to use one IP address for all CES services. However, clients that use the SMB, NFS, and Object protocols must not share the IP address for these protocols with the IP addresses that are used by the HDFS service to avoid impacting the clients of other protocols during an HDFS failover. However, applications that use the CES IP addresses are not impacted. SMB failover is not transparent.

Monitoring

CES monitors the state of the protocol services. Monitoring ensures that the CES addresses are assigned to the appropriate node and that the processes that implement the services in the CES cluster are running correctly. Upon failure detection, CES marks the node as temporarily unable to participate in the CES cluster and reassigns the IP addresses to another node.

Protocol support

CES supports the following export protocols: NFS, SMB, HDFS, and Object. Each protocol can be enabled or disabled in the cluster. If a protocol is enabled in the CES cluster, all CES nodes serve that protocol.

The following are examples of enabling and disabling protocol services by using the mmces command:
mmces service enable nfs
Enables the NFS protocol in the CES cluster.
mmces service disable obj
Disables the Object protocol in the CES cluster.

Commands

To set or configure CES options, the following commands are used:
mmces
Manages the CES address pool and other CES cluster configuration options.
mmhdfs
Manages the HDFS configuration operations.
mmnfs
Manages NFS exports and sets the NFS configuration attributes.
mmobj
Manages the Object configuration operations.
mmsmb
Manages SMB exports and sets the SMB configuration attributes.
mmuserauth
Configures the authentication methods that are used by the protocols.

For more information, see mmces command, mmhdfs command, mmnfs command, mmobj command, mmsmb command, and mmuserauth command.

Restrictions

For an up-to-date list of supported operating systems, specific distributions, and other dependencies for CES, refer to the IBM Storage Scale FAQ in IBM® Documentation.

A CES cluster can contain a maximum of 32 CES nodes. All the nodes in a CES cluster must be running on the same operating system and they must be of the same CPU architecture. If the SMB protocol is enabled, then the CES cluster is limited to a total of 16 CES nodes.

Each CES node must have network adapters capable of supporting all IP addresses in the CES address pool. The primary address of these adapters must not be used as a CES address.