Cluster Export Services overview
Cluster Export Services (CES) provides highly available file and object services to a GPFS cluster by using Network File System (NFS), Hadoop Distributed File System (HDFS), Object, or Server Message Block (SMB) protocols.
IBM Storage Scale includes Cluster Export Services (CES) infrastructure to support the integration of the NFS, HDFS, SMB, and Object servers.
- The NFS server supports NFS v3 and the mandatory features in NFS v4.0 and NFS v4.1.
- The SMB server supports SMB 2, SMB 2.1, and the mandatory features of SMB 3.0 and 3.11.
- The Object server supports the Train release of OpenStack Swift along with Keystone v3.
- CES supports HDFS protocols. HDFS Transparency is supported from IBM Storage Scale version 3.1.1. The IBM Storage Scale Big Data Analytics Integration Toolkit for HDFS
Transparency or
Toolkit for HDFS
must be installed. TheToolkit for HDFS
is supported from version 1.0.0.0. See Support Matrix under CES HDFS in Big data and analytics support documentation.
The CES infrastructure is responsible for:
- Managing the setup for high-availability clustering that is used by the protocols.
- Monitoring the health of these protocols on the protocol nodes and raising events or alerts during failures.
- Managing the addresses that are used for accessing these protocols by including failover and failback of these addresses because of protocol node failures.
High availability
With GPFS, you can configure a subset of nodes in the cluster to provide a highly available solution for exporting GPFS file systems by using the Network File System (NFS), Server Message Block (SMB), Hadoop Distributed File System (HDFS), and Object protocols. The participating nodes are designated as Cluster Export Services (CES) nodes or protocol nodes. The set of CES nodes is frequently referred to as the CES cluster.
A set of IP addresses, the CES address pool, is defined and distributed among the CES nodes. As nodes enter and leave the GPFS cluster, the addresses in the pool can be redistributed among the CES nodes to provide high availability. GPFS nodes that are not CES nodes that are leaving or entering the GPFS cluster do not redistribute the IP addresses. It is possible to use one IP address for all CES services. However, clients that use the SMB, NFS, and Object protocols must not share the IP address for these protocols with the IP addresses that are used by the HDFS service to avoid impacting the clients of other protocols during an HDFS failover. However, applications that use the CES IP addresses are not impacted. SMB failover is not transparent.
Monitoring
CES monitors the state of the protocol services. Monitoring ensures that the CES addresses are assigned to the appropriate node and that the processes that implement the services in the CES cluster are running correctly. Upon failure detection, CES marks the node as temporarily unable to participate in the CES cluster and reassigns the IP addresses to another node.
Protocol support
CES supports the following export protocols: NFS, SMB, HDFS, and Object. Each protocol can be enabled or disabled in the cluster. If a protocol is enabled in the CES cluster, all CES nodes serve that protocol.
- mmces service enable nfs
- Enables the NFS protocol in the CES cluster.
- mmces service disable obj
- Disables the Object protocol in the CES cluster.
Commands
- mmces
- Manages the CES address pool and other CES cluster configuration options.
- mmhdfs
- Manages the HDFS configuration operations.
- mmnfs
- Manages NFS exports and sets the NFS configuration attributes.
- mmobj
- Manages the Object configuration operations.
- mmsmb
- Manages SMB exports and sets the SMB configuration attributes.
- mmuserauth
- Configures the authentication methods that are used by the protocols.
For more information, see mmces command, mmhdfs command, mmnfs command, mmobj command, mmsmb command, and mmuserauth command.
Restrictions
For an up-to-date list of supported operating systems, specific distributions, and other dependencies for CES, refer to the IBM Storage Scale FAQ in IBM® Documentation.
A CES cluster can contain a maximum of 32 CES nodes. All the nodes in a CES cluster must be running on the same operating system and they must be of the same CPU architecture. If the SMB protocol is enabled, then the CES cluster is limited to a total of 16 CES nodes.
Each CES node must have network adapters capable of supporting all IP addresses in the CES address pool. The primary address of these adapters must not be used as a CES address.