CES and protocol nodes support in ESS

Overview of CES and protocol nodes

Cluster Export Services (CES) provides highly available file and object services to an IBM Spectrum Scale™ cluster by using Network File System (NFS), Object, or Server Message Block (SMB) protocols. Because CES has specific hardware and software requirements, the code must be installed on nodes designated to run the CES software stack. These nodes are called protocol nodes.

Protocol nodes can be added to an IBM Spectrum Scale cluster containing an ESS building block. They can also exist in non-ESS IBM Spectrum Scale clusters. SMB and NFS functions on protocol nodes can exist in clusters in which their storage is remotely mounted.

For more information, see the following resources.

Supported protocol node configurations

The following protocol node configurations that are currently supported.

5148-22L protocol node hardware

With the ESS 5.3.1.1 release, a protocol node feature code is introduced. This protocol node feature code allows the purchase of Power8 nodes with a very specific hardware configuration, tested and tuned by IBM for providing CES services. The machine type and model (MTM) for protocol nodes is 5148-22L and it ships with the following hardware configuration.
  • 8247-22L Power8 model
  • 2 x 10core 3.34 Ghz Power8 Processors
  • 128 GB or greater memory
  • Two 600 GB 10k RPM SAS HDDs in RAID10 mirror using IPRaid adapter
  • 1 GbE 4port network adapter in slot C12
  • 1 Three x 16 or x8 network adapters in slots C5, C6, C7
  • 1 Four x 8 network adapters in slots C2, C3, C10, C11 available by additional card orders (through MES)

1 It is recommended to plan for the GPFS admin or daemon network to use separate network adapter(s) from the Cluster Export Services.

For more information, see 5148-22L protocol node diagrams.

5148-22L protocol node software

With the ESS 5.3.1.1 release, the ESS EMS node contains the capability to manage certain aspects of a 5148-22L protocol node. This allows the protocol node(s) and ESS building block to have OS, driver, and firmware levels kept in synchronization. To understand which toolsets manage each aspect of the protocol node and the supported code levels, see the following table.
5148-22L protocol node components Supported level Managed by
Power8 FW 01SV860_138 (FW860.42) (in OPAL mode) User (using update_flash)
OS RHEL7.4 PPC64LE EMS node (using gssdeploy)
kernel 3.10.0-693.35.1.el7  
systemd 219-42.el7_4.11 EMS node (using gssdeploy)
network manager 1.8.0-12.el7_4 EMS node (using gssdeploy)

Mellanox OFED

(Firmware and driver)

Mellanox OFED2

(Packaged to support Mellanox CX-2 adapter)
Note: For the CX-2 adapter, only the driver is updated. Firmware is not updated.

Driver: MLNX_OFED_LINUX-4.3-1.0.1.1

OFED2 driver: MLNX_OFED_LINUX-4.1-4.1.6.0

 EMS node (using gssdeploy)

Power8 IPR

(Firmware and driver)

 18518200 EMS node (using gssdeploy)
sysctl, tuned, udev rules OS tuning for RHEL 7.4 PPC64LE protocol nodes EMS node (using gssdeploy)

gpfs.gss.tools

5.3.1.1

 EMS node (using gssdeploy)
Red Hat Enterprise Linux OS syslog ---

User managed for debug

(contained on each Protocol node)
SSH key setup ---

EMS node (using gssdeploy)

or User
/etc/hosts ---

EMS node (using gssdeploy)

or User
Repositories ---

EMS node (using gssdeploy)

(persistent)

IBM Spectrum Scale installation toolkit

(only while running)
/etc/resolv.conf --- User
GPFS network configuration --- User (using gssgennetwork)
CES base network configuration --- User
CES shared root creation --- EMS node (using gssgenvdisks --crcesfs)

IBM Spectrum Scale code

(RPM install, node add, license)

 5.0.1.2 or later
Note: It is recommended to keep IBM Spectrum Scale at the same level across all nodes, but a higher level of IBM Spectrum Scale can be used for protocol nodes, if desired. For more information, see IBM Spectrum Scale supported upgrade paths.

IBM Spectrum Scale installation toolkit

(installation phase)

IBM Spectrum Scale CES code

(rpm install, CES IPs, protocol enablement, license)

IBM Spectrum Scale installation toolkit

(deployment phase)
CES protocol performance monitoring sensors

IBM Spectrum Scale installation toolkit

(deployment phase)
CES authentication configuration

User (using mmuserauth)

or IBM Spectrum Scale installation toolkit
Call home config EMS (via gsscallhomeconf)
GUI integration of protocol nodes

EMS node (automatically detects CES nodes)

User (using GUI rack location adjustment)
GPFS configuration parameters GPFS configuration parameters for protocol nodes User (using mmchconfig)

User (using GUI)
Health checks ---

EMS (using gssinstallcheck)

EMS (using gssstoragequickcheck)

IBM Spectrum Scale installation toolkit (prechecks and postchecks)

User (using mmhealth and GUI)

Customer supplied protocol node hardware recommendations

Non-5148-22L protocol nodes can be based upon PPC64LE, PPC64BE, or x86_64 architectures and they are suggested to adhere to the guidance in IBM Spectrum Scale FAQ. All hardware, cabling or connections, and power sequencing is owned by the customer.

Support for IBM Spectrum Scale on the chosen hardware is handled by the general IBM Spectrum Scale support and not by ESS Solution Support.

Customer supplied protocol node software management recommendations

Only 5148-22L protocol nodes are supported for management by the EMS node. If a protocol node is not ordered with this exact model and type from IBM manufacturing, it cannot be added to the EMS xCAT server. While non-5148-22L protocol nodes of type PPC64LE, PPC64BE, or x86_64, might be joined to an ESS cluster using the IBM Spectrum Scale installation toolkit or the mmaddnode command, they must be owned, installed, deployed, upgraded, and managed by the customer. The ESS toolsets cannot be used for the management of OS, kernel, network manager, systemd, OFED, or firmware on non-5148-22L nodes. Using the ESS toolsets, including the EMS xCAT server, to manage customer supplied non-5148-22L protocol nodes is not supported.

The IBM Spectrum Scale code on customer supplied non-5148-22L protocol nodes is also managed separately from the ESS toolsets. The IBM Spectrum Scale installation toolkit can be used in this configuration to install or deploy and upgrade the IBM Spectrum Scale code on any customer supplied non-5148-22L protocol nodes. For more information about the installation toolkit, see IBM Spectrum Scale installation toolkit.

For latest information on supported levels, see IBM Spectrum Scale FAQ.

Parent topic: Protocol nodes deployment and upgrade