Configuration 1: 5148-22L protocol nodes ordered and racked with a new 5148 ESS (PPC64LE)

In this configuration, both a new 5148 ESS and new 5148-22L protocol nodes are ordered and racked together. The EMS node, I/O server nodes, and protocol nodes have OS, kernel, systemd, network manager, firmware, and OFED, kept in synchronization as xCAT running on the EMS is used to manage and coordinate these levels. It is recommended to match IBM Spectrum Scale code levels between the ESS and protocol nodes, but this is not mandatory.

Note: All protocol nodes in a cluster must be at the same code level.

Overall flow

A) Starting point and what to expect upon initial power-on
B) Power on the protocol nodes and perform health checks
C) Decide which adapter(s) to use for the GPFS network(s) vs CES protocol network(s)
D) Set up the high-speed network for GPFS
E) Configure network adapters to be used for CES protocols
F) Create a CES shared root file system for use with protocol nodes G) Locate the IBM Spectrum Scale protocols package on a protocol node in /root
H) Extract the IBM Spectrum Scale protocols package
I) Configure the IBM Spectrum Scale installation toolkit
J) Installation phase of IBM Spectrum Scale installation toolkit
K) Deployment phase of IBM Spectrum Scale installation toolkit
L) Tune the protocol nodes as desired
M) GUI configuration
N) Call home setup
O) Movement of quorum or management function to protocol nodes and off EMS or I/O nodes

A) Starting point and what to expect upon initial power-on

This configuration assumes Fusion mode has been run, thus retaining the shipped xCAT configuration of EMS, I/O server nodes, and protocol nodes. If Fusion mode has not been run and the ESS system has been reinstalled, then start with Configuration 2.
The EMS and I/O server nodes are installed, high speed network has been created, and GPFS is active and healthy.
The GUI has not yet been started nor configured. This is done following the protocol node configuration.
Hardware and software call home is not yet configured. This is done after protocols are deployed.
Protocol nodes exist in XCAT as prt01, prt0N objects.
Run the following command to verify.
```
# lsdef
```
Protocol nodes are part of the ces_ppc64 xCAT group.
Run the following command to verify.
```
# lsdef -t group ces_ppc64
```
An xCAT OS image specific for CES exists (rhels7.4-ppc64le-install-ces).
Run the following command to verify.
```
# lsdef -t osimage
```
Protocol nodes come with authentication prerequisites pre-installed upon them (includes sssd, ypbind, openldap-clients, krb5-workstation).
A deploy template exists, already customized for the protocol nodes.
Check for /var/tmp/gssdeployces.cfg on the EMS node.
Protocol nodes already have RHEL 7.4 OS, kernel, network manager, systemd, OFED, iprraid, and firmware loaded at the same levels as the EMS and I/O server nodes. This is verified in this step.
Protocol nodes do not have any GPFS RPMS installed except gpfs.gss.tools.
Protocol nodes are already cabled to the internal ESS switch for use with the xCAT network and the FSP network. For more information, see Figure 2.

B) Power on the protocol nodes and perform health checks

Important: Before proceeding, power on the protocol nodes, if they have not been powered on yet.

Run gssstoragequickcheck -G ces_ppc64 to verify network adapter types, slots, and machine type model of the protocol nodes.
Run gssinstallcheck -G ces_ppc64 to verify code and firmware levels on the protocol nodes.

C) Decide which adapter(s) to use for the GPFS network(s) vs CES protocol network(s)

It is recommended to plan for separation of the GPFS and CES networks, both by subnet and by card.

Note: Before proceeding, protocol nodes must be cabled up to the GPFS cluster network and to the CES network.

D) Set up the high-speed network for GPFS

Customer networking requirements are site-specific. The use of bonding to increase fault-tolerance and performance is advised but guidelines for doing this have not been provided in this document. Consult with your local network administrator before proceeding further. Before creating network bonds, carefully read ESS networking considerations.

Make sure that the protocol nodes high speed network IPs and host names are present in /etc/hosts on all nodes.

Here is an example excerpt from /etc/hosts, showing the -hs suffix IPs and host names to be used for the GPFS cluster configuration.

172.31.250.3 ems1-hs.gpfs.net ems1-hs
172.31.250.1 gssio1-hs.gpfs.net gssio1-hs
172.31.250.2 gssio2-hs.gpfs.net gssio2-hs
172.31.250.11 prt01-hs.gpfs.net prt01-hs
172.31.250.12 prt02-hs.gpfs.net prt02-hs
172.31.250.13 prt03-hs.gpfs.net prt03-hs

Note:

All nodes must be able to resolve all IPs, FQDNs, and host names, and ssh-keys must work.
If the /etc/hosts file is already set up on the EMS node, copy it to the protocol node(s) first and then modify it. Each protocol node must have the same /etc/hosts file.

To set up bond over IB, run the following command.

gssgennetworks -G ces_ppc64 --create --ipoib --suffix=-hs --mtu 4092

In this example, MTU is set to 4092. The default MTU is 2048 (2K) and the gssgennetworks command supports 2048 (2K) and 4092 (4K) MTU. Consult your network administrator for the proper MTU setting.

To set up bond over Ethernet, run the following command.

gssgennetworks -N ems1,gss_ppc64 --suffix=-hs --create-bond

For information on Infiniband issue with multiple fabrics, see Infiniband with multiple fabric in Customer networking considerations.

E) Configure network adapters to be used for CES protocols

Before deploying protocols, it is important to understand the customer network and protocol access requirements. CES protocols use a pool of CES IPs which float between nodes, providing redundancy in the case of node failure or degradation. The CES IPs are assigned and aliased by IBM Spectrum Scale to an adapter on each protocol node that has a matching predefined route and subnet. It is important that each protocol node has a base network adapter or bonded group of network adapters or ports with an established IP and routing so that CES IPs can be assigned by IBM Spectrum Scale code during protocol deployment.

Note: CES IPs are never assigned using ifcfg or nmcli commands. This is handled by the IBM Spectrum Scale code.

The following must be taken into account when planning this network:

Bandwidth requirements per protocol node (how many ports per bond, bonding mode, and adapter speed)
Redundancy of each protocol node, if needed. This determines the bonding mode used.
Authentication domain and DNS. This determines the subnet(s) required for each protocol node.
Are VLAN tags needed?
Set aside 1 IP per protocol node, per desired CES subnet. You will be using these when configuring the CES base adapter(s) on each protocol node. These IPs must be setup for forward and reverse DNS lookup.
Set aside a pool of CES IPs for later use. These IPs must be in DNS and be setup for both forward and reverse DNS lookup. You will not be assigning these IPs to network adapters on protocol nodes.
Prepare to configure each protocol node to point to the authentication domain or DNS. You need to do this manually using ifcfg or nmcli commands and by verifying /etc/resolv.conf after the settings have taken effect. When deployed from an EMS node, each protocol node might already have a default domain of gpfs.net present in /etc/resolv.conf and the ifcfg files. This default domain can be removed so that it does not interfere with the authentication setup and DNS for protocols.

Proceed with either configuring the CES protocol adapters manually using ifcfg or nmcli commands or by using gssgennetworks. The gssgennetworks command cannot be used if your CES protocol network requires VLAN tags nor does it set up additional domain or DNS servers.

When the network is configured on each protocol nodes, verify it using the mmnetverify command with these actions:

Make sure all protocol nodes can ping each other's base CES network by IP, host name, and FQDN.
Make sure all protocol nodes can ping the authentication server by IP, host name, and FQDN.
Make sure the authentication server can ping each protocol node base CES network by IP, host name, and FQDN
Spot check the desired NFS, SMB, or OBJ clients, external to the GPFS cluster and verify that they can ping each protocol node base CES network by IP, hostname, and FQDN
Even though the CES IP pool is not yet set up, because protocols are not deployed, double check that each protocol node can resolve each CES IP or host name using nslookup.

For an example showing how CES IP aliasing relies upon an established base adapter with proper subnets or routing, see CES IP aliasing to network adapters on protocol nodes.

For an example of CES-IP configuration that can be performed after deployment of protocols, see Configuring CES protocol service IP addresses.

F) Create a CES shared root file system for use with protocol nodes

CES protocols require a shared file system to store configuration and state data. This file system is called CES shared root and it is a replicated file system that is recommended to be of size between 4GB and 10GB. The following ESS command automatically creates this file system with the recommended size and mirroring.

Run the following command from the EMS node.
```
# gssgenvdisks --create-vdisk --create-nsds --create-filesystem --crcesfs
# mmmount cesSharedRoot -N ems1-hs
```
A file system named cesSharedRoot with a mount path of /gpfs/cesSharedRoot is created and mounted. Later in these steps, the IBM Spectrum Scale installation toolkit is pointed to this file system to use when deploying protocols.

G) Locate the IBM Spectrum Scale protocols package on a protocol node in /root

An example package name is: Spectrum_Scale_Protocols_Data_Management-5.0.1.2-ppc64LE-Linuxinstall

Each protocol node ships with an IBM Spectrum Scale protocols package in /root. The version and the license of this package matches with the ESS version that the protocol node was ordered with.

If the package is of the desired version and license, proceed with extraction.
If a different level is desired, proceed to IBM FixCentral to download and replace this version.
If replacing this version, the following rules apply:
- The IBM Spectrum Scale version must be 5.0.1.2 or later.
- The CPU architecture must be PPC64LE.
- The package must be a protocols package (The title and the file name must specifically contain Protocols).

H) Extract the IBM Spectrum Scale protocols package

Enter the following at the command prompt: /root/Spectrum_Scale_Protocols_Data_Management-5.0.1.2-ppc64LE-Linux-install
By default, the package is extracted to /usr/lpp/mmfs/5.0.1.2/.

I) Configure the IBM Spectrum Scale installation toolkit

Change directory to the installation toolkit directory:
```
cd /usr/lpp/mmfs/5.0.1.2/installer
```
View the installation toolkit usage help as follows.
```
/usr/lpp/mmfs/5.0.1.2/installer/spectrumscale -h
```
Set up the installation toolkit by specifying which local IP to use for communicating with the rest of the nodes. Preferably, this IP should be the same IP used for the GPFS network. Set the toolkit to the ESS mode.
```
/usr/lpp/mmfs/5.0.1.2/installer/spectrumscale setup -s IP_Address -st ess
```
Populate the installation toolkit configuration file with the current cluster configuration by pointing it to the EMS node.
```
/usr/lpp/mmfs/5.0.1.2/installer/spectrumscale config populate -N EMSNode
```
There are limits to the config populate functionality. If it does not work, simply add the EMS node to the installation toolkit and continue.
View the current cluster configuration as follows.
```
/usr/lpp/mmfs/5.0.1.2/installer/spectrumscale node list
/usr/lpp/mmfs/5.0.1.2/installer/spectrumscale config gpfs
```
Note: ESS I/O nodes do not get listed in the installation toolkit node list.

Configure the details of the protocol nodes to be added to the cluster.

./spectrumscale node add ems1-hs.gpfs.net -e -a -g   ## No need to perform this step 
                                                     ## if the config populate ran without error
./spectrumscale node add prt01-hs.gpfs.net -p     ## Designate a protocol node
./spectrumscale node add prt02-hs.gpfs.net -p     ## Designate a protocol node
./spectrumscale node add prt03-hs.gpfs.net –p     ## Designate a protocol node
./spectrumscale node add client01-hs.gpfs.net     ## Example of a non-protocol client node, if desired 
./spectrumscale node add nsd01-hs.gpfs.net        ## Example of a non-ESS nsd node, if desired
./spectrumscale enable smb                        ## If you'd like to enable and use the SMB protocol 
                                                  ## (it will be installed regardless) 
./spectrumscale enable nfs                        ## If you'd like to enable and use the NFS protocol
                                                  ## (it will be installed regardless)
./spectrumscale enable object                     ## If you'd like to enable and use the Object protocol 
                                                  ## (it will be installed regardless) 
./spectrumscale config protocols -e CESIP1,CESIP2,CESIP3   ## Input the CES IPs set aside from step (G) of
                                                           ## this procedure. Toolkit assigns IPs listed. 
./spectrumscale config protocols -f cesSharedRoot -m /gpfs/cesSharedRoot  ## FS name and mount point for  
                                                                          ## CES shared root, previously 
                                                                          ## setup during step (D)

./spectrumscale config object -e <endpoint IP or hostname>  ## This address should be an RRDNS or similar address
                                                            ## that resolves to the pool of CES IP addresses.
./spectrumscale config object -o Object_Fileset             ## This fileset will be created during deploy
./spectrumscale config object -f ObjectFS -m /gpfs/ObjectFS ## This must point to an existing FS 
                                                            ## create the FS on EMS if it doesn't already exist
./spectrumscale config object -au admin -ap -dp             ## Usernames and passwords for Object
  
./spectrumscale config perfmon –r on              ## Turn on performance sensors for the protocol nodes.  
                                                  ## EMS GUI picks up sensor data once protocols are deployed 
./spectrumscale node list                         ## Lists out the node config (ESS IO nodes never show up here)
./spectrumscale config protocols                  ## Shows the protocol config

For more information, see IBM Spectrum Scale installation toolkit.

J) Installation phase of IBM Spectrum Scale installation toolkit

Run the installation toolkit installation precheck.
```
./spectrumscale install --precheck
```
Run the installation toolkit installation procedure.
```
./spectrumscale install
```

Installation toolkit performs the following actions and it can be re-run in the future to:

Install GPFS, call home, performance monitoring, license RPMs on each node specified to the installation toolkit. The EMS and I/O server nodes are not acted upon by the installation toolkit.
Add nodes to the cluster (protocol, client, NSD).
Add non-ESS NSDs, if desired.
Start GPFS and mount all file systems on the newly added nodes.
Configure performance monitoring sensors.
Set client or server licenses.

GPFS configuration parameters such as pagepool, maxFilesToCache, verbsPorts need to be set up manually. You can do this after completing the installation phase or after completing the deployment phase. For more information about these parameters, see GPFS configuration parameters for protocol nodes.

K) Deployment phase of IBM Spectrum Scale installation toolkit

Run the installation toolkit deployment precheck.
```
./spectrumscale deploy --precheck
```
Run the installation toolkit installation procedure.
```
./spectrumscale deploy
```

Installation toolkit performs the following actions during deployment and it can be re-run in the future to:

Install SMB, NFS, and object RPMs on each protocol node specified to the installation toolkit.
Enable one or more protocols.
Assign CES IPs. IBM Spectrum Scale code aliases these IPs to the CES base network adapter configured during step G.
Enable authentication for file or object.
Create additional file systems using non-ESS NSD nodes. You must run installation first to add more non-ESS NSDs.
Add additional protocol nodes. You must run installation first to add more nodes and then run deployment for the protocol specific piece.

L) Tune the protocol nodes as desired

Protocol nodes should already be tuned with the same tuned and sysctl settings, and udev rules as the I/O server nodes. For more information, see OS tuning for RHEL 7.4 PPC64LE protocol nodes.

At this point, the main tuning settings to be aware of includes:

RDMA. If IB RDMA is in use (check using mmlsconfig verbsRDMA), issue mmlsconfig and verify that the verbPorts parameter refers to the correct ports on each protocol node.
pagepool. Use mmlsconfig to view the pagepool settings of the EMS and I/O server nodes. The protocol nodes do not have pagepool defined at this point. Define pagepool using mmchconfig -N cesNodes pagepool=XX command.
Where XX is typically 25% the system memory for protocol nodes in a cluster containing ESS. For more information, see GPFS configuration parameters for protocol nodes.
maxFilesToCache. Use mmlsconfig to view the maxFilesToCache settings of the EMS and I/O server nodes. The protocol nodes do not have maxFilesToCache defined at this point. Define maxFilesToCache using mmchconfig -N cesNodes maxFilesToCache=XX command.
Where XX is typically 2M for protocol nodes in a cluster containing ESS. For more information, see GPFS configuration parameters for protocol nodes.

M) GUI configuration

Configure the GUI as follows.

Enable and start gpfsgui:
```
systemctl enable gpfsgui.service
systemctl start gpfsgui
```
Note: If the GUI was started before protocol nodes were installed, then restart the GUI with systemctl restart gpfsgui command.
To launch the ESS GUI in a browser, go to: https://EssGuiNode where ESSGuiNode is the host name or IP address of the management server node for GUI access. To log in, type admin in the User Name field and your password in the Password field on the login page. The default password for admin is admin001. Walk through each panel and complete the GUI Setup Wizard.

During GUI configuration, you are allowed to specify rack locations of all components, including the protocol nodes. After the GUI configuration has completed, protocol nodes are a part of the GUI. Protocols (NFS, SMB) and Object panels show up in the GUI depending on which protocols were enabled during the installation toolkit deployment phase. Performance sensors for protocols are available on the Monitoring > Statistics page.

N) Call home setup

Now that protocol nodes are deployed, call home needs to be configured.

Check if call home has already been configured and if so, record the settings. Reconfiguring call home might require entering the settings again.
- Check the hardware call home settings on the EMS node.
```
gsscallhomeconf --show -E ems1
```
- Check the software call home setup.
```
mmcallhome info list
```

Set up call home from the EMS node.

gsscallhomeconf -N ems1,gss_ppc64,ces_ppc64 --suffix=-hs -E ems1 --register all --crvpd

For more information, see call home documentation in Elastic Storage Server: Problem Determination Guide.

O) Movement of quorum or management function to protocol nodes and off EMS or I/O nodes

Quorum and management functions can be resource intensive. In an ESS cluster that also has extra nodes, such as protocols, within the same cluster, it is recommended to move these functions to the protocol nodes. For information on changing node designation, see mmchnode command.