Configuring and verifying the installation prerequisites

Use the following steps to configure and verify the prerequisite software components for an IBM Storage Scale DAS deployment.

  • IBM Storage Scale storage cluster
  • IBM Storage Scale file system
  • Red Hat OpenShift Container Platform (OCP) cluster
  • IBM Storage Scale container native
  • IBM Storage Scale CSI
All steps must be executed in the specified order.

The IBM Storage Scale storage cluster must be installed and configured. For more information about installing and configuring IBM Storage Scale, see IBM Storage Scale documentation.

  1. Verify that the IBM Storage Scale storage cluster has the required software version and it is in a healthy state.
    1. From one of the storage cluster nodes, view the software version.
      For example,
      mmdsh -N all rpm -q gpfs.base
      A sample output is as follows:
      emsdas-hs.test.net:  gpfs.base-5.1.3-0.220203.103103.ppc64le
ess3200das1a-hs.test.net:  gpfs.base-5.1.3-0.x86_64
ess3200das2a-hs.test.net:  gpfs.base-5.1.3-0.x86_64
ess3200das1b-hs.test.net:  gpfs.base-5.1.3-0.x86_64
ess3200das2b-hs.test.net:  gpfs.base-5.1.3-0.x86_64
    2. From one of the storage cluster nodes, view the storage cluster health state.
      For example,
      mmhealth cluster show
      A sample output is as follows:
      Component            Total         Failed       Degraded        Healthy          Other
--------------------------------------------------------------------------------------
NODE                     5              0              0              5              0
GPFS                     4              0              0              5              0
NETWORK                  4              0              0              4              0
FILESYSTEM               4              0              0              1              0
DISK                    20              0              0             20              0
FILESYSMGR               3              0              0              3              0
GUI                      1              0              0              1              0
NATIVE_RAID              3              0              0              4              0
PERFMON                  4              0              0              4              0
THRESHOLD                4              0              0              4              0

To report unauthorized access to S3 buckets and S3 objects, IBM Storage Scale DAS requires that the IBM Storage Scale storage cluster is configured with Security-Enhance Linux (SELinux) in permissive mode.

  1. If you plan to enable SELinux on the storage cluster, then enable SELinux in permissive mode on the EMS and ESS I/O nodes. For more information, see Enabling SELinux in ESS.
    You can view the SELinux mode enabled on the storage cluster by issuing the following command from one of the storage cluster nodes.
    mmdsh -N all getenforce
    A sample output is as follows:
    emsdas-hs.test.net:  Permissive
ess3200das2b-hs.test.net:  Permissive
ess3200das1a-hs.test.net:  Permissive
ess3200das2a-hs.test.net:  Permissive
ess3200das1b-hs.test.net:  Permissive

The IBM Storage Scale file system used for IBM Storage Scale DAS must be configured in the NFSv4 mode.

  1. From one of the storage cluster nodes, verify that the -D and -k options are set to nfsv4 for the IBM Storage Scale file system that is being used for IBM Storage Scale DAS.
    For example,
    mmlsfs fs1 -D -k
    A sample output is as follows:
    flag                value                    description
------------------- ------------------------ -----------------------------------
 -D                 nfs4                     File locking semantics in effect
 -k                 nfs4                     ACL semantics in effect

IBM Storage Scale CSI requires the quota configuration of IBM Storage Scale file system to be customized. For more information, see Performing pre-installation tasks for CSI Operator deployment.

  1. From one of the storage cluster nodes, verify the quota configuration of the IBM Storage Scale file system that is configured for IBM Storage Scale CSI.
    For example,
    mmlsfs fs1 -Q --perfileset-quota
    A sample output is as follows:
    flag                value                    description
------------------- ------------------------ -----------------------------------
 -Q                 user;group;fileset       Quotas accounting enabled
                    user;group;fileset       Quotas enforced
                    none                     Default quotas enabled
 --perfileset-quota no                       Per-fileset quota enforcement

IBM Storage Scale DAS supports protecting data by using SELinux Multi-Category Security (MCS) labels. Therefore, the IBM Storage Scale file system that is used for IBM Storage Scale DAS can be configured with SELinux MCS labels.

  1. If you have enabled SELinux on the storage cluster, then from one of the storage cluster nodes, configure the SELinux MCS labels on the IBM Storage Scale file system.
    1. List the default mount point of the file system.
      For example,
      mmlsfs fs1 -T
      A sample output is as follows:
      flag                value                    description
------------------- ------------------------ -----------------------------------
 -T                 /data/fs1                Default mount point
    2. Set the SELinux MCS labels for the mount point of the file system.
      For example,
      chcon system_u:object_r:container_file_t:s0:c111,c234 /data/fs1
    3. List the security context for the mount point of the file system to verify that the SELinux MCS levels are set correctly.
      For example,
      ls -laZ /data/fs1/
      A sample output is as follows:
      total 257
drwxr-xr-x. 2 root root system_u:object_r:container_file_t:s0:c111,c234 262144 Mar 10 10:07 .
drwxr-xr-x  3 root root ?                                                 17 Mar 10 10:07 ..
dr-xr-xr-x  2 root root ?                                               8192 Dec 31  1969 .snapshots

IBM Storage Scale DAS requires customers to provide a compact Red Hat OpenShift Container Platform (OCP) cluster. For more information about installing and configuring OCP, see Red Hat OpenShift Container Platform documentation.

  1. Verify that the OCP cluster has the required software version and it is in a healthy state.
    1. From a node configured to work with the OCP cluster, view the software version.
      For example,
      oc get clusterversion
      A sample output is as follows:
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.13   True        False         14d     Cluster version is 4.13.13
    2. From a node configured to work with the OCP cluster, view the OCP node status.
      For example,
      oc get nodes
      A sample output is as follows:
      NAME                       STATUS   ROLES           AGE   VERSION
dan1.dasocp4.example.com   Ready    master,worker   21d   v1.22.3+e790d7f
dan2.dasocp4.example.com   Ready    master,worker   21d   v1.22.3+e790d7f
dan3.dasocp4.example.com   Ready    master,worker   21d   v1.22.3+e790d7f

IBM Storage Scale DAS requires customers to install and configure IBM Storage Scale container native. For more information about installing and configuring IBM Storage Scale container native cluster, see IBM Storage Scale container native documentation.

  1. Verify that the IBM Storage Scale container native cluster has the required software version, all pods are running, and file system is mounted and properly configured for IBM Storage Scale DAS.
    1. From a node configured to work with the OCP cluster, view the IBM Storage Scale container native pods that are running.
      For example,
      oc -n ibm-spectrum-scale get pods
      A sample output is as follows:
      NAME                                                   READY   STATUS    RESTARTS   AGE
das-dan1                                               2/2     Running   0          3d13h
das-dan2                                               2/2     Running   0          3d13h
das-dan3                                               2/2     Running   0          3d13h
ibm-spectrum-scale-gui-0                               4/4     Running   0          3d13h
ibm-spectrum-scale-gui-1                               4/4     Running   0          3d13h
ibm-spectrum-scale-pmcollector-0                       2/2     Running   0          3d13h
ibm-spectrum-scale-pmcollector-1                       2/2     Running   0          3d13h
    2. From a node configured to work with the OCP cluster, view the software version.
      For example,
      oc -n ibm-spectrum-scale rsh -c gpfs $(oc -n ibm-spectrum-scale get pods -l app.kubernetes.io/name=core -o=jsonpath='{.items[0].metadata.name}') mmdiag --version
      A sample output is as follows:
      Current GPFS build: "5.1.9.1 ".
Built on Feb 20 2023 at 14:31:36
Running 3 days 22 hours 29 minutes 14 secs, pid 1364
    3. From a node configured to work with the OCP cluster, verify that the IBM Storage Scale file system is mounted on the IBM Storage Scale container native cluster.
      For example,
      1. List the nodes on which the file system is mounted.
        oc -n ibm-spectrum-scale rsh -c gpfs $(oc -n ibm-spectrum-scale get pods -l app.kubernetes.io/name=core -o=jsonpath='{.items[0].metadata.name}') mmlsmount fs1 -L
        A sample output is as follows:
        File system fs1 (dasnode1.example.com:fs1) is mounted on 6 nodes:
  192.0.2.64        sc42-n3.example.com       dasnode1.example.com
  192.0.2.78        sc42-n1.example.com       dasnode1.example.com
  192.0.2.27        sc42-n2.example.com       dasnode1.example.com
  198.51.100.212    worker0                   ibm-spectrum-scale.example.com
  198.51.100.134    worker1                   ibm-spectrum-scale.example.com
  198.51.100.161    worker2                   ibm-spectrum-scale.example.com
      2. List the file system mount point.
        oc -n ibm-spectrum-scale rsh -c gpfs $(oc -n ibm-spectrum-scale get pods -l app.kubernetes.io/name=core -o=jsonpath='{.items[0].metadata.name}') mmlsfs fs1 -T
        A sample output is as follows:
        flag                value                    description
------------------- ------------------------ -----------------------------------
 -T                 /mnt/fs1                 Default mount point
      3. If you have enabled SELinux on the storage cluster, then view the security context of the file system.
        oc -n ibm-spectrum-scale rsh -c gpfs $(oc -n ibm-spectrum-scale get pods -l app.kubernetes.io/name=core -o=jsonpath='{.items[0].metadata.name}') ls -laZ /mnt/fs1
        A sample output is as follows:
        total 257
drwxrwxrwx. 3 root root system_u:object_r:container_file_t:s0 262144 Mar 11 17:31 .
drwxr-xr-x. 3 root root system_u:object_r:var_t:s0                        17 Mar 11 16:12 ..
dr-xr-xr-x. 2 root root system_u:object_r:container_file_t:s0                8192 Jan  1  1970 .snapshots
drwxrwx--x. 3 root root system_u:object_r:container_file_t:s0                4096 Mar 11 17:08 primary-fileset-fs1-1036623172086751852
        Note: The IBM Storage Scale fileset primary-fileset-fs1-nnnnnnnnnnn is implicitly created during the installation of IBM Storage Scale container native. The fileset name includes the name of the remote file system - in this example, it is fs1. The nnnnnnnnnnn value refers to the cluster ID of the IBM Storage Scale container native cluster. This fileset is discussed in more detail in the following step.
    4. From a node configured to work with the OCP cluster, verify that the IBM Storage Scale file system that is configured for IBM Storage Scale DAS is properly configured by checking the status of the gpfs, remotecluster, and filesystem resources created in the ibm-spectrum-scale namespace.
      1. Verify that Status.Conditions.Status is True for the IBM Storage Scale storage cluster that owns the IBM Storage Scale file system configured for IBM Storage Scale DAS.
        For example,
        oc -n ibm-spectrum-scale get remoteclusters
        A sample output is as follows:
        NAME   HOST                  READY   AGE
sc42   sc42-n1.example.com   True    177m 

        oc -n ibm-spectrum-scale describe remoteclusters | grep ^Status -A 7
        A sample output is as follows:
        Status:
  Conditions:
    Last Transition Time:  2022-03-11T16:12:01Z
    Message:               The remote cluster has been configured successfully.
    Reason:                AuthCreated
    Status:                True
    Type:                  Ready
Events:
      2. Verify that Status.Conditions.Status is True for the IBM Storage Scale container native cluster.
        For example,
        oc -n ibm-spectrum-scale get gpfs
        A sample output is as follows:
        NAME                 EDITION       AGE
ibm-spectrum-scale   data-access   179m

        oc -n ibm-spectrum-scale describe gpfs ibm-spectrum-scale | grep ^Status -A 7
        A sample output is as follows:
        Status:
  Conditions:
    Last Transition Time:  2022-03-11T17:11:07Z
    Message:               The cluster resources have been created successfully.
    Reason:                Configured
    Status:                True
    Type:                  Success
Events:                    <none>
      3. Verify that Status.Conditions.Status is True for the IBM Storage Scaleer file system configured for IBM Storage Scale DAS.
        For example,
        oc -n ibm-spectrum-scale get filesystem
        A sample output is as follows:
        NAME   ESTABLISHED   AGE
fs1    True          3h1m 

        oc -n ibm-spectrum-scale describe filesystems fs1 | grep ^Status -A 7
        A sample output is as follows:
        Status:
  Conditions:
    Last Transition Time:  2022-03-11T17:11:06Z
    Message:               Filesystem is created.
    Reason:                Created
    Status:                True
    Type:                  Success
  Maintenance Mode:        not supported

IBM Storage Scale container native implicitly installs IBM Storage Scale Container Storage Interface (CSI).

  1. Verify that the IBM Storage Scale CSI has the required software version and all pods are running.
    1. From a node configured to work with the OCP cluster, view the running IBM Storage Scale CSI pods.
      For example,
      oc -n ibm-spectrum-scale-csi get pods
      A sample output is as follows:
      NAME                                               READY   STATUS    RESTARTS        AGE
ibm-spectrum-scale-csi-attacher-0                  1/1     Running   20 (138m ago)   179m
ibm-spectrum-scale-csi-attacher-1                  1/1     Running   21 (25m ago)    179m
ibm-spectrum-scale-csi-ff5lh                       3/3     Running   13 (142m ago)   179m
ibm-spectrum-scale-csi-gvwxr                       3/3     Running   13 (142m ago)   179m
ibm-spectrum-scale-csi-operator-78979c7c59-97g8h   1/1     Running   7 (26m ago)     9h
ibm-spectrum-scale-csi-provisioner-0               1/1     Running   0               179m
ibm-spectrum-scale-csi-resizer-0                   1/1     Running   1 (25m ago)     179m
ibm-spectrum-scale-csi-snapshotter-0               1/1     Running   0               179m
ibm-spectrum-scale-csi-vt9f5                       3/3     Running   0               25m
    2. From a node configured to work with the OCP cluster, view the software version.
      For example,
      oc -n ibm-spectrum-scale-csi get pod -l app=ibm-spectrum-scale-csi -o=jsonpath='{.items[0].metadata.annotations.productVersion}{"\n"}'
      A sample output is as follows:
      2.10.0
    3. From a node configured to work with the OCP cluster, view the default IBM Storage Scale CSI storage classes.
      For example,
      oc get storageclass
      A sample output is as follows:
      NAME                          PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ibm-spectrum-scale-internal   kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  4h38m
ibm-spectrum-scale-sample     spectrumscale.csi.ibm.com      Delete          Immediate              false                  3h2m
      Note: By default, IBM Storage Scale CSI configures two storage classes.
    4. From one of the IBM Storage Scale storage cluster nodes, view the details of the IBM Storage Scale CSI primary fileset.
      Note: IBM Storage Scale CSI configures an IBM Storage Scale fileset, referred to as the primary fileset, for IBM Storage Scale CSI internal metadata. For more information, see IBM Storage Scale CSI documentation.
      1. List the filesets in the IBM Storage Scale file system configured for IBM Storage Scale DAS.
        For example,
        mmlsfileset fs1 -L
        A sample output is as follows:
        Filesets in file system 'fs1':
Name                            Id      RootInode  ParentId Created                   InodeSpace  MaxInodes  AllocInodes Comment
root                             0              3        -- Thu Mar 10 10:07:17 2022      0       615424     503808      root
                                                                                                                         fileset
primary-fileset-fs1-             1         524291         0 Fri Mar 11 09:08:52 2022      1       1048576    55296       Fileset 
1036623172086751852                                                                                                      created by IBM 
                                                                                                                         Container Storage 
                                                                                                                         Interface driver
      2. List the details of the primary fileset.
        For example,
        mmlsfileset fs1 primary-fileset-fs1-1036623172086751852
        A sample output is as follows:
        Filesets in file system 'fs1':
Name                                       Status    Path
primary-fileset-fs1-1036623172086751852    Linked    /data/fs1/primary-fileset-fs1-1036623172086751852
      3. If you have enabled SELinux on the storage cluster, then view the security context of the primary fileset.
        For example,
        ls -laZ /data/fs1/primary-fileset-fs1-1036623172086751852
        A sample output is as follows:
        total 258
drwxrwx--x. 3 root root system_u:object_r:unlabeled_t:s0                4096 Mar 11 09:08 .
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c111,c234 262144 Mar 11 09:31 ..
dr-xr-xr-x. 2 root root system_u:object_r:unlabeled_t:s0                8192 Dec 31  1969 .snapshots
drwxrwx--x. 2 root root system_u:object_r:unlabeled_t:s0                4096 Mar 11 09:08 .volumes