Summary of changes

• Introduced support for setting 60 mins RPO interval for AFM-DR fileset. For more information, see mmchfileset command, mmchconfig command and mmcrfileset command.
• Introduced --list-policy option to view installed policy. For more information, see mmafmcosctl command.

• Introduced usage of prefix to organize AFM to cloud object storage fileset data inside a bucket. You can upload and download fileset data to the prefix inside the target bucket by creating an AFM to cloud object storage fileset by using user-defined prefix. For more information, see Synchronization of AFM to cloud object storage data to the bucket by using prefix and mmafmcosconfig command .
• Introduced directory object support, to set Access Control Lists (ACLs) and extended attributes on empty as well as non-empty directories. These empty or non-empty directories are transferred to home along with the files as normal operations. For more information, see AFM to cloud object storage directory object support and mmafmcosconfig command.

  To support this feature, a new afmObjectDirectoryObj parameter is introduced. For more information about the parameter, see Configuration parameters for AFM, AFM-DR and AFM to COS.

• Introduced setlocal and resetlocal options in the mmafmctl command to provide support for local files and directories in the AFM to cloud object storage fileset. For more information, see mmafmctl command.
• Introduced support of more than 2k metadata on AFM to Cloud Object Storage fileset. For more information, see AFM to cloud object storage support for more than 2 K metadata.
• Introduced policy-based data tiering to create and run policies and upload data by using policy. For more information, see AFM to cloud object storage policy based upload for manual updates mode.
• Introduced support to synchronize data between AFM and S3 Object Storage to Google cloud platform. For more information, see Support of Google cloud storage platform for AFM to cloud object storage and Configuring AFM to cloud object storage to use Google cloud storage.

Use of CCR backup files for cluster and local node restoration

CCR backup files can be used to restore the entire cluster or only local nodes. For more information, see mmsdrrestore command.

Support for the filesetdf feature on AIX

IBM Spectrum Scale supports the --filesetdf feature on AIX through the statfs and statvfs interfaces.

GPFS scanning in online mode by using the new mmfsckx command

The mmfsckx command scans GPFS file systems and reports metadata corruptions, if any, even while the file system is mounted and in use. For more information, see mmfsckx command.

Control fileset access for remote clusters

Administrators can now configure access to remote cluster nodes for only a subset of filesets instead of the entire file system. For more information, see Fileset access control for remote clusters.

Auto increase in inode space

The maximum number of inodes per inode space in the file system can be automatically increased. For more information, see mmcrfs command.

Increase in the number of independent filesets

In IBM Spectrum Scale the maximum number of independent filesets is increased from 1000 to 3000.

NUMA interleave policy defaults to NUMA nodes with CPUs

The numaMemoryInterleave parameter sets the memory interleave policy for GPFS by defining a list of NUMA CPU nodes with memory for the numactl --interleave option. For more information, see mmchconfig command.

Extension of readReplicaPolicy to identify local reads for stretch clusters

You can use the nsdDiskAccessDistance parameter as an extension of the readReplicaPolicy to specify the access distance to disks in a file system failure group. It helps to identify the local site disks or NSD servers for using the preferred replica in stretch clusters. For more information, see mmchconfig command.

New option for viewing configuration details of nsdDiskAccessDistance parameter

The configuration details of the nsdDiskAccessDistance option in the mmchconfig command can be viewed by using the new option nsdDiskAccessConfig of the mmdiag command. For more information, see mmdiag command.

The preferDesignatedMnode parameter configuration enables metanode placement

With IBM Spectrum Scale you can control and specify the placement of metanodes with the new preferDesignatedMnode configuration parameter. For more information, see mmchconfig command.

Selection of TCP connections based on message size

With IBM Spectrum Scale you can dedicate some TCP connections for small messages and the rest for large messages to reduce the contention on TCP connection level. By configuring values for new attributes like tscSmallMsgSizeThreshold and tscSmallMsgTcpConnPct along with the existing maxTcpConnsPerNodeConn attribute, it is possible to dedicate one or more TCP connections for small messages. For more information, see mmchconfig command.

New hint for Fine Grain Write Sharing applications

The performance of non-overlapping small strided writes to a shared file from a parallel application can now be optimized through the gpfsFineGrainWriteSharing_t hint. The optimization can be tuned by using the configuration parameters that are prefixed with dataship and defined in the mmchconfig command. For more information, see gpfs_fcntl() subroutine and mmchconfig command.

New read hint to optimize performance of applications running on multiple nodes

The Fine Grain Read Sharing hint optimizes the performance of applications which run on multiple nodes where tasks issue small strided reads that are less than a full block. For more information, see gpfs_fcntl() subroutine.

New comment field in node definition when creating a cluster

With IBM Spectrum Scale you can add a comment to a node definition when adding or changing nodes in a cluster. For more information , see mmcrcluster command, mmaddnode command, or mmchnode command.

Support for UUID length of 65 characters

IBM Spectrum Scale has increased the maximum key UUID length to 65 characters.

The option verbsRdmaWriteFlush is added to the mmchconfig command to avoid potential data integrity issues

This release added RDMA atomic operations to flush RDMA write operations to avoid potential data integrity issues that have been observed in environments with nodes with multiple HCAs (Host Channel Adapter / RDMA adapter) and a high utilization of the PCI bus. By default, older HCAs not supporting atomic operations are disabled for RDMA traffic. These HCAs should either receive a firmware upgrade to allow for RDMA atomic operations (if available) or be replaced with a more recent model supporting atomic operations. The default behavior can be changed by setting the verbsRdmaWriteFlush to "no" which allows to keep RDMA traffic going with older HCAs. In this case, if there is at least one HCA on a node that does not support atomic operations, the RDMA write flush will be disabled on the node and the cluster is susceptible to potential data integrity issues:

• If at least one HCA on a client node does not support atomic operations, the RDMA write flush will be disabled only for that client node.
• If at least one HCA on an NSD server node does not support atomic operations, the RDMA write flush will be disabled on that server node and as a consequence also for all clients of that server node.

The following check shows on a cluster node whether the HCAs support atomic operations:

# ibv_devinfo -v | egrep "hca_id|atomic"

Support is indicated by the following result:

hca_id: mlx5_0
        atomic_cap:                     ATOMIC_HCA (1)

The following result is shown if the HCA does not support the atomic operations:

hca_id: mlx5_0
        atomic_cap:                     ATOMIC_NONE (0)

• Support for Ubuntu 22.04 on x86_64.
• Support for Red Hat Enterprise Linux® 8.6 on x86_64, PPC64LE, and s390x.
• Modified the command to enable workload prompt to allow administrators to stop and migrate workloads before a node is shut down for upgrade. The spectrumscale upgrade config workloadprompt command replaces the spectrumscale upgrade config workload command.

  For more information, see Upgrading IBM Spectrum Scale components with the installation toolkit and spectrumscale command.

• Support for Ubuntu 22.04 on x86.
• Support for Red Hat Enterprise Linux® 8.6 on x86_64, PPC64LE, and s390x.

• GET /bucket
• GET filesystems/{filesystemName}/directory/snapdir
• POST Filesystems/{filesystemName}/filesets/{filesetName}/cos/downloadStats
• POST Filesystems/{filesystemName}/filesets/{filesetName}/cos/uploadStats
• POST filesystems/{filesystemName}/filePath/{filePath}/diagnostic
• DELETE Filesystems/{filesystemName}/filesets/{filesetName}/cos

• Truststore CA Certificates do not require any manual backups to be done. For more information, see Configuring external authentication for GUI users.
• GUI service once installed can be disabled without restricting access to the REST API service. For more information, see Manually installing IBM Spectrum Scale management GUI.

For compatibility reasons on IBM Spectrum Scale 5.1.0.x and later on Red Hat Enterprise Linux 7.x (7.7 and later), a few Python 2 files are packaged and they might trigger dependency-related messages. In certain scenarios, Python 2.7 might also be required to be installed. Multiple versions of Python can co-exist on the same system. For more information, see the entry about mmadquery in Guidance for Red Hat Enterprise Linux 8.x on IBM Spectrum Scale nodes.

The Python code in IBM Spectrum Scale 5.0.y or earlier continues to be in Python 2.

IBM Spectrum Scale 5.1.x supports Python 3.6 or later. It is recommended that Python 3.6 is installed through the OS package manager (For example, yum install python3). If you install Python 3.6 by other means, then unexpected results might occur, such as failure to install gpfs.base for prerequisite checks, and workarounds might be required.

• Added a predefined threshold rule AFMInQueue_Rule that monitors the AFM gateway in-queue memory usage. For more information, see the Predefined and user-defined thresholds.

• New events added for the following:

  • Disk events
  • File system events
  • GPFS events

New commands

mmfsckx

New structures

There are no new structures.

New subroutines

There are no new subroutines.

New user exits

There are no new user exits.

Changed commands

• mmaddcallback
• mmaddnode
• mmbackup
• mmafmcosconfig
• mmafmcosctl
• mmafmctl
• mmapplypolicy
• mmauth
• mmchconfig
• mmchfileset
• mmchfs
• mmchnode
• mmcrcluster
• mmcrfileset
• mmcrfs
• mmdiag
• mmhealth
• mmrepquota
• mmsdrrestore
• mmlscluster
• mmlsfs
• mmlsquota
• mmshutdown
• mmxcp
• spectrumscale

Changed structures

There are no changed structures.

Changed subroutines

gpfs_fcntl()

Deleted commands

There are no deleted commands.

Deleted structures

There are no deleted structures.

Deleted subroutines

There are no deleted subroutines.

New messages

6027-3412

Changed messages

6027-2050

Deleted messages

6027-1922

List of documentation changes in product guides and respective IBM Documentation sections

The following is a list of documentation changes including changes in topic titles, changes in placement of topics, and deleted topics:

Table 1. List of changes in documentation

Guide	IBM Documentation section	List of changes
Concepts, Planning, and Installation Guide	Upgrading	Removed the topic Upgrading with clustered watch folder enabled
Concepts, Planning, and Installation Guide	Planning	Removed the topic General authentication support matrix .