Summary of changes

For information on changes in IBM Spectrum Scale Big Data and Analytics support, see Big Data and Analytics - summary of changes.

For information on changes in IBM Spectrum Scale on AWS 1.3.1, see Summary of changes.

For information on changes in the IBM Spectrum Scale Container Storage Interface driver, see Summary of changes.

For information on changes in the IBM Spectrum Scale Erasure Code Edition, see Summary of changes.

In IBM Spectrum Scale 5.0.5, fileset auditing and skip fileset auditing are available. You can choose up to 20 filesets to apply fileset auditing to, which means that file system activity is only audited if it occurs in the specified filesets. Or, you can choose up to 20 filesets to skip fileset auditing from, which means that all file system events are audited except in the specified filesets. For more information, see Enabling or skipping filesets with file audit logging.

Windows Server 2019 NSD Server and NSD Client are supported

IBM Spectrum Scale provides support for Windows Server 2019 NSD server and NSD Client. For more information, see IBM Spectrum Scale FAQ in IBM® Documentation.

File encryption supports IBM Security Key Lifecycle Manager (SKLM) 4.0

File encryption supports SKLM 4.0 as a Remote Key Management (RKM) server. For more information, see Firewall recommendations for IBM SKLM and Preparation for encryption.

File encryption supports Vormetric Data Security Management (DSM) version 6.3 and 6.4

File encryption supports Vormetric DSM 6.3 and 6.4 as a Remote Key Management server. For more information, see Preparation for encryption.

Thin provisioning improvements are added to space-reclaimable capabilities

The following features are added to thin provisioning:

• Online disk type change for space reclaim - The thin provisioned disk type (thinDiskType) can be enabled or disabled without having to remove and re-add the NSD, starting from this release.
• Auto-detection on space reclaimable disk- Determines whether the disk is space reclaimable for UNMAP, WRITE-SAME, or TRIM or not and also to set up appropriate mechanism for later space reclaim. The auto keyword is used to let GPFS perform auto detection.
• Methods to reclaim space - Based on the device attributes, various appropriate methods to reclaim space can be used, such as TRIM for SSD, UNMAP, and WRITE-SAME or thin-provisioned disks.

For more information, see mmcrnsd command . Similar changes are made to the commands mmlsdisk, mmadddisk, mmrpldisk , mmimportfs, mmchnsd, and mmcrfs.

For more information on thin provisioned devices and TRIM supporting NVMe SSDs, see IBM Spectrum Scale with thin provisioned devices and IBM Spectrum Scale with TRIM-supporting NVMe SSDs.

Subroutines gpfs_getacl_fd() and gpfs_putacl_fd() save and restore ACL information

The subroutines gpfs_getacl_fd() and gpfs_putacl_fd() are intended for use by a backup program to save and restore the ACL information for a file. These subroutines perform the same function as gpfs_getacl() and gpfs_putacl() but they take a file descriptor as an input rather than a file path name. For more information, see gpfs_getacl_fd() subroutine and gpfs_putacl_fd() subroutine.

The cp --preserve=xattr Linux command copies ACL attributes from source to destination file

The cp --preserve=xattr Linux command copies either the POSIX or the NFSv4 ACL extended attributes when an IBM Spectrum Scale file is copied. Also, the following system calls are extended when they are applied to files in IBM Spectrum Scale file systems:

• The listxattr() system call lists the attributes that represent the POSIX or NFSv4 ACL.
• The getxattr() system call retrieves the specified POSIX or NFSv4 ACL attribute. The content of the ACL is retrieved in the system.posix_acl_access attribute or the system.gpfs_nfs4_acl attribute.
• The setxattr() system call writes the content of the specified POSIX or NFSv4 ACL attribute to the corresponding ACL.

In versions of IBM Spectrum Scale earlier than 5.0.5, neither POSIX nor NFSv4 ACLs are supported in this way. However, it is possible to copy the POSIX ACL by issuing the cp --preserve=mode command.

The minIndBlkDescs attribute sets the number of cached indirect block descriptors

Previously the number of cached indirect block descriptors was always the same as the value of maxFilesToCache. With the minIndBlkDescs attribute of the mmchconfig command, you can ensure that more indirect block descriptors are cached in memory. Increasing the value of this attribute can improve the performance of the following types of applications that do direct I/O:

• Applications that do direct I/O on very large files.
• Applications that do direct I/O on files that have an indirection level greater than 0, especially if snapshots are being used, even if the files are not very large.

For more information, see mmchconfig command.

The maxReceiverThreads attribute improves TCP performance

With the maxReceiverThreads attribute of the mmchconfig command, you can increase the number of threads that handle incoming TCP packets up to the number of logical CPUs on the node. Increasing the number of receiver threads improves TCP performance by reducing the number of sockets that each receiver thread must handle. The default value is 16. IBM Spectrum Scale limits the value on each node to the number of logical CPUs on the node. For more information, see mmchconfig command.

The mmchattr --compact option recovers unused space from files

Two suboptions of the mmchattr --compact option increase available disk space by recovering unused indirect blocks or data subblocks from files:

• The --compact indblk option removes redundant indirect blocks from files that were truncated to zero length.
• The --compact fragment option reduces the last logical block of data of the file to the actual number of subblocks that are required. This same action is done automatically whenever a file is closed after being written to, but in rare cases the action fails, leaving a data block with some empty subblocks at the end of a file.

For more information, see mmchattr command.

Renew expired client certificates without recreating a new client

When encryption is configured by using the simplified setup method with SKLM, you can renew an expired client certificate by using mmkeyserv client update command. For more information, see mmkeyserv command.

Planning for pagepool size with Mellanox InfiniBand VERBS RDMA

Configuration of the Mellanox InfiniBand adapter returns an error if the size of the Memory Translation Table (MTT) for the adapter is less than twice the size of the IBM Spectrum Scale pagepool. Set the size of the pagepool small enough to avoid this error. For more information and examples of setting the Mellanox configuration variables log_mtts_per_seg and log_num_mtt, see Configuring Mellanox Memory Translation Table (MTT) for GPFS RDMA VERBS Operation.

Repair functionality of mmfsck command in online mode is deprecated

The repair functionality of mmfsck command in online mode is no longer available. The report-only operation still works in the online mode. For more information, see mmfsck command.

Performance improvements

The following operations are significantly faster:

• Listing files in a directory that contains a large number of fileset junctions. Performance is greatly improved.
• Quota checking by mmcheckquota when it traverses sparse sections of very large inode0 files.
• Processing by mmap() when multiple threads read the same file.

Documentation updates

The following documentation updates were made:

• Information was added to the instructions for renewing key client certificates:
  • For the simplified setup, instructions were added for updating the key client certificate using the mmkeyserv client update command, which was introduced in 5.0.5.0.
  • Corrections and additions were made to the instructions for renewing a client certificate in the regular setup method with SKLM or DSM when the Security Key Lifecycle Manager (SKLM) key server is running with a CA-signed chain of certificates.
  For more information, see Renewing expired client certificates.

• The following changes were made in the instructions for the regular setup method when the SKLM server is using a CA-signed certificate chain:
  • Correction: The endpoint certificate, not the root certificate, is imported into the SKLM server.
  • Addition: Save the files of the certificate chain to a secure location, in case they are needed again.
  For more information, see Regular setup: Using SKLM with a certificate chain.

• In the instructions for the regular setup method with SKLM, reminders were added that with SKLM the simplified setup method is easier to use and more powerful than the regular setup method. For more information, see the following topics:
  • Regular setup: Using SKLM with a self-signed certificate.
  • Regular setup: Using SKLM with a certificate chain.

• In the description of error messages for expired server or client certificates, instructions were added for checking the expiration dates. For more information, see Certificate expiration dates and error messages.

• Support for Red Hat Enterprise Linux® 8.2, 8.3, 8.4, and 8.5 on x86_64, PPC64LE, and s390x
• Support for Red Hat Enterprise Linux 7.8 and 7.9 on x86_64, PPC64, PPC64LE, and s390x
• Support for packages and repository metadata signed with a GPG (GNU Privacy Guard) key. For more information, see Verifying signature of IBM Spectrum Scale packages.
• Enhanced handling of host entries in the /etc/hosts file. Support for both FQDN and short name.

• Added the following endpoints:
  • PUT /filesystems/{filesystemName}/resume
  • PUT /filesystems/{filesystemName}/suspend
  • PUT /filesystems/{filesystemName}/directoryCopy/{sourcePath}
  • PUT /filesystems/{filesystemName}/filesets/{filesetName}/directoryCopy/{sourcePath}
  • PUT /filesystems/{filesystemName}/filesets/{filesetName}/snapshotCopy/{snapshotName}
  • PUT /filesystems/{filesystemName}/filesets/{filesetName}/snapshotCopy/{snapshotName}/path/{sourcePath}
  • PUT /filesystems/{filesystemName}/snapshotCopy/{snapshotName}
  • PUT /filesystems/{filesystemName}/snapshotCopy/{snapshotName}/path/{sourcePath}
  • GET /remotemount/authenticationkey
  • POST /remotemount/authenticationkey
  • PUT /remotemount/authenticationkey
  • GET /remotemount/owningclusters
  • POST /remotemount/owningclusters
  • DELETE /remotemount/owningclusters/{owningCluster}
  • GET /remotemount/owningclusters/{owningCluster}
  • PUT /remotemount/owningclusters/{owningCluster}
  • GET /remotemount/remoteclusters
  • POST /remotemount/remoteclusters
  • DELETE /remotemount/remoteclusters/{remoteCluster}
  • GET /remotemount/remoteclusters/{remoteCluster}
  • PUT /remotemount/remoteclusters/{remoteCluster}
  • POST /remotemount/remoteclusters/{remoteCluster}/access/{owningClusterFilesystem}
  • PUT /remotemount/remoteclusters/{remoteCluster}/access/{owningClusterFilesystem}
  • DELETE /remotemount/remoteclusters/{remoteCluster}/deny/{owningClusterFilesystem}
  • GET /remotemount/remotefilesystems
  • POST /remotemount/remotefilesystems
  • DELETE /remotemount/remotefilesystems/{remoteFilesystem}
  • GET /remotemount/remotefilesystems/{remoteFilesystem}
  • PUT /remotemount/remotefilesystems/{remoteFilesystem}
• Modified the following endpoints:
  • DELETE Filesystems/{filesystemName}/filesets/{filesetName}/link
  • POST Filesystems/{filesystemName}/filesets
  • GET Filesystems/{filesystemName}/filesets
  • PUT Filesystems/{filesystemName}/filesets/{filesetName}
  • PUT smb/shares/shareName
  
• Introduced the support for TLS 1.3.

• Client-specific statistics are supported.
  For the performance analysis of the CES NFS server, it is important to understand the workload that is exerted by the NFS clients. This new feature replaced dependency on customers to understand the client-side workload. It provides the following client-specific statistics:

  1. client_io_ops - This statistic counting is enabled by default. This statistic counting provides information on all I/O-specific operations such as READ, WRITE of all supported NFS protocols (NFSv3, NFSv4.0) exerted by the NFS client (client IP address). These statistics help to understand I/O workload exerted by a specific NFS client.
  2. client_all_ops - This statistics counting provides information on all operations of NFS protocols such as NFSv3, NLM, NFSv4.0. This statistic counting is disabled by default. You must enable it to extract the statistics. These statistics help to understand all workload exerted by the NFS client.

  Use the ganesha_stats command to enable, disable, or retrieve these statistics.
• DNS-related stats can be captured as a part of the authentication statistics collection.
• Various improvements in metadata caching module (MDCACHE), which includes the directory listing area.
• The rpc.statd service fails with "Permission denied". For more information about this error, see NFS error events.

• Added the best practices for using SMB. For more information, see SMB best practices.
• Currency: The SMB service has been upgraded.
• Recovery lock: The SMB clustering component now uses a global file lock to maintain cluster integrity. The recovery lock is placed in the cesroot as /<ces-shared-root>/smb/ctdb-recovery-lock and is held by the ctdb services on one of the CES cluster nodes.
• File handles for Mac clients: Modern Mac OS clients have changed handling FileID=0 or File_ids_off=yes in /etc/nsmb.conf. The SMB service now returns file handles built upon the GPFS inode number, generation number, and snapshot number to Mac clients instead of just 0. Thus the “File_ids_off=yes” setting in /etc/nsmb.conf must be removed.
• Error messages and troubleshooting procedures have been improved for SMB share modes.

New events added

New events have been added to the following sections:

• Authentication
• CES network events
• Call Home events
• GPFS events

For more information, see Events.

New sub-commands added to the mmces command

The mmces interface command shows the state of one or more nodes in the cluster. For more information, see mmces command.

New sub-commands added to the mmhealth command

The mmhealth resolve command manually resolves error events. For more information, see mmhealth command.

Usability improvements to the mmperfmon command

The following improvements have been made to the mmperfmon command:

New sub-commands added to the mmperfmon command

The mmperfmon report top command returns a report of the top processes.

Command updates for mmperfmon config update.

The mmperfmon command can be used to set the filter conditions to manage the amount of metadata or keys within the performance monitoring tool .

Command output updates for the mmperfmon query.

The mmperfmon query command can be used to list metrics by key, for a given node, sensor group and metrics.

For more information, see mmperfmon command.

Usability improvements to system health

The following improvements have been made to system health:

GPFS service monitor extended

The following GPFS services have been changed:

Warning event triggered for memory usage issues

A warning event is triggered if the node encounters an OUT-OF-MEMORY issue. A user can remove this warning event by issuing the mmhealth event resolve out_of_memory command after solving the cause of the memory consumption. For more information, see the out_of_memory event in GPFS events.

A new TIPS event numactl_not_installed is raised when numaMemoryInterleave is enabled but the required /usr/bin/numactl command is missing

When the configuration option numaMemoryInterleave is enabled, the mmfsd daemon is supposed to allocate memory from all NUMA nodes. However, when the numactl tool is missing, mmfsd allocates memory from a single NUMA memory region only. For more information, see the numactl not installed event in GPFS events.

Enabling delay_initial_poll feature

The Performance Monitoring Tool (PMT) supports a Boolean flag, delay_initial_poll, to mitigate the system overload impact due to frequent restart of performance monitoring sensors. For more information, see Configuring the initial sensor data poll delay for long periods.

Automatically deactivate predefined threshold rules

A predefined threshold rule is automatically deactivated if no metric keys for the defined threshold rule exist in the performance monitoring metadata or if the corresponding sensor is not enabled. For more information about predefined threshold rules, see Predefined and user-defined thresholds.

Verifying the state of a rule

The current state of a rule can be verified by issuing the mmhealth thresholds list --verbose command. A rule can be in one of the following states:

• active
• inactive
• unknown

For more information, see Use case 6: Observe the running state of the defined threshold rules in Threshold monitoring use cases.

Migrating the pmcollector

The pmcollector can now be migrated from an old node to a new now. For more information, see Migrating the pmcollector.

Usability improvements to mmprotocoltrace command

The following improvements have been made to the mmprotocoltrace command:

• Reduced the number of commands and options to perform most workloads.
• Cleaned up the command output, hiding irrelevant details and adding details, requested from the field.

For more information, see the mmprotocoltrace command.

New sub-commands added to the mmcallhome command

mmcallhome group addnode/deletenode to add and remove child nodes to or from the existing call home groups.

mmcallhome status diff to see the configuration changes, performed between two time points, where scheduled call home uploads are used as configuration snapshots.

For more information, see Use cases for detecting system changes by using the mmcallhome command.

mmcallhome test connection was extended to enable testing of the connectivity to the IBM infrastructure without the need to create call home groups for non-group members.

Mandatory mmcallhome settings

Providing information for all fields in mmcallhome info is now compulsory for a call home configuration. This ensures that the data is better matched to the customers, thus improving their service experience. If after an upgrade of IBM Spectrum Scale an older configuration does not have enough data to be properly matched to a customer, the mmhealth command shows corresponding TIPS for the CALLHOME component. For more information, see the mmcallhome command.

New commands

There are no new commands.

New structures

There are no new structures.

New subroutines

Two new subroutines are added as follows:

• gpfs_getacl_fd
• gpfs_putacl_fd

New user exits

There are no new user exits.

Changed commands

The following commands are changed:

• mmafmconfig
• mmafmctl
• mmaudit
• mmcallhome
• mmces
• mmchconfig
• mmchfileset
• mmchattr
• mmhealth
• mmkeyserv
• mmchfs
• mmlsquota
• mmmount
• mmnfs
• mmperfmon
• mmprotocoltrace
• mmstartup
• mmwatch
• spectrumscale

Changed structures

There are no changed structures.

Changed subroutines

gpfs_fcntl

Deleted commands

There are no deleted commands.

Deleted structures

There are no deleted structures.

Deleted subroutines

There are no deleted subroutines.

New messages

6027-1761, 6027-2413 and 6027-4210

Changed messages

None.

Deleted messages

None.

Product guides in EPUB format

Product guides in EPUB format are no longer shipped with the documentation.

List of documentation changes in product guides and respective IBM Knowledge Center sections

The following is a list of documentation changes including changes in topic titles, changes in placement of topics, and deleted topics:

Table 4. List of changes in documentation

Guide	Knowledge center section	List of changes
Concepts, Planning, and Installation Guide	Installing	Renamed the following topic: Requirements and limitations for file audit logging to Requirements, limitations, and support for file audit logging
Administration Guide	Administering	Moved the following topics: Exceptions and limitations to NFS V4 ACLs support from under Considerations for GPFS applications to under NFS V4 ACL administration General CES NFS Linux limitations from under Considerations for GPFS applications to under Implementing Cluster Export Services Considerations for GPFS applications to Programming reference Exceptions to Open Group technical standards to under Considerations for GPFS applications in Programming reference Determining if a file system is controlled by GPFS to under Considerations for GPFS applications in Programming reference Considerations for the use of direct I/O (O_DIRECT) to under Considerations for GPFS applications in Programming reference
Problem determination Guide	Monitoring	Removed the following topic: Monitoring the message queue and ZooKeeper status Moved the following topics: Understanding call home from under Monitoring the IBM Spectrum Scale system by using call home to under Product overview. Space requirements for call home data upload from under Monitoring the IBM Spectrum Scale system by using call home to under Planning. Configuring call home to enable manual and automated data upload, Configuring call home using GUI, and Call home configuration examples from under Monitoring the IBM Spectrum Scale system by using call home to under Configuring call home. Displaying the port that the Kafka broker servers are using from under Monitoring file audit logging to under Monitoring the message queue Determining the current topic generation number that is being used in the file system from under Monitoring file audit logging to under Monitoring the message queue
Problem Determination Guide	Troubleshooting	Renamed the following topic: GUI issues to GUI and monitoring issues

Documents migrated from IBM Developer wiki to Knowledge Center

The following are the new topics, sections and guides that have been moved from IBM Developer wiki to the IBM Spectrum Scale Knowledge Center:

IBM Power Tuning Guide

IBM Power Tuning Guide is migrated from the IBM Developer wiki to the IBM Spectrum Scale 5.0.5 Knowledge Center. For more information, see Configuring IBM Power Systems for IBM Spectrum Scale.

SMB best practices

The SMB best practices document is migrated from the IBM Developer wiki to the IBM Spectrum Scale 5.0.5 Knowledge Center. For more information, see SMB best practices.