Summary of changes
This topic summarizes changes to the IBM Spectrum
Scale licensed program and the IBM Spectrum
Scale library. Within each topic, these
markers (
) surrounding
text or illustrations indicate technical changes or additions that are made to the previous edition
of the information.
![Start of change](./delta.gif)
for IBM Spectrum Scale version 5.0.5
as updated, April 2022
- Feature updates
- Documented commands, structures, and subroutine
- Messages
- Deprecated items
- Changes in documentation
- Documents migrated from IBM Developer wiki to Knowledge Center
- AFM and AFM DR-related changes
-
- Migration enhancements for AFM data migration. For more information, see Data migration by using AFM migration enhancements.
- Support of fast create for AFM and AFM-DR. The parameter afmFastCreate must be enabled for the fast create feature. For more information, see Fast create.
Support for disabling afmRPO attribute when using with mmchfileset command. For more information, see mmchfileset command.
- Big data and analytics changes
-
For information on changes in IBM Spectrum Scale Big Data and Analytics support, see Big Data and Analytics - summary of changes.
- IBM Spectrum Scale on AWS changes
-
For information on changes in IBM Spectrum Scale on AWS 1.3.1, see Summary of changes.
- IBM Spectrum Scale Container Storage Interface driver changes
-
For information on changes in the IBM Spectrum Scale Container Storage Interface driver, see Summary of changes.
- IBM Spectrum Scale Erasure Code Edition changes
-
For information on changes in the IBM Spectrum Scale Erasure Code Edition, see Summary of changes.
- File audit logging changes
-
In IBM Spectrum Scale 5.0.5, fileset auditing and skip fileset auditing are available. You can choose up to 20 filesets to apply fileset auditing to, which means that file system activity is only audited if it occurs in the specified filesets. Or, you can choose up to 20 filesets to skip fileset auditing from, which means that all file system events are audited except in the specified filesets. For more information, see Enabling or skipping filesets with file audit logging.
- File system core improvements
-
- Windows Server 2019 NSD Server and NSD Client are supported
- IBM Spectrum
Scale provides support for Windows Server 2019 NSD server and NSD Client. For more
information, see IBM Spectrum
Scale FAQ in IBM® Documentation.
- File encryption supports IBM Security Key Lifecycle Manager (SKLM) 4.0
- File encryption supports SKLM 4.0 as a Remote Key Management (RKM) server. For more information,
see Firewall recommendations for IBM SKLM and Preparation for encryption.
- File encryption supports Vormetric Data Security Management (DSM) version 6.3 and 6.4
- File encryption supports Vormetric DSM 6.3 and 6.4 as a Remote Key Management server. For more information, see Preparation for encryption.
- Thin provisioning improvements are added to space-reclaimable capabilities
- The following features are added to thin provisioning:
- Online disk type change for space reclaim - The thin provisioned disk type (thinDiskType) can be enabled or disabled without having to remove and re-add the NSD, starting from this release.
- Auto-detection on space reclaimable disk- Determines whether the disk is space reclaimable for UNMAP, WRITE-SAME, or TRIM or not and also to set up appropriate mechanism for later space reclaim. The auto keyword is used to let GPFS perform auto detection.
- Methods to reclaim space - Based on the device attributes, various appropriate methods to reclaim space can be used, such as TRIM for SSD, UNMAP, and WRITE-SAME or thin-provisioned disks.
For more information on thin provisioned devices and TRIM supporting NVMe SSDs, see IBM Spectrum Scale with thin provisioned devices and IBM Spectrum Scale with TRIM-supporting NVMe SSDs.
- Subroutines gpfs_getacl_fd() and gpfs_putacl_fd() save and restore ACL information
- The subroutines gpfs_getacl_fd() and gpfs_putacl_fd() are
intended for use by a backup program to save and restore the ACL information for a file. These
subroutines perform the same function as gpfs_getacl() and
gpfs_putacl() but they take a file descriptor as an input rather than a file path
name. For more information, see gpfs_getacl_fd() subroutine
and gpfs_putacl_fd() subroutine.
- The cp --preserve=xattr Linux command copies ACL attributes from source to destination file
- The cp --preserve=xattr Linux command copies either the POSIX or the NFSv4
ACL extended attributes when an IBM Spectrum
Scale file is
copied. Also, the following system calls are extended when they are applied to files in IBM Spectrum
Scale file systems:
- The listxattr() system call lists the attributes that represent the POSIX or NFSv4 ACL.
- The getxattr() system call retrieves the specified POSIX or NFSv4 ACL attribute. The content of the ACL is retrieved in the system.posix_acl_access attribute or the system.gpfs_nfs4_acl attribute.
- The setxattr() system call writes the content of the specified POSIX or NFSv4 ACL attribute to the corresponding ACL.
- The minIndBlkDescs attribute sets the number of cached indirect block descriptors
- Previously the number of cached indirect block descriptors was always the same as the value of
maxFilesToCache. With the minIndBlkDescs attribute of the
mmchconfig command, you can ensure that more indirect block descriptors are
cached in memory. Increasing the value of this attribute can improve the performance of the
following types of applications that do direct I/O:
- Applications that do direct I/O on very large files.
- Applications that do direct I/O on files that have an indirection level greater than 0, especially if snapshots are being used, even if the files are not very large.
- The maxReceiverThreads attribute improves TCP performance
- With the maxReceiverThreads attribute of the mmchconfig
command, you can increase the number of threads that handle incoming TCP packets up to the number of
logical CPUs on the node. Increasing the number of receiver threads improves TCP performance by
reducing the number of sockets that each receiver thread must handle. The default value is 16.
IBM
Spectrum Scale limits the value on each node to the
number of logical CPUs on the node. For more information, see mmchconfig command.
- The mmchattr --compact option recovers unused space from files
- Two suboptions of the mmchattr --compact option increase available disk space
by recovering unused indirect blocks or data subblocks from files:
- The --compact indblk option removes redundant indirect blocks from files that were truncated to zero length.
- The --compact fragment option reduces the last logical block of data of the file to the actual number of subblocks that are required. This same action is done automatically whenever a file is closed after being written to, but in rare cases the action fails, leaving a data block with some empty subblocks at the end of a file.
- Renew expired client certificates without recreating a new client
- When encryption is configured by using the simplified setup method with SKLM, you can renew an expired client certificate by using mmkeyserv client update command. For more information, see mmkeyserv command.
- Planning for pagepool size with Mellanox InfiniBand VERBS RDMA
- Configuration of the Mellanox InfiniBand adapter returns an error if the size of the Memory
Translation Table (MTT) for the adapter is less than twice the size of the IBM
Spectrum Scale pagepool. Set the size of the pagepool small
enough to avoid this error. For more information and examples of setting the Mellanox configuration
variables log_mtts_per_seg and log_num_mtt, see Configuring Mellanox Memory Translation Table (MTT) for GPFS RDMA VERBS Operation.
- Repair functionality of mmfsck command in online mode is deprecated
- The repair functionality of mmfsck command in online mode is no longer available. The report-only operation still works in the online mode. For more information, see mmfsck command.
- Performance improvements
- The following operations are significantly faster:
- Listing files in a directory that contains a large number of fileset junctions. Performance is greatly improved.
- Quota checking by mmcheckquota when it traverses sparse sections of very large inode0 files.
- Processing by mmap() when multiple threads read the same file.
- Documentation updates
- The following documentation updates were made:
- Information was added to the instructions for renewing key client certificates:
- For the simplified setup, instructions were added for updating the key client certificate using the mmkeyserv client update command, which was introduced in 5.0.5.0.
- Corrections and additions were made to the instructions for renewing a client certificate in the regular setup method with SKLM or DSM when the Security Key Lifecycle Manager (SKLM) key server is running with a CA-signed chain of certificates.
- The following changes were made in the instructions for the regular setup method when the SKLM
server is using a CA-signed certificate chain:
- Correction: The endpoint certificate, not the root certificate, is imported into the SKLM server.
- Addition: Save the files of the certificate chain to a secure location, in case they are needed again.
- In the instructions for the regular setup method with SKLM, reminders were added that with SKLM
the simplified setup method is easier to use and more powerful than the regular setup method. For
more information, see the following topics:
- Regular setup: Using SKLM with a self-signed certificate.
- Regular setup: Using SKLM with a certificate chain.
- In the description of error messages for expired server or client certificates, instructions
were added for checking the expiration dates. For more information, see
Certificate expiration dates and error messages.
- Information was added to the instructions for renewing key client certificates:
- Installation toolkit changes
-
Support for Red Hat Enterprise Linux® 8.2, 8.3, 8.4, and 8.5 on x86_64, PPC64LE, and s390x
- Support for Red Hat Enterprise Linux 7.8 and 7.9 on x86_64, PPC64, PPC64LE, and s390x
- Support for packages and repository metadata signed with a GPG (GNU Privacy Guard) key. For more information, see Verifying signature of IBM Spectrum Scale packages.
- Enhanced handling of host entries in the /etc/hosts file. Support for both FQDN and short name.
- Management API changes
-
- Added the following endpoints:
- PUT /filesystems/{filesystemName}/resume
- PUT /filesystems/{filesystemName}/suspend
- PUT /filesystems/{filesystemName}/directoryCopy/{sourcePath}
- PUT /filesystems/{filesystemName}/filesets/{filesetName}/directoryCopy/{sourcePath}
- PUT /filesystems/{filesystemName}/filesets/{filesetName}/snapshotCopy/{snapshotName}
- PUT /filesystems/{filesystemName}/filesets/{filesetName}/snapshotCopy/{snapshotName}/path/{sourcePath}
- PUT /filesystems/{filesystemName}/snapshotCopy/{snapshotName}
- PUT /filesystems/{filesystemName}/snapshotCopy/{snapshotName}/path/{sourcePath}
- GET /remotemount/authenticationkey
- POST /remotemount/authenticationkey
- PUT /remotemount/authenticationkey
- GET /remotemount/owningclusters
- POST /remotemount/owningclusters
- DELETE /remotemount/owningclusters/{owningCluster}
- GET /remotemount/owningclusters/{owningCluster}
- PUT /remotemount/owningclusters/{owningCluster}
- GET /remotemount/remoteclusters
- POST /remotemount/remoteclusters
- DELETE /remotemount/remoteclusters/{remoteCluster}
- GET /remotemount/remoteclusters/{remoteCluster}
- PUT /remotemount/remoteclusters/{remoteCluster}
- POST /remotemount/remoteclusters/{remoteCluster}/access/{owningClusterFilesystem}
- PUT /remotemount/remoteclusters/{remoteCluster}/access/{owningClusterFilesystem}
- DELETE /remotemount/remoteclusters/{remoteCluster}/deny/{owningClusterFilesystem}
- GET /remotemount/remotefilesystems
- POST /remotemount/remotefilesystems
- DELETE /remotemount/remotefilesystems/{remoteFilesystem}
- GET /remotemount/remotefilesystems/{remoteFilesystem}
- PUT /remotemount/remotefilesystems/{remoteFilesystem}
Modified the following endpoints:
- DELETE Filesystems/{filesystemName}/filesets/{filesetName}/link
- POST Filesystems/{filesystemName}/filesets
- GET Filesystems/{filesystemName}/filesets
- PUT Filesystems/{filesystemName}/filesets/{filesetName}
PUT smb/shares/shareName
- Introduced the support for TLS 1.3.
- Added the following endpoints:
- Management GUI changes
- The following changes are made to the GUI:
- NFS changes
-
- Client-specific statistics are supported.For the performance analysis of the CES NFS server, it is important to understand the workload that is exerted by the NFS clients. This new feature replaced dependency on customers to understand the client-side workload. It provides the following client-specific statistics:
- client_io_ops - This statistic counting is enabled by default. This statistic counting provides information on all I/O-specific operations such as READ, WRITE of all supported NFS protocols (NFSv3, NFSv4.0) exerted by the NFS client (client IP address). These statistics help to understand I/O workload exerted by a specific NFS client.
- client_all_ops - This statistics counting provides information on all operations of NFS protocols such as NFSv3, NLM, NFSv4.0. This statistic counting is disabled by default. You must enable it to extract the statistics. These statistics help to understand all workload exerted by the NFS client.
- DNS-related stats can be captured as a part of the authentication statistics collection.
- Various improvements in metadata caching module (MDCACHE), which includes the directory listing area.
The rpc.statd service fails with "Permission denied". For more information about this error, see NFS error events.
- Client-specific statistics are supported.
- SMB changes
-
- Added the best practices for using SMB. For more information, see SMB best practices.
- Currency: The SMB service has been upgraded.
- Recovery lock: The SMB clustering component now uses a global file lock to maintain
cluster integrity. The recovery lock is placed in the
cesroot
as /<ces-shared-root>/smb/ctdb-recovery-lock and is held by thectdb
services on one of the CES cluster nodes. - File handles for Mac clients: Modern Mac OS clients have changed handling FileID=0 or File_ids_off=yes in /etc/nsmb.conf. The SMB service now returns file handles built upon the GPFS inode number, generation number, and snapshot number to Mac clients instead of just 0. Thus the “File_ids_off=yes” setting in /etc/nsmb.conf must be removed.
- Error messages and troubleshooting procedures have been improved for SMB share modes.
- System Health changes
-
- New events added
- New events have been added to the following sections:
- Authentication
- CES network events
- Call Home events
- GPFS events
- New sub-commands added to the mmces command
- The mmces interface command shows the state of one or more nodes in the cluster. For more information, see mmces command.
- New sub-commands added to the mmhealth command
- The mmhealth resolve command manually resolves error events. For more information, see mmhealth command.
- Usability improvements to the mmperfmon command
- The following improvements have been made to the mmperfmon command:
- New sub-commands added to the mmperfmon command
- The mmperfmon report top command returns a report of the top processes.
- Command updates for mmperfmon config update.
- The mmperfmon command can be used to set the filter conditions to manage the amount of metadata or keys within the performance monitoring tool .
- Command output updates for the mmperfmon query.
- The mmperfmon query command can be used to list metrics by key, for a given node, sensor group and metrics.
For more information, see mmperfmon command.
- Usability improvements to system health
- The following improvements have been made to system health:
- GPFS service monitor extended
- The following GPFS services have been changed:
- Warning event triggered for memory usage issues
- A warning event is triggered if the node encounters an
OUT-OF-MEMORY
issue. A user can remove this warning event by issuing the mmhealth event resolve out_of_memory command after solving the cause of the memory consumption. For more information, see theout_of_memory
event in GPFS events. - A new TIPS event
numactl_not_installed
is raised whennumaMemoryInterleave
is enabled but the required /usr/bin/numactl command is missing - When the configuration option
numaMemoryInterleave
is enabled, themmfsd
daemon is supposed to allocate memory from all NUMA nodes. However, when the numactl tool is missing,mmfsd
allocates memory from a single NUMA memory region only. For more information, see thenumactl not installed
event in GPFS events.
Enabling delay_initial_poll feature
The Performance Monitoring Tool (PMT) supports a Boolean flag,
delay_initial_poll
, to mitigate the system overload impact due to frequent restart of performance monitoring sensors. For more information, see Configuring the initial sensor data poll delay for long periods.- Automatically deactivate predefined threshold rules
- A predefined threshold rule is automatically deactivated if no metric keys for the defined threshold rule exist in the performance monitoring metadata or if the corresponding sensor is not enabled. For more information about predefined threshold rules, see Predefined and user-defined thresholds.
- Verifying the state of a rule
- The current state of a rule can be verified by issuing the mmhealth thresholds list
--verbose command. A rule can be in one of the following states:
- active
- inactive
- unknown
Migrating the pmcollector
The pmcollector can now be migrated from an old node to a new now. For more information, see Migrating the pmcollector.
- Usability improvements to mmprotocoltrace command
- The following improvements have been made to the mmprotocoltrace command:
- Reduced the number of commands and options to perform most workloads.
- Cleaned up the command output, hiding irrelevant details and adding details, requested from the field.
- Call Home changes
- Call home has the following improvements:
- New sub-commands added to the mmcallhome command
-
mmcallhome group addnode/deletenode to add and remove child nodes to or from the existing call home groups.
- Mandatory mmcallhome settings
-
Providing information for all fields in mmcallhome info is now compulsory for a call home configuration. This ensures that the data is better matched to the customers, thus improving their service experience. If after an upgrade of IBM Spectrum Scale an older configuration does not have enough data to be properly matched to a customer, the mmhealth command shows corresponding TIPS for the
CALLHOME
component. For more information, see the mmcallhome command.
- Security-Enhanced Linux (SELinux) enforcing mode is supported
- IBM Spectrum
Scale now runs on supported Red Hat Enterprise Linux operating systems with
Security-Enhanced Linux (SELinux) enforcing mode and targeted policies. For more
information, see Security-Enhanced Linux support.
- Documented commands, structures, and subroutines
- The following section lists the modifications to the documented commands, structures, and subroutines:
- New commands
- There are no new commands.
- New structures
- There are no new structures.
- New subroutines
- Two new subroutines are added as follows:
- gpfs_getacl_fd
- gpfs_putacl_fd
- New user exits
- There are no new user exits.
- Changed commands
- The following commands are changed:
- mmafmconfig
- mmafmctl
- mmaudit
- mmcallhome
- mmces
- mmchconfig
mmchfileset
- mmchattr
- mmhealth
- mmkeyserv
mmchfs
mmlsquota
- mmmount
- mmnfs
- mmperfmon
- mmprotocoltrace
- mmstartup
- mmwatch
- spectrumscale
- Changed structures
- There are no changed structures.
- Changed subroutines
gpfs_fcntl
- Deleted commands
- There are no deleted commands.
- Deleted structures
- There are no deleted structures.
- Deleted subroutines
- There are no deleted subroutines.
- Messages
- The following are the new, changed, and deleted messages:
- New messages
- 6027-1761, 6027-2413 and 6027-4210
- Changed messages
- None.
- Deleted messages
- None.
- List of stabilized, deprecated, and discontinued features in 5.0.5
-
A feature is a stabilized feature if there is no plan to deprecate or remove this capability in a subsequent release of the product. It will remain supported and current with updates to the operating systems. You do not need to change any of your existing applications and scripts that use a stabilized function now. You should not expect significant new functionality or enhancements to these features.
Table 1. Features stabilized in Version 5.0.5 Category Stabilized Functionality Recommended Action Transparent Cloud Tiering (TCT) All TCT can continue to be used for existing purposes. There are no plans to extend its purpose to more use cases. FPO All FPO and SNC remain available. However, it is recommend to limit the size of deployments to 32 nodes. There are no plans for significant new functionality in FPO nor increases in scalability. The strategic direction for storage using internal drives and storage rich servers is Spectrum Scale Erasure Code Edition (ECE).
cNFS All IBM’s strategic path is to invest in User Space solutions for NFS support of Scale workloads. Once User Space performance and function are considered to be sufficient to replace cNFS, anticipate that the support for cNFS will be deprecated. HDFS Transparency HDP support Plan to migrate to Cloudera supported releases. A feature is a deprecated feature if that specific feature is supported in the current release but the support might be removed in a future release. In some cases, it might be advisable to plan to discontinue the use of deprecated functionality.Table 2. Features deprecated in Version 5.0.5 Category Deprecated functionality Recommended Action Security The use of TLS 1.0 and 1.1 for authorization within and between the IBM Spectrum Scale clusters (Security mode). Upgrade to TLS 1.2 or later. Set the nistCompliance parameter to SP800-131A to ensure that only TLS 1.2 is used. Support for Vormetric DSM V5 with IBM Spectrum Scale encryption. Upgrade to Vormetric DSM V6.2 or later. Note: The plan is to discontinue support for Vormetric DSM V5 in the next release.GUI/REST API The use of TLS 1.0 and 1.1 for authorization with the GUI/REST API server Upgrade to TLS 1.2 or later Cloud Storage Enabler for Containers (SEC) Adopt the CSI driver. Note that the plan is to discontinue support for SEC in the next release. Platforms Encryption acceleration library for Power7 (CliC) No functional impact: performance only on Power7. If encryption performance is critical, plan to migrate to newer generations of Power systems. The plan is to discontinue support for CLiC by the end of 2020.
Platforms Support for Power7 Plan to migrate to newer generations of Power systems. iSCSI target Use of IBM Spectrum® Scale as an iSCSI target for remote boot of servers. Plan to replace the use of Scale as a target through iSCSI with other block storage providers. The watch API and tswf sample program The watch API and sample program was for creating a single node watch using the API. Currently, provides more resilient and fully integrated cluster watch with the mmwatch command. Plan to use the improved mmwatch command to start clustered watches. Kafka IBM Spectrum Scale will no longer support gpfs.kafka on IBM Spectrum Scale clusters. This means that there will be no concept of brokers or zookeepers. Although, gpfs.librdkafka is still available. No actions needed. IBM Spectrum Scale will provide a single command to do this conversion at the time it is removed. Message Queue The message queue will no longer be needed since Kafka will be removed. The mmmsgqueue command will be removed entirely and no longer needed to run mmwatch or mmaudit commands. No actions needed. IBM Spectrum Scale will provide a single command to do this conversion at the time it is removed. Audit fileset residing on separate file system IBM Spectrum Scale will no longer support creating the audit fileset on a filesystem that is not the one being audited. This means that the audit fileset must be belonged to the audited file system. Reconfigure audit with the mmaudit command to change this configuration. mmaudit Device enable [--log-fileset FilesetName]
A feature is a Discontinued feature if it has been removed in a release and is no longer available. You need to make changes if you were using that functionality in previous releases.Table 3. Features discontinued in Version 5.0.5 Category Discontinued functionality Recommended Action HDFS Transparency Support for IBM BigInsights for Apache Hadoop IBM BigInsights for Apache Hadoop is no longer supported by IBM. Consequently, IBM Spectrum Scale no longer supports the use of BigInsights as a Hadoop connection. Platforms Windows 7 and Windows Server 2008 R2 Upgrade to a supported version of Windows.
- Changes in documentation
-
- Product guides in EPUB format
- Product guides in EPUB format are no longer shipped with the documentation.
- List of documentation changes in product guides and respective IBM Knowledge Center sections
-
The following is a list of documentation changes including changes in topic titles, changes in placement of topics, and deleted topics:
Table 4. List of changes in documentation Guide Knowledge center section List of changes Concepts, Planning, and Installation Guide Installing Renamed the following topic: - Requirements and limitations for file audit logging to Requirements, limitations, and support for file audit logging
Administration Guide Administering Moved the following topics: - Exceptions and limitations to NFS V4 ACLs support from under Considerations for GPFS applications to under NFS V4 ACL administration
- General CES NFS Linux limitations from under Considerations for GPFS applications to under Implementing Cluster Export Services
- Considerations for GPFS applications to Programming reference
- Exceptions to Open Group technical standards to under Considerations for GPFS applications in Programming reference
- Determining if a file system is controlled by GPFS to under Considerations for GPFS applications in Programming reference
- Considerations for the use of direct I/O (O_DIRECT) to under Considerations for GPFS applications in Programming reference
Problem determination Guide Monitoring Removed the following topic: - Monitoring the message queue and ZooKeeper status
- Understanding call home from under Monitoring the IBM Spectrum Scale system by using call home to under Product overview.
- Space requirements for call home data upload from under Monitoring the IBM Spectrum Scale system by using call home to under Planning.
- Configuring call home to enable manual and automated data upload, Configuring call home using GUI, and Call home configuration examples from under Monitoring the IBM Spectrum Scale system by using call home to under Configuring call home.
- Displaying the port that the Kafka broker servers are using from under Monitoring file audit logging to under Monitoring the message queue
- Determining the current topic generation number that is being used in the file system from under Monitoring file audit logging to under Monitoring the message queue
Problem Determination Guide Troubleshooting Renamed the following topic: - GUI issues to GUI and monitoring issues
- Documents migrated from IBM Developer wiki to Knowledge Center
- The following are the new topics, sections and guides that have been moved from IBM Developer
wiki to the IBM Spectrum
Scale Knowledge Center:
- IBM Power Tuning Guide
- IBM Power Tuning Guide is migrated from the IBM Developer wiki to the IBM Spectrum Scale 5.0.5 Knowledge Center. For more information, see Configuring IBM Power Systems for IBM Spectrum Scale.
- SMB best practices
- The SMB best practices document is migrated from the IBM Developer wiki to the IBM Spectrum Scale 5.0.5 Knowledge Center. For more information, see SMB best practices.
![End of change](./deltaend.gif)