IBM Spectrum Scale: GPUDirect Storage (GDS) is available as a Technical Preview feature in Spectrum Scale 5.1.1.

Content

• The storage must not be locally attached to the GDS clients as GDS requires the NSD path for storage access. ESS and non-ESS storage servers are supported. The storage servers may be in the same or different cluster as the GDS clients.
• GDS requires Mellanox RDMA over InfiniBand between GDS clients and storage servers (RoCE is not supported).
• Hardware:
  • GDS clients: x86 with a GPU model that supports GDS (refer to the NVIDIA GDS documentation)
  • Network: EDR or HDR InfiniBand
  • InfiniBand adapter: Mellanox CX4, CX5 or CX6 (CX4 firmware must be 12.27.4000 or higher)
• Software:
  • MOFED:
    • GDS clients: MOFED 5.2.1.0.4.0
    • Storage Servers:
      • ESS storage servers: reinstall MOFED adding “--upstream-libs” (use MOFED version that ships with the ESS)
        • "ofed_uninstall.sh --force"
        • mount MOFED iso found in "/install/ess/sync/rhels8/x86_64/mofed" on IO servers
        • "mlnxofedinstall --add-kernel-support --disable-kmp --without-fw-update --upstream-libs"
      • non-ESS storage servers:  install MOFED 5.2.1.0.4.0
  • Spectrum Scale 5.1.1 PTF 1 (GDS clients and storage servers)
  • Nvidia (GDS clients)
    • CUDA:  11.0
    • GDS driver: refer to the link - partners.nvidia.com
  • OS (GDS clients): RHEL 8.3, Ubuntu 20.04
• Before starting the Spectrum Scale daemons, set the verbs configuration as follows using "mmchconfig" on the GDS clients and storage servers:
  • verbsRdma enable 
  • verbsRdmaSend yes
  • verbsPorts (see "Additional Considerations" section below)
  • verbsRdmaCm disable
• Before building the Spectrum Scale Linux portability layer on the GDS clients, remove the comment around "#define GPU_DIRECT_STORAGE" in"/usr/lpp/mmfs/src/gpl-linux/verdep.h"

• Set "mmchconfig IgnoreNonDioInstCount=yes" before starting the Spectrum Scale daemons.

• In the NVIDIA GDS config file cufile.json on the GDS clients, set allow_compat_mode to false.

• Spectrum Scale does not support GDS write (cuFileWrite).
• Spectrum Scale does not support GDS on files less than 4096 bytes in length.
• Spectrum Scale does not support GDS on sparse files or files with pre-allocated storage (for example, fallocate(), gpfs_prealloc(), etc)
• Spectrum Scale does not support GDS on files that are encrypted.
• Spectrum Scale does not support GDS on memory-mapped files.
• Spectrum Scale does not support GDS on files that are compressed or marked for deferred compression (for more information on compression, refer to www.ibm.com/docs/en/spectrum-scale/5.1.0?topic=reference-file-compression)
• Spectrum Scale does not support GDS if the mmchconfig option "disableDIO" is set to "true" (the default value of "disableDIO" is "false")

• Spectrum Scale does not support GDS on files that use data tiering, include transparent cloud tiering. 
• Spectrum Scale does not support the NVIDIA GDS asynchronous "poll" mode. The NVIDIA GDS lib implicitly converts a poll mode request on a file in a Spectrum Scale mount to a synchronous GDS IO request.
• Spectrum Scale does not support GDS on files in snapshots or clones. If a GDS read is issued on a file in a snapshot or clone, -EIO is returned to the user application.

• Reading a file with GDS read concurrent with a buffered read will not deliver GDS performance for the GDS thread. This limitation holds whether the concurrent threads are part of the same or different user application. In this context, "buffered" is defined as non-GDS, non-DirectIO.
• The NVIDIA GDS utility "gdscheck -p" should be run prior to GDS workloads to verify the environment. Particular attention should be given to ACS and the IOMMU, as these affect GDS function and performance. Refer to the NVIDIA GDS documentation for more information.
• GDS clients and the storage servers that serve the target GDS filesystems must have Spectrum Scale 5.1.1 PTF 1 installed.
• GDS clients and storage servers must be configured to use one InfiniBand (IB) network for verbs and RDMA. For clusters with two IB networks, adjust the mmchconfig option "verbPorts" to restrict verbs and RDMA to one of the two IB networks. Note this requirement doesn't limit the number of IB links (connected to the one IB network) at the storage servers and GDS clients.
• The IP over IB addresses specified in the NVIDIA GDS config file cufile.json must be consistent with the setting of verbsPorts on the GDS clients. Refer to the NVIDIA GDS documentation for more information on the NVIDIA config file.

• Tuning GDS performance is dependent on the workload and machine architecture of the GDS client. It is generally advised to contact IBM for tuning assistance with the 5.1.1 GDS tech preview, but when using DGX-A100 machines as GDS clients, the following two settings in the NVIDIA GDS config file cufile.json are typically sufficient to achieve good performance when using the IB NICs attached directly to the CPU complex, sometimes referred to as the "storage" NICs:

  • "rdma_load_balancing_policy": "RoundRobin"

  • "rdma_access_mask": "0x1f"

• Running a GDS workload concurrent with a non-GDS workload on the same client is not recommended. Mixed workloads on a client can eliminate the GDS performance benefit, and in rare cases, cause a kernel crash in GDS driver de-registration.