Start of change

Installing GPUDirect Storage for IBM Spectrum Scale

IBM Spectrum Scale support for GPUDirect Storage (GDS) enables a direct path between GPU memory and storage.

For information about the prerequisites, see Planning for GPUDirect Storage.

Perform the following steps to install GDS:

  1. Install IBM Spectrum Scale with NSD servers for your file system. For more information, see Installing IBM Spectrum Scale on Linux nodes and deploying protocols.
  2. Set up the InfiniBand fabric. For details about the installation instructions, see Mellanox OFED installation. Complete the following steps to configure InfiniBand fabric:
    1. Use the latest MOFED driver. For details about the supported driver versions, see Components required for GDS in IBM Spectrum Scale FAQ in IBM® Documentation.
    2. Set up IP over InfiniBand. For more information, see Configuring InfiniBand and RDMA networks.
    3. If your NSD servers are part of ESS, then reinstall the MOFED driver with the --upstream-libs flag. Use the MOFED version that ships with the ESS.
  3. Install CUDA. For more information about the supported version of CUDA, see Components required for GDS in IBM Spectrum Scale FAQ in IBM Documentation.

    For CUDA installation details, see NVIDIA GPUDirect Storage Installation and Troubleshooting Guide.

  4. Configure IBM Spectrum Scale. For more information, see Configuring the GPFS cluster.
  5. Configure the NVIDIA components. For more information, see Configuring CUDA.
  6. Start IBM Spectrum Scale by using the mmstartup -a command.

Checking the installation

Perform the following steps to check the installation of GDS:

  1. Run the NVIDIA GDS utility gdscheck -p before you run the GDS workloads to verify the environment. You need Python3 installed on the node to run this utility.
  2. Verify the status of PCIe Access Control Services (ACS) and PCIe Input/Output Memory Management Unit (IOMMU), as these components affect GDS function and performance. The output of the gdscheck -p utility must display the following status for IOMMU and ACS components: : 
    IOMMU disabled
ACS disabled
  3. Check for IBM Spectrum Scale support in the output of gdscheck -p as shown in the following example: 
    # gdscheck -p | grep "Spectrum Scale"

 IBM Spectrum Scale : Supported
  4. If the GPFS log (mmfs log) contains the following information after you start IBM Spectrum Scale, it indicates that the GDS support is successfully enabled: 
    [I] VERBS DC API loaded.
    and 
    
[I] VERBS DC API initialized.
End of change