Troubleshooting
Problem
On RDMA enabled Spectrum Scale clusters with Infiniband or RoCE interconnects, this cluster might show
- an intermittent restart of the Spectrum Scale daemon
- a client node failing to mount a filesystem.
The Spectrum Scale logs show entries like
logAssertFailed: wcOpcode == IBV_WC_SEND || wcOpcode == IBV_WC_RDMA_READ || wcOpcode == IBV_WC_RDMA_WRITE
Cause
The MOFED layer 5.4-2.X can return incompatible/unsupported responses to I/O requests issued by Spectrum Scale. As a part of recovery, the Spectrum Scale daemon might terminate and restart. This issue was introduced in MOFED version 5.4-2.0. Earlier versions of MOFED are not impacted.
Environment
All Spectrum Scale releases are impacted.
Resolving The Problem
While a permanent fix is being worked on, downgrading MOFED to version 5.4-1.X is an interim circumvention of the issue. See below for steps on how to downgrade to MOFED version to 5.4-1.0.3.0.
Downgrading MOFED to version to 5.4-1.0.3.0:
More information regarding MOFED 5.4-1.0.3.0 can be found here: https://docs.nvidia.com/networking/display/MLNXOFEDv541030/NVIDIA+MLNX_OFED+Documentation+Rev+5.4-1.0.3.0
Note: Firmware levels will not change in this process and will remain the same as with 5.4-2
Prerequisites:
- Check MOFED level on each cluster node to see whether the cluster is affected ofed_info -s
- Make sure you have the right Drivers for the installation
-rw-r--r-- 1 root root 366528512 Jul 16 11:57 MLNX_OFED_LINUX-5.4-1.0.3.0-rhel8.2-ppc64le.iso
-rw-r--r-- 1 root root 410331136 Jul 16 11:58 MLNX_OFED_LINUX-5.4-1.0.3.0-rhel8.2-x86_64.iso
Steps for downgrading :
Note: These are high-level instructions on how to downgrade from MOFED 5.4-2 to MOFED 5.4-1. This instruction set assumes the cluster stays online. The examples below are based on the ppc64le architecture. Make the changes where appropriate.
- Log in to the first node requiring MOFED downgrade
- ssh node1
- Check the MOFED version (downgrade required if > 5.4-1.0.3.0)
- ofed_info -s
- Verify quorum availability (if cluster must remain online).
- mmgetstate -s Verifies quorum and help determine whether taking down one or more nodes is safe to keeping the cluster up.
- Shutdown Spectrum Scale (on this node)
- mmshutdown
- Verify Spectrum Scale is shutdown (on this node)
- mmgetstate
- Uninstall the existing MOFED version
- /sbin/ofed_uninstall.sh --force
- Copy MOFED iso and firmware binary the node (example by using ppc64le and sftp)
- sftp node1
- cd /tmp
- mput MLNX_OFED_LINUX-5.4-1.0.3.0-rhel8.2-ppc64le.iso
- Mount MOFED iso
- cd /tmp ; mount -o loop MLNX_OFED_LINUX-5.4-1.0.3.0-rhel8.2-ppc64le.iso /mnt
- Install MOFED (with sample options)
- cd /mnt ; ./mlnxofedinstall --add-kernel-support --disable-kmp --without-fw-update
- Remove MOFED udev rules (if applicable)
- Mellanox OFED v5.x applies custom networking udev rules that might interfere with user-defined versions.
- How to determine whether a modification is needed?
- cat /proc/cmdline
- If you see the following lines => [biosdevname=0 net.ifnames=0] do the following:
- mv -f /lib/udev/rules.d/82-net-setup-link.rules /lib/udev/rules.d/82-net-setup-link.rules.bak
- Re-create initramfs (example for RHEL using dracut)
- dracut -f
- Reboot the node
- systemctl reboot
- Confirm MOFED driver is running
- lbstat
- ofed_info -s
- ibdev2netdev
- Start Spectrum Scale
- mmstartup
- Confirm Spectrum Scale is active and quorum met
- mmgetstate -a
- mmgetstate -s
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB26","label":"Storage"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"STXKQY","label":"IBM Spectrum Scale"},"ARM Category":[{"code":"a8m3p000000PC7yAAG","label":"non-GPFS"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
16 February 2022
UID
ibm16552842