Troubleshooting
Problem
IBM has identified an issue in IBM Spectrum Scale where a Spectrum Scale fix is required to run on the latest Intel x86_64 hardware (Ice Lake) with 5-level paging.
This issue affects all IBM Spectrum Scale versions running on any newer processors with 5-level paging.
Symptom
To verify if your system is supporting 5-level paging, run the following command:
cat /proc/cpuinfo
If the "flags" contain the flag "la57", 5-level paging support is active in the kernel. This is also indicated by the "address sizes" of "46 bits physical, 57 bits virtual".
Systems that attempt to install Spectrum Scale on a newer Intel x86_64 processor with 5-level page may encounter an assert that looks like the following:
The assert:
[X] logAssertFailed: (UIntPtr)(vmallocStart) >= 0xFFFF880000000000 [X] return code 0, reason code 0, log record tag 0 /usr/lpp/mmfs/bin/runmmfs: line 560: 33380: Memory fault(coredump)
dmesg/journal:
mmfs[32751]: /usr/lpp/mmfs/bin/tsctl: Error=MMFS_GENERIC, ID=0x30D9195E, Tag=0 mmfs[32751]: Generic error in /project/sprelmax510/build/rmax510s002a/src/avs/fs/mmfs/ts/tasking/linux/mastslav.C line 173 retCode 0, reasonCode 0
The assert:
[X] logAssertFailed: (UIntPtr)(vmallocStart) >= 0xFFFF880000000000 [X] return code 0, reason code 0, log record tag 0 /usr/lpp/mmfs/bin/runmmfs: line 560: 33380: Memory fault(coredump)
dmesg/journal:
mmfs[32751]: /usr/lpp/mmfs/bin/tsctl: Error=MMFS_GENERIC, ID=0x30D9195E, Tag=0 mmfs[32751]: Generic error in /project/sprelmax510/build/rmax510s002a/src/avs/fs/mmfs/ts/tasking/linux/mastslav.C line 173 retCode 0, reasonCode 0
Cause
With the introduction of the 5-level page tables, supported by Intel’s Ice Lake processor generation, user space memory gets expanded by a factor of 512. This resulted in the change in the kernel base address.
Diagnosing The Problem
Installing Spectrum Scale on new Intel x86_64 5-level paging systems.
Refer to "Symptom" section above.
Resolving The Problem
Workaround:
Disable 5-level page setting by adding no5lvl to the kernel command line and then rebooting the node. Check the documentation of the Linux distribution used for details on how to apply this change.
For example, on RHEL8:
# grubby --update-kernel=ALL --args="no5lvl"
Disable 5-level page setting by adding no5lvl to the kernel command line and then rebooting the node. Check the documentation of the Linux distribution used for details on how to apply this change.
For example, on RHEL8:
# grubby --update-kernel=ALL --args="no5lvl"
# cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-240.10.1.el8_3.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet net.ifnames=0 biosdevname=0 no5lvl
Recommendation:
Users running IBM Spectrum Scale V5.1.0 through V5.1.1.0, should apply IBM Spectrum Scale V5.1.1.1 or later, available from fix central at TBD:
If you cannot apply one of the above PTF levels, contact IBM Service to obtain and apply an efix for your level of code:
For IBM Spectrum Scale 5.0.5.x, reference APAR IJ32626
For IBM Spectrum Scale 5.1.0 through 5.1.1.0, reference APAR IJ32608
It is strongly recommended to not install Spectrum Scale on 5-level paging systems until the official fixes are in the corresponding 5.1.1.x and 5.0.5.x release streams.
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB26","label":"Storage"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"STXKQY","label":"IBM Spectrum Scale"},"ARM Category":[{"code":"a8m0z0000000AsiAAE","label":"Kernel Module"},{"code":"a8m0z0000000AsiAAE","label":"Kernel Module"},{"code":"a8m0z0000000AsiAAE","label":"Kernel Module"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"5.0.0;5.1.0;5.1.1"}]
Was this topic helpful?
Document Information
Modified date:
11 June 2021
UID
ibm16450385