Troubleshooting
Problem
Sometimes, when you upgrade the GPFS / IBM Spectrum Scale pattern type, the cluster may hang indefinitely awaiting active state of GPFS / IBM Spectrum Scale.
When you upgrade only the Kernel version and not other Kernel packages, the difference in versions causes the cluster to hang indefinitely awaiting the active state of GPFS.
Cause
The issue may occur whenever you upgrade the Kernel version without changing the versions of other Kernel packages. For more details about Kernel and Kernel packages, see Building IBM Spectrum Scale portability layer after Linux kernel updates topic of the Knowledge Center.
Resolving The Problem
As a resolution, run the following manual steps to recover the cluster from hung state and to start the auto revert:
- Compile GPFS portability layer for this kernel version in a different virtual machine by using the steps mentioned in the following Knowledge Center link: https://www.ibm.com/support/knowledgecenter/SSL5ES_2.3.0.1/intel/GPFS12/gpfs_build_portlayer.html.
Note: In the Building IBM Spectrum Scale portability layer after Linux kernel updates topic, you can skip the sub-steps of step 3 to start the node and check for node active state.
- Copy the content that is available in the "/lib/modules/<upgraded kernel version>/extra" folder from the system where the GPFS portability layer is successful and paste it in the "/lib/modules/<upgraded kernel version>/extra" folder of the virtual machine where the upgrade failed.
- Run the following command to start GPFS:
su - gpfsprod -c 'sudo /usr/lpp/mmfs/bin/mmstartup’ - Run the following command to check whether all the nodes are in active state:
su - gpfsprod -c 'sudo /usr/lpp/mmfs/bin/mmgetstate -aL
Document Location
Worldwide
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFQSV","label":"IBM Cloud Pak System Software"},"ARM Category":[],"Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"2.3.0;2.3.1;2.3.2;2.3.3"}]
Was this topic helpful?
Document Information
Modified date:
11 September 2020
UID
ibm11101861