IBM Support

CP4S: Unable to attach or mount volumes after making changes in Vsphere account

Troubleshooting


Problem

Changing the password of the Vsphere account that provisioned the CP4S cluster or disabling this account affects the PVC mounts and CP4S becoming inaccessible.

Symptom

The following events would be visible on pods that are not starting the following is a sample with icp-mongodb pod:
  Warning  FailedMount         <invalid>  kubelet                  Unable to attach or mount volumes: unmounted volumes=[mongodbdir], unattached volumes=[ibm-mongodb-operand-token-xb7n2 configdir config mongodbdir keydir tmp-mongodb ca init install tmp-metrics]: timed out waiting for the condition
  Warning  FailedMount         <invalid>  kubelet                  Unable to attach or mount volumes: unmounted volumes=[mongodbdir], unattached volumes=[ca tmp-metrics init config install keydir tmp-mongodb configdir ibm-mongodb-operand-token-xb7n2 mongodbdir]: timed out waiting for the condition
CP4S user Interface is not accessible.

Environment

-Openshift
-Vsphere
-Cloud Pak for Security

Diagnosing The Problem

By running:
$ oc get events
You are able to see the messages the pods Events complaining about 
Unable to attach or mount volumes: unmounted volumes
By confirming that were changes in the Vsphere account such as user password changes or user was disabled or username changed.

Resolving The Problem

Solution 1:  Revert Changes made in the Vsphere account to the previous status, for example change password back to the previous one, or if the account was disabled, enable the same back to make the mount work.
Solution 2: Update the secret vsphere-creds under the kube-config namespace.
  •  Verify secret content, username, and password are base64 encoded for the vcsa keys. 
    $ oc get secret vsphere-creds -n kube-system -o yaml
  • Take a backup of the current secret and keep it safe for a backout plan you can replace again. 
    $ oc get secret vsphere-creds -o yaml -n kube-system > creds_backup.yaml
$ oc get cm cloud-provider-config -o yaml -n openshift-config > cloud.yaml
  • The cloud-provider looks for credentials from the vsphere-creds secret.  
    $ echo -n "OpenShifttestpassword" | base64 -w0
T3BlbnNoaWZ0dGVzdHBhc3N3b3Jk
    Note: It is important to use the command to generate the base64-encoded passwords, as otherwise "new-line" characters can be encoded as well.
  • Edit, Save the file, and replace it.
    $ cp creds_backup.yaml creds.yaml
$ vi creds.yaml

    apiVersion: v1
    data:
      vcsa-ci.vmware.devcluster.openshift.com.password: T3BlbnNoaWZ0dGVzdHBhc3N3b3Jk    >>> copy your base64 encoded password here
      vcsa-ci.vmware.devcluster.openshift.com.username: YWRtaW5AdnNwaGVyZS5sb2NhbAo=
    kind: Secret
    metadata:

$ oc replace -f creds.yaml
  • Verify the password in the secret by decoding the base64 secret value for the password. 
    echo 'T3BlblNoaWZ0dGVzdHBhc3N3b3Jk' | base64 -d
OpenShifttestpassword[root@localhost ~]#
     
  • If the password was not changed, try the following command to force a redeployment of the kube-controller-manager. 
    $ oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSTDPP","label":"IBM Cloud Pak for Security"},"ARM Category":[{"code":"a8m0z000000Xat9AAC","label":"Documentation"}],"ARM Case Number":"TS007633489","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
27 January 2022

UID

ibm16541932

Page Feedback