IBM Support

IBM Maximo Application Suite - Installation fails with "Wait for db2u instance to be ready"

Troubleshooting


Problem

During the IBM Maximo Application Suite deployment you can see that the Db2 deployment fails with "Wait for db2u instance to be ready". The problem described here is in relation with OpenShift Container Storage usage.

Symptom

The following events and logs can be seen in logs:
Events: 
Type    Reason            Age                  From              Message 
----    ------            ----                 ----              ------- 
Warning FailedScheduling  22m  (x31 over 177m) default-scheduler 0/6 nodes are available: 6 pod has unbound immediate PersistentVolumeClaims. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.

Diagnosing The Problem

There are a few indications for why the Db2 cluster cannot become in Ready state:

1) Requested PVCs to utilize the ODF storage cluster are in pending

2) OCS storage cluster shows the error:
"error":"Not enough nodes found: Expected 3, found 0"
Helpful traces : 
Line 227: 2024-01-10T00:37:16.164896030Z {"level":"error","ts":1704847036.1648421,"logger":"controllers.StorageCluster","msg":"Failed to set node Topology Map for StorageCluster.","Request.Namespace":"openshift-storage","Request.Name":"ocs-storagecluster","StorageCluster":{"name":"ocs-storagecluster","namespace":"openshift-storage"},"error":"Not enough nodes found: Expected 3, found 0","stacktrace":"github.com/red-hat-storage/ocs-operator/controllers/storagecluster.(*StorageClusterReconciler).reconcilePhases\n\t/remote-source/app/controllers/storagecluster/reconcile.go:402\ngithub.com/red-hat-storage/ocs-operator/controllers/storagecluster.(*StorageClusterReconciler).Reconcile\n\t/remote-source/app/controllers/storagecluster/reconcile.go:166\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"} 
Line 229: 2024-01-10T00:37:16.174935134Z {"level":"error","ts":1704847036.1749022,"msg":"Reconciler error","controller":"storagecluster","controllerGroup":"ocs.openshift.io","controllerKind":"StorageCluster","storageCluster":{"name":"ocs-storagecluster","namespace":"openshift-storage"},"namespace":"openshift-storage","name":"ocs-storagecluster","reconcileID":"79fb5060-36b5-471f-a2c9-a688b5c2c7af","error":"Not enough nodes found: Expected 3, found 0","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"} 
Line 233: 2024-01-10T00:37:38.394662881Z {"level":"error","ts":1704847058.394608,"logger":"controllers.StorageCluster","msg":"Failed to set node Topology Map for StorageCluster.","Request.Namespace":"openshift-storage","Request.Name":"ocs-storagecluster","StorageCluster":{"name":"ocs-storagecluster","namespace":"openshift-storage"},"error":"Not enough nodes found: Expected 3, found 0","stacktrace":"github.com/red-hat-storage/ocs-operator/controllers/storagecluster.(*StorageClusterReconciler).reconcilePhases\n\t/remote-source/app/controllers/storagecluster/reconcile.go:402\ngithub.com/red-hat-storage/ocs-operator/controllers/storagecluster.(*StorageClusterReconciler).Reconcile\n\t/remote-source/app/controllers/storagecluster/reconcile.go:166\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
3) Nooba operator traces: 
2024-01-10T00:43:40.194150688Z time="2024-01-10T00:43:40Z" level=info msg="❌ Not Found: NooBaa \"noobaa\"\n" 
2024-01-10T00:43:40.194150688Z time="2024-01-10T00:43:40Z" level=error msg="❌ NooBaa \"noobaa\" failed to add mandatory meta fields"
4) The investigation indicates that the installation of the ODF and OCS storage clusters is not completed yet. Below logs from storage-cluster can provide a clear indication of the problem: 
conditions: 
    - lastHeartbeatTime: "2024-01-10T00:37:38Z" 
      lastTransitionTime: "2024-01-09T23:37:07Z" 
      message: 'Error while reconciling: Not enough nodes found: Expected 3, found 0'
      reason: ReconcileFailed
      status: "False"

Resolving The Problem

The reason of the failure in this scenario is caused by the missing labels for the OCS storage class on the worker nodes of the OpenShift cluster. To schedule the odf pods on nodes we should have the OCS labels on those OpenShift nodes. Once the nodes have that label cluster will start creating the ODF pods on those nodes. After all pods are up you can utilize the storage cluster.

Steps to fix the problem:

1) Login into the OpenShift cluster with an admin user 
# oc login
2) Check if the OCP nodes already have the ocs label
# oc get nodes --show-labels | grep ocs
3) Get Nodes to label 
# oc get nodes
4) Label the nodes with the cluster.ocs.openshift.io/openshift-storage to make them available for the storage cluster
# oc label node <node-name> cluster.ocs.openshift.io/openshift-storage
where <node-name> is the output of the first command. 

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB59","label":"Sustainability Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSRHPA","label":"IBM Maximo Application Suite"},"ARM Category":[{"code":"a8m3p000000hAeeAAE","label":"Maximo Application Suite-\u003ECore"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
16 January 2024

UID

ibm17107760

Page Feedback