Troubleshooting
Problem
After upgrading the IBM Foundational Services (Common Services) operator Version 3.19.9+ to 3.20+, the Platform Navigator operator failed to complete the update.
- Upgrading from Common Services 3.19.9 and above to Common Services version 3.20.x, 3.21.x and 3.22.x is not supported.
- You can upgrade from Common Service 3.19.9 and above directly to Common Services version 3.23.X.
- Rollback of IBM Foundational Service operators is supported. You will run into a similar issue.
Symptom
The Cloud Pak for Integration operator `Platform navigator' waiting on the Zen operator to complete the installation and might see a missing Rollback flag in the logs
Waiting for Zen to install. This can take up to 30 minutes, please wait.
The ZenService object in the [stg-cp4i-cmn] namespace is being fulfilled.
The ZenService status is [Rollback operation is requested from 4.8.7 to 4.7.1 but the acceptRollback flag is not set to True
Error found in prep_playrun.yaml from zen-ansible-utils].
Cause
The ibm-common-service-catalog source has multiple channels dedicated to different versions of the IBM foundational services (Common services)
Channel V3 - Common services version 3.19.x and bellow
Channel V3.20 - Common services version 3.20.x
Channel V3.21 - Common services version 3.21.x
Channel V3.23 - Common services version 3.23.x
Channel V3, a Long Term Support Release (LTSR), includes the Foundational Services 3.19.x versions and the relevant operands and the Zen operator
The foundational services 3.19.9 version was rebased with 3.23.1 code to make it stable and include all the relevant fixes for all the Common services operators. The V3 channel has the same operator versions that were made available for the Foundational service 3.23.1 in Channel 3.23
For example:
opm alpha list bundles icr.io/cpopen/ibm-operator-catalog:latest | "zen|common-service-operator " latest-bundlelist.txt | awk '{print $2 "\t" $3 }' | sort -u
v3 ibm-common-service-operator.v3.19.14
v3 ibm-zen-operator.v1.8.7
v3 zen-cpp-operator.v1.8.7
v3.20 ibm-common-service-operator.v3.20.0
v3.20 ibm-common-service-operator.v3.20.1
v3.20 ibm-zen-operator.v1.7.1 <<<< These operators are lower version than the Channel V3
v3.20 zen-cpp-operator.v1.7.0 <<<< These operators are lower version than the Channel V3
v3.21 ibm-common-service-operator.v3.21.0
v3.21 ibm-zen-operator.v1.7.2 <<<< These operators are lower version than the Channel V3
v3.21 zen-cpp-operator.v1.7.1 <<<< These operators are lower version than the Channel V3
v3.22 ibm-common-service-operator.v3.22.0
v3.22 ibm-zen-operator.v1.8.0 <<<< These operators are lower version than the Channel V3
v3.22 zen-cpp-operator.v1.8.0 <<<< These operators are lower version than the Channel V3
v3.23 ibm-common-service-operator.v3.23.6
v3.23 ibm-zen-operator.v1.8.7
v3.23 zen-cpp-operator.v1.8.7
To upgrade the "IBM Foundational services" operator from 3.19.9+ by changing the channel to V 3.20 or v 3.21, many common service operands will likely move to a lower version and experience upgrade or runtime issues. The Zen operator rolled back from ibm-zen-operator.v1.8.7 to ibm-zen-operator.v1.7.2, causing the Platform Navigator upgrade issues.
Environment
Cloud Pak for integration upgraded environment and common services upgraded from 3.19.9+ to Common services 3.20
oc get csv -A
Diagnosing The Problem
1. Check if the common services operator was upgraded from v3.19.9+ to v3.20.x
oc get csv -A | grep ibm-common-service-operator
2-- Check the Platform navigator operator logs for error
Get the pods name
oc get pods -A | grep platform-navigator
openshift-operators ibm-integration-platform-navigator-operator-6d569dc759-qwr7l 1/1 Running 0 33d
oc logs ibm-integration-platform-navigator-operator-6d569dc759-qwr7l -n openshift-operators | egrep error
2023-09-13T21:25:28.599975611Z {"level":"error","ts":1694640328.5999086,"logger":"controller_platformnavigator",
"msg":"Waiting for Zen to install. This can take up to 30 minutes, please wait. The ZenService object in the [stg-cp4i-cmn] namespace is being fulfilled.
The ZenService status is [Running reconciliation].","Namespace":"stg-cp4i-cmn","Name":"platform-navigator","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
2023-09-13T21:14:26.342672106Z {"level":"error","ts":1694639666.3425958,"logger":"controller_platformnavigator",
"msg":"Waiting for Zen to install. This can take up to 30 minutes, please wait. The ZenService object in the [stg-cp4i-cmn] namespace is being fulfilled.
The ZenService status is [Rollback operation is requested from 4.8.7 to 4.7.1 but the acceptRollback flag is not set to True\nError found in prep_playrun.yaml from zen-ansible-utils].","Namespace":"stg-cp4i-cmn","Name":"platform-navigator","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
Resolving The Problem
The common services catalog channel v3.23 includes the latest common services operands and the Zen versions. Upgrade the common services operator to the newest version available on the V 3.23 channel.
1- Review the currently installed Cloud Pack Operator versions and ensure the Cloud Pak Operators are compatible with the common services v3.23.
1- Review the currently installed Cloud Pack Operator versions and ensure the Cloud Pak Operators are compatible with the common services v3.23.
(Most likely, they are compatible since the operators running with the v3.19.9+ are rebased from v3.23.1)
2- In the Openshift console -> operator -> installed Operators, select the "IBM Foundational services operator,"
Under the subscription tab, change the channel to v3.23.
Approve the install plans if they are set to manual and pending approval.
This should start the IBM Foundational services upgrade and should also upgrade the Zen operator.
3-- Wait for all the operators to complete the installation and validate the current versions.
oc get csv -A
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB67","label":"IT Automation \u0026 App Modernization"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SS8QTD","label":"IBM Cloud Pak for Integration"},"ARM Category":[{"code":"a8m0z0000001hogAAA","label":"Common Services"}],"ARM Case Number":"TS014024266","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"},{"Type":"MASTER","Line of Business":{"code":"LOB10","label":"Data and AI"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSHGYS","label":"IBM Cloud Pak for Data"},"ARM Category":[{"code":"a8m3p000000UoRRAA0","label":"Administration-\u003EUpgrade"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"},{"Type":"MASTER","Line of Business":{"code":"LOB10","label":"Data and AI"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSBYVB","label":"IBM Cloud Pak for Business Automation"},"ARM Category":[{"code":"a8m3p000000hBR2AAM","label":"Other-\u003ECloudPak4Automation Platform-\u003EFoundation Services-\u003EZen Service"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
09 May 2024
UID
ibm17032559