Reconciliation loop after upgrading the Db2 service

Important: IBM Cloud Pak for Data Version 4.6 will reach end of support (EOS) on 31 July, 2025. For more information, see the Discontinuance of service announcement for IBM Cloud Pak for Data Version 4.X.

Upgrade to IBM Software Hub Version 5.1 before IBM Cloud Pak for Data Version 4.6 reaches end of support. For more information, see Upgrading IBM Software Hub in the IBM Software Hub Version 5.1 documentation.

The db2oltp and db2aaservice operator pod will repeatedly perform a reconciliation loop while stating that the reconciliation process has completed. To fix the issue, delete then recreate the custom resource.

Symptoms

  • The db2oltp operator pod repeatedly states set db2oltp status to Completed, yet the operator pod logs are in reconciliation. To view the status of the logs, use the following command:
    oc logs  $(oc get po -n ${PROJECT_CPD_OPS} |  grep ibm-db2oltp-cp4d-operator-controller-manager | cut -f1 -d' ') -n ${PROJECT_CPD_OPS}  | grep "set db2oltp status to Completed"
  • The message field in the db2oltp custom resource states Last reconciliation succeeded while also stating Running reconciliation in another message field. To view the db2oltp custom resource, use the following command:
    oc get db2oltpservice db2oltp-cr -ncpd-instance -oyaml
    -lastTransitionTime: "<timestamp>"
    message: Last reconciliation succeeded
    reason: Successful
    status: "False"
    type: Successful
    -lastTransitionTime: "<timestamp>"
    message: Running reconciliation
    reason: Running
    status: "True"
    type: Running
    db2oltpStatus: InProgress
  • The db2aaservice operator pod repeatedly states set db2aaservice status to Completed, yet the operator pod logs are in reconciliation. To view the status of the logs, use the following command:
    oc logs  $(oc get po -n ${PROJECT_CPD_OPS} |  grep ibm-db2aaservice-cp4d-operator-controller-manager | cut -f1 -d' ') -n ${PROJECT_CPD_OPS}  | grep "set db2aaservice status to Completed"
  • The message field in the db2aaservice custom resource states Last reconciliation succeeded while also stating Running reconciliation in another message field. To view the db2aaservice custom resource, use the following command:
    oc get db2aaserviceservice db2aaservice-cr -ncpd-instance -oyaml
    -lastTransitionTime:" <timestamp>"
    message: Last reconciliation succeeded
    reason: Successful
    status: "False"
    type: Successful
    -lastTransitionTime: "<timestamp>"
    message: Running reconciliation
    reason: Running
    status: "True"
    type: Running
    db2aaserviceStatus: InProgress

Resolving the problem

  1. Delete the db2aaservice custom resource with the following command:
    oc delete db2aaserviceservice db2aaservice-cr -ncpd-instance
  2. Wait for the db2oltpStatus: field to change from InProgress to Completed. Use the following command to retrieve the custom resource status and find this field:
    oc get db2oltpservice db2oltp-cr -ncpd-instance -oyaml | grep Status
  3. Create the new db2aaservice custom resource by modifying the following yaml file with details of your environment:
    cat<<EOF|ocapply-f-
    apiVersion: databases.cpd.ibm.com/v1
    kind: Db2aaserviceService
    metadata:
      name: db2aaservice-cr
      namespace: cpd-instance
    spec:
      license:
        accept: true
        license: Enterprise
    
    EOF
  4. Wait for the db2aaserviceStatus and db2oltpStatus fields in the custom resource to change to Completed.
    1. Use the following command to retrieve the db2aaservice custom resource and find this field:
      oc get db2aaserviceservice db2aaservice-cr -ncpd-instance -oyaml | grep Status
    2. Use the following command to retrieve the db2oltp custom resource and find this field:
      oc get db2oltpservice db2oltp-cr -ncpd-instance -oyaml | grep Status