IBM Support

Upgrade to API Connect 10.0.1.4/10.0.3 fails with data populate 409 error

Troubleshooting


Problem

When you upgrade to 10.0.1.4 or 10.0.3, the following error occurs during data population and the upgrade aborts.

Example of error from the apim-data-populate pod:

[00000000-0000-0000-0000-000000000000] Error in POST post:/api/tasks (my_task.js:create)
  - status : 409
  - message: Another request might be operating on the resource. Please try again later.
  - stack  : Error: Another request might be operating on the resource. Please try again later.
    at error (/app/node_modules/bhendi/lib/bhendiUtil.js:59:11)
    at execute (/app/node_modules/bhendi/lib/postgres/db.js:1192:23)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Object.insert (/app/node_modules/bhendi/lib/postgres/db.js:725:5)
    at async Object.insert (/app/node_modules/bhendi/lib/db.js:182:20)
    at async MyTask.create (/app/node_modules/bhendi/lib/controller.js:436:9)
    at async dispatch (/app/node_modules/bhendi/mw/dispatcher.js:419:20)
    at async Array.<anonymous> (/app/node_modules/bhendi/mw/dispatcher.js:329:9)

Symptom

Upgrade failure.

Diagnosing The Problem

"kubectl get jobs -n <namespace>" shows a failed data populate job.
For example:
root@m00:~# kubectl get jobs
NAME                                            COMPLETIONS   DURATION   AGE
apiconnect-v10-upgrade-load                     1/1           22m        21h
backrest-backup-mgmt-site1-postgres             1/1           42s        78m
mgmt-up-apim-data-populate-0-to-14-52e3c753     1/1           49s        21h
mgmt-up-apim-data-populate-14-to-134-afd2749b   0/1   <=========        69m        69m
mgmt-up-apim-schema-1-to-14-52e3c753            1/1           114s       21h
mgmt-up-apim-schema-14-to-134-afd2749b          1/1           3m10s      73m
mgmt-up-lur-data-populate-0-to-3-52e3c753       1/1           12s        21h
mgmt-up-lur-schema-1-to-3-52e3c753              1/1           43s        21h

Resolving The Problem

10.0.1.4 apim data populates LA ifix steps

  1. Upload the image to customer registry
    • Fixcentral private link for: apim_custom_upgrade_fp4_fix.tar.gz https://ibm.biz/Bdfek8
    • icr entitled registry image location: 
      icr.io/cp/apic/ibm-apiconnect-management-apim:custom_upgrade_fp4_fix
  2. Make sure the image is tagged.
    • For example, if docker load < apim_custom_upgrade_fp4_fix.tar.gz was used. And the image appears to not be tagged like so (docker images): 
      REPOSITORY    TAG      IMAGE ID            CREATED          SIZE
<none>       <none>    5fae4d4b9e02        8 weeks ago      581.3 MB
      Then tag the image for easy use: docker tag <image-id> <customer_registry/apim:apim_custom_upgrade_fp4_fix>
  3. Manually Update mgmt (or top apic cr if that's being used) CR to 10.0.1.4-eus
    • kubectl edit mgmt Change value spec.version to 10.0.1.4-eus
    • NOTE: there are 3 main yaml blocks. 
      metadata:
.....
spec:
....
status:
...
      The spec.version that needs to change is explicitly found in the spec: block and only 1 value is changed for this field.
  4. Wait for operator to show the apim-data-populate job, which it will likely fail.
    • watch kubectl get pods To watch the pods list. The pod looks like: <mgmt-cr-name>-up-apim-data-populate-<number>-to-134-<job_id>-<pod-id>
    • watch kubectl get jobs To watch the jobs list. The job looks like: <mgmt-cr-name>-up-apim-data-populate-<number>-to-134-<job_id>
  5. Scale down operator to prevent a rollback.
    • kubectl -n <namespace> scale deploy ibm-apiconnect --replicas=0
  6. Capture the job yaml and save it as yaml locally to be applied later
    • kubectl get job <name_of_apim-data-populate-job> -o yaml > apim-data-populate.yaml
    • The name of the data populate job gets generated so run kubectl get jobs and look to find the ....apim-data-populate.... job that is marked 0/1 Completions
  7. Delete the apim-data-populate job
    • kubectl delete job <name_of_apim-data-populate-job>
  8. Modify the captured job yaml by changing container.image to the new custom APIM image, as well as remove the 2 fields and values marked controller-uid
    • vi apim-data-populate.yaml
    • Change image: <original_image_location> to image: <customer_registry/apim:apim_custom_upgrade_fp4_fix>
    • Remove the 2 fields and values marked controller-uid: <id>
  9. Create the job and make sure job succeeds
    • kubectl create -f apim-data-populate.yaml
  10. Wait for the new populate job to Complete Successfully
  11. Scale up operator and now it detects that the upgrade is completed
    • kubectl -n <namespace> scale deploy ibm-apiconnect --replicas=1

10.0.3.0 apim data populate LA ifix steps

  1. Upload the image to customer registry
    • Fixcentral private link for: apim_custom_upgrade_mod3_fix.tar.gz  https://ibm.biz/Bdfekp
    • icr entitled registry image location: 
      icr.io/cp/apic/ibm-apiconnect-management-apim:custom_upgrade_mod3_fix
  2. Make sure image is tagged.
    • For example, if docker load < apim_custom_upgrade_mod3_fix.tar.gz was used. And the image appears to not be tagged like so (docker images): 
      REPOSITORY    TAG      IMAGE ID            CREATED          SIZE
<none>       <none>    5fae4d4b9e02        8 weeks ago      581.3 MB
      Then tag the image for easy use: docker tag <image-id> <customer_registry/apim:apim_custom_upgrade_mod3_fix>
  3. Manually Update mgmt (or top apic cr if that's being used) CR to 10.0.3.0
    • kubectl edit mgmt Change value spec.version to 10.0.3.0
    • Also, update the spec.license field to the appropriate license value the customer is using for 10.0.3.0
    • NOTE: there are 3 main yaml blocks. 
      metadata:
.....
spec:
....
status:
...
      The spec.version and spec.license that needs to change is explicitly found in the spec: block and only 1 value is changed for each field.
  4. Wait for operator to show the apim-data-populate job, which will likely fail
    • watch kubectl get pods To watch the pods list. The pod looks like: <mgmt-cr-name>-up-apim-data-populate-<number>-to-134-<job_id>-<pod-id>
    • watch kubectl get jobs To watch the jobs list. The job looks like: <mgmt-cr-name>-up-apim-data-populate-<number>-to-134-<job_id>
  5. Scale down operator to prevent a rollback.
    • kubectl -n <namespace> scale deploy ibm-apiconnect --replicas=0
  6. Capture the job yaml and save it as yaml locally to be applied later
    • kubectl get job <name_of_apim-data-populate-job> -o yaml > apim-data-populate.yaml
    • The name of the data populate job gets generated so run kubectl get jobs and look to find the ....apim-data-populate.... job that is marked 0/1 Completions
  7. Delete the apim-data-populate job
    • kubectl delete job <name_of_apim-data-populate-job>
  8. Modify the captured job yaml by changing container.image to the new custom APIM image, as well as remove the 2 fields and values marked controller-uid
    • vi apim-data-populate.yaml
    • Change image: <original_image_location> to image: <customer_registry/apim:apim_custom_upgrade_mod3_fix>
    • Remove the 2 fields and values marked controller-uid: <id>
  9. Create the job and make sure job succeeds
    • kubectl create -f apim-data-populate.yaml
  10. Wait for the new populate job to Complete Successfully
  11. Scale up operator and now it detects that the upgrade is completed
    • kubectl -n <namespace> scale deploy ibm-apiconnect --replicas=1

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSMNED","label":"IBM API Connect"},"ARM Category":[{"code":"a8m50000000CeCAAA0","label":"API Connect->Management and Monitoring (MM)->Upgrade\/Downgrade"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"10.0.0"}]

Document Information

Modified date:
17 August 2021

UID

ibm16481993

Page Feedback