Troubleshooting
Problem
When you upgrade to 10.0.1.4 or 10.0.3, the following error occurs during data population and the upgrade aborts.
Example of error from the apim-data-populate
pod:
[00000000-0000-0000-0000-000000000000] Error in POST post:/api/tasks (my_task.js:create)
- status : 409
- message: Another request might be operating on the resource. Please try again later.
- stack : Error: Another request might be operating on the resource. Please try again later.
at error (/app/node_modules/bhendi/lib/bhendiUtil.js:59:11)
at execute (/app/node_modules/bhendi/lib/postgres/db.js:1192:23)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async Object.insert (/app/node_modules/bhendi/lib/postgres/db.js:725:5)
at async Object.insert (/app/node_modules/bhendi/lib/db.js:182:20)
at async MyTask.create (/app/node_modules/bhendi/lib/controller.js:436:9)
at async dispatch (/app/node_modules/bhendi/mw/dispatcher.js:419:20)
at async Array.<anonymous> (/app/node_modules/bhendi/mw/dispatcher.js:329:9)
Symptom
Upgrade failure.
Diagnosing The Problem
"kubectl get jobs -n <namespace>" shows a failed data populate job.
For example:
root@m00:~# kubectl get jobs
NAME COMPLETIONS DURATION AGE
apiconnect-v10-upgrade-load 1/1 22m 21h
backrest-backup-mgmt-site1-postgres 1/1 42s 78m
mgmt-up-apim-data-populate-0-to-14-52e3c753 1/1 49s 21h
mgmt-up-apim-data-populate-14-to-134-afd2749b 0/1 <========= 69m 69m
mgmt-up-apim-schema-1-to-14-52e3c753 1/1 114s 21h
mgmt-up-apim-schema-14-to-134-afd2749b 1/1 3m10s 73m
mgmt-up-lur-data-populate-0-to-3-52e3c753 1/1 12s 21h
mgmt-up-lur-schema-1-to-3-52e3c753 1/1 43s 21h
Resolving The Problem
10.0.1.4 apim data populates LA ifix steps
- Upload the image to customer registry
- Fixcentral private link for:
apim_custom_upgrade_fp4_fix.tar.gz
https://ibm.biz/Bdfek8 - icr entitled registry image location:
icr.io/cp/apic/ibm-apiconnect-management-apim:custom_upgrade_fp4_fix
- Fixcentral private link for:
- Make sure the image is tagged.
- For example, if
docker load < apim_custom_upgrade_fp4_fix.tar.gz
was used. And the image appears to not be tagged like so (docker images
):REPOSITORY TAG IMAGE ID CREATED SIZE <none> <none> 5fae4d4b9e02 8 weeks ago 581.3 MB
docker tag <image-id> <customer_registry/apim:apim_custom_upgrade_fp4_fix>
- For example, if
- Manually Update mgmt (or top apic cr if that's being used) CR to 10.0.1.4-eus
kubectl edit mgmt
Change valuespec.version
to 10.0.1.4-eus- NOTE: there are 3 main yaml blocks.
metadata: ..... spec: .... status: ...
spec.version
that needs to change is explicitly found in thespec:
block and only 1 value is changed for this field.
- Wait for operator to show the apim-data-populate job, which it will likely fail.
watch kubectl get pods
To watch the pods list. The pod looks like:<mgmt-cr-name>-up-apim-data-populate-<number>-to-134-<job_id>-<pod-id>
watch kubectl get jobs
To watch the jobs list. The job looks like:<mgmt-cr-name>-up-apim-data-populate-<number>-to-134-<job_id>
- Scale down operator to prevent a rollback.
kubectl -n <namespace> scale deploy ibm-apiconnect --replicas=0
- Capture the job yaml and save it as yaml locally to be applied later
kubectl get job <name_of_apim-data-populate-job> -o yaml > apim-data-populate.yaml
- The name of the data populate job gets generated so run
kubectl get jobs
and look to find the....apim-data-populate....
job that is marked0/1
Completions
- Delete the apim-data-populate job
kubectl delete job <name_of_apim-data-populate-job>
- Modify the captured job yaml by changing container.image to the new custom APIM image, as well as remove the 2 fields and values marked
controller-uid
vi apim-data-populate.yaml
- Change
image: <original_image_location>
toimage: <customer_registry/apim:apim_custom_upgrade_fp4_fix>
- Remove the 2 fields and values marked
controller-uid: <id>
- Create the job and make sure job succeeds
kubectl create -f apim-data-populate.yaml
- Wait for the new populate job to
Complete
Successfully - Scale up operator and now it detects that the upgrade is completed
kubectl -n <namespace> scale deploy ibm-apiconnect --replicas=1
10.0.3.0 apim data populate LA ifix steps
- Upload the image to customer registry
- Fixcentral private link for:
apim_custom_upgrade_mod3_fix.tar.gz
https://ibm.biz/Bdfekp - icr entitled registry image location:
icr.io/cp/apic/ibm-apiconnect-management-apim:custom_upgrade_mod3_fix
- Fixcentral private link for:
- Make sure image is tagged.
- For example, if
docker load < apim_custom_upgrade_mod3_fix.tar.gz
was used. And the image appears to not be tagged like so (docker images
):REPOSITORY TAG IMAGE ID CREATED SIZE <none> <none> 5fae4d4b9e02 8 weeks ago 581.3 MB
docker tag <image-id> <customer_registry/apim:apim_custom_upgrade_mod3_fix>
- For example, if
- Manually Update mgmt (or top apic cr if that's being used) CR to 10.0.3.0
kubectl edit mgmt
Change valuespec.version
to 10.0.3.0- Also, update the
spec.license
field to the appropriate license value the customer is using for 10.0.3.0 - NOTE: there are 3 main yaml blocks.
metadata: ..... spec: .... status: ...
spec.version
andspec.license
that needs to change is explicitly found in thespec:
block and only 1 value is changed for each field.
- Wait for operator to show the apim-data-populate job, which will likely fail
watch kubectl get pods
To watch the pods list. The pod looks like:<mgmt-cr-name>-up-apim-data-populate-<number>-to-134-<job_id>-<pod-id>
watch kubectl get jobs
To watch the jobs list. The job looks like:<mgmt-cr-name>-up-apim-data-populate-<number>-to-134-<job_id>
- Scale down operator to prevent a rollback.
kubectl -n <namespace> scale deploy ibm-apiconnect --replicas=0
- Capture the job yaml and save it as yaml locally to be applied later
kubectl get job <name_of_apim-data-populate-job> -o yaml > apim-data-populate.yaml
- The name of the data populate job gets generated so run
kubectl get jobs
and look to find the....apim-data-populate....
job that is marked0/1
Completions
- Delete the apim-data-populate job
kubectl delete job <name_of_apim-data-populate-job>
- Modify the captured job yaml by changing container.image to the new custom APIM image, as well as remove the 2 fields and values marked
controller-uid
vi apim-data-populate.yaml
- Change
image: <original_image_location>
toimage: <customer_registry/apim:apim_custom_upgrade_mod3_fix>
- Remove the 2 fields and values marked
controller-uid: <id>
- Create the job and make sure job succeeds
kubectl create -f apim-data-populate.yaml
- Wait for the new populate job to
Complete
Successfully - Scale up operator and now it detects that the upgrade is completed
kubectl -n <namespace> scale deploy ibm-apiconnect --replicas=1
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSMNED","label":"IBM API Connect"},"ARM Category":[{"code":"a8m50000000CeCAAA0","label":"API Connect->Management and Monitoring (MM)->Upgrade\/Downgrade"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"10.0.0"}]
Was this topic helpful?
Document Information
Modified date:
17 August 2021
UID
ibm16481993