Troubleshooting
Problem
The synchronization process from information assets to the default catalog assets of IBM Watson® Knowledge Catalog (WKC) is stuck.
Symptom
What’s happening?
When migrating large amounts of data and a catalog-api pod terminates unexpectedly, you need to release the locks to resume synchronization.
Cause
Check:
- If information assets are not synced from Information Governance Catalog to the default catalog assets of Watson Knowledge Catalog.
- If any of the catalog-api pods has been restarted recently.
- If catalog-api logs contain indications of locking problems, such as frequent "Cannot acquire lock <lock name>" messages.
- If the script (below) runs in PREVIEW mode and finds locks that need to be removed.
Resolving The Problem
Run the script (below) in CLEAN UP mode to remove the locks. The lock removal triggers the synchronization process from information assets to the default catalog assets of Watson Knowledge Catalog to start again.
How to run the script to check or remove the locks
The following steps describe how to check or remove locks:
- SSH into a node from which you can run kubectl commands. For a stand-alone installation of IBM Cloud Pak™ for Data, SSH into the master node (master-1) of your cluster. You must use a user with sufficient privileges to run the kubectl commands.
- Determine the name of the redis master pod (<redis pod name>) in your environment. This is part of the command in step 5. To determine the redis master pod name run the following command:
kubectl exec -n <namespace> redis-ha-server-0 /usr/local/bin/redis-cli info | grep role
If the response is:role:master
If the response is:role:slave
- Copy and paste the following script into a file named "script.lua":
-- Redis LUA script that finds and optionally removes catalog-api locks that have been held -- for over one hour. -- -- Usage: -- -- 1) SSH into a node from which you can run kubectl commands. -- For a stand-alone installation of IBM(R) Cloud Pak(TM) for Data, SSH into the master node (master-1) of your cluster. -- You must use a user with sufficient privileges to run the kubectl commands. -- 2) Determine the name of the redis master pod (<redis pod name>) in your environment. -- This is part of the command in step 5. To determine the redis master pod name run the following command: -- kubectl exec -n <namespace> redis-ha-server-0 /usr/local/bin/redis-cli info | grep role -- where <namespace> is the namespace used by Cloud Pak for Data (typically either wkc or zen). -- If the response is: role:master, then redis-ha-server-0 is the redis master pod. -- If the response is: role:slave, then redis-ha-server-1 is the redis master pod. -- 3) Copy and paste this script into a file named "script.lua". -- 4) Copy the file to the node that you used in step 1. -- 5) Run the script on the node as described below. The script returns a list of keys found that need to be removed. -- If you see errors when running the script, verify that the redis pod name and the namespace in the command match your environment. -- -- Note: It is recommended that you first run the script in PREVIEW mode before cleaning up the locks. -- If no keys are reported in the output, there is no need for clean-up. -- --- -- TO RUN THE SCRIPT IN PREVIEW MODE: -- -- kubectl -n <namespace> exec -i <redis pod name> -- /usr/local/bin/redis-cli --eval /dev/stdin < script.lua -- -- TO RUN THE SCRIPT AND CLEAN UP THE LOCKS -- -- kubectl -n <namespace> exec -i <redis pod name> -- /usr/local/bin/redis-cli --eval /dev/stdin , true < script.lua -- -- Note: -- <redis pod name> is the name of the redis master pod that was determined in step 2 above. -- <namespace> is the namespace used by IBM Cloud Pak for Data (typically either wkc or zen). -- local matches = redis.call('keys', 'WKC-CATALOG-ENV-wkc-*com.ibm.catalog.utils.synchronization.SynchronizationUtils*'); local keys_removed={}; local removed_idx = 1; local one_year_in_seconds = 31556952; local one_day_in_seconds = 86400; for _,key in ipairs(matches) do local types_table = redis.call('TYPE', key); local key_type = types_table['ok']; local ttl = tonumber(redis.call('TTL', key)); local lock_age = one_year_in_seconds - ttl; -- In newer code, TTL is 30 seconds by default -- To be safe, don't remove any locks with a TTL less than a day to avoid removing -- all locks created after the default TTL was changed if (key_type == 'hash' and ttl > one_day_in_seconds and lock_age > 3600) then if (ARGV[1] == 'true') then redis.call('DEL', key); redis.call('PUBLISH', 'redisson_lock__channel:{' .. key .. '}', 0); keys_removed[removed_idx] = 'Removed key: ' .. key; else keys_removed[removed_idx] = 'Found key: ' .. key; end; removed_idx = removed_idx + 1; end; end; return keys_removed;
- Copy the file to the node that you used in step 1.
- Run the script on the node as described below. The script returns a list of keys found that need to be removed. If you see errors when running the script, verify that the redis pod name and the namespace in the command match your environment.
Note: It is recommended that you first run the script in PREVIEW mode before cleaning up the locks. If no keys are reported in the output, there is no need for clean-up.
Commands to run the script:
- In PREVIEW mode:
kubectl -n <namespace> exec -i <redis pod name> -- /usr/local/bin/redis-cli --eval /dev/stdin < script.lua
- In CLEAN UP mode to clean up the locks:
kubectl -n <namespace> exec -i <redis pod name> -- /usr/local/bin/redis-cli --eval /dev/stdin , true < script.lua
Where:
- <redis pod name> is the name of the redis master pod that was determined in step 2 above.
- <namespace> is the namespace used by Cloud Pak for Data (typically either wkc or zen).
Command examples:
The following commands show how to run the file script.lua using the namespace zen and the redis master pod name redis-ha-server-1:
- In PREVIEW mode:
kubectl -n zen exec -i redis-ha-server-1 -- /usr/local/bin/redis-cli --eval /dev/stdin < script.lua
- In CLEAN UP mode:
kubectl -n zen exec -i redis-ha-server-1 -- /usr/local/bin/redis-cli --eval /dev/stdin , true < script.lua
Document Location
Worldwide
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSHGYS","label":"IBM Cloud Pak for Data"},"Component":"IBM Watson Knowledge Catalog","Platform":[{"code":"PF040","label":"RedHat OpenShift"}],"Version":"2.5.0","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]
Product Synonym
InfoSphere Information Governance Catalog; IGC; information assets; default catalog; Watson Knowledge Catalog; WKC; Cloud Pak for Data; CPD; cp4d;
Was this topic helpful?
Document Information
Modified date:
09 June 2020
UID
ibm12801163