IBM Support

Synchronization process from information assets to the default catalog of the IBM Watson Knowledge Catalog is stuck

Troubleshooting


Problem

The synchronization process from information assets to the default catalog assets of  IBM Watson® Knowledge Catalog (WKC) is stuck.

Symptom

What’s happening?
When migrating large amounts of data and a catalog-api pod terminates unexpectedly, you need to release the locks to resume synchronization.

Cause

Check:
  • If information assets are not synced from Information Governance Catalog to the default catalog assets of Watson Knowledge Catalog.
  • If any of the catalog-api pods has been restarted recently.
  • If catalog-api logs contain indications of locking problems, such as frequent "Cannot acquire lock <lock name>" messages.
  • If the script (below) runs in PREVIEW mode and finds locks that need to be removed.

Resolving The Problem

Run the script (below) in CLEAN UP mode to remove the locks.  The lock removal triggers the synchronization process from information assets to the default catalog assets of Watson Knowledge Catalog to start again.
How to run the script to check or remove the locks
The following steps describe how to check or remove locks:
  1. SSH into a node from which you can run kubectl commands.  For a stand-alone installation of IBM Cloud Pak™ for Data, SSH into the master node (master-1) of your cluster.  You must use a user with sufficient privileges to run the kubectl commands.
  2. Determine the name of the redis master pod (<redis pod name>)  in your environment. This is part of the command in step 5. To determine the redis master pod name run the following command:
    kubectl exec -n <namespace> redis-ha-server-0 /usr/local/bin/redis-cli info | grep role
    where <namespace> is the namespace used by Cloud Pak for Data (typically either wkc or zen).
    If the response is:
    role:master
    then redis-ha-server-0 is the redis master pod.
    If the response is:
    role:slave
    then redis-ha-server-1 is the redis master pod.
  3. Copy and paste the following script into a file named "script.lua":
    -- Redis LUA script that finds and optionally removes catalog-api locks that have been held
    -- for over one hour.
    --
    -- Usage:
    --
    -- 1) SSH into a node from which you can run kubectl commands.  
    --    For a stand-alone installation of IBM(R) Cloud Pak(TM) for Data, SSH into the master node (master-1) of your cluster.  
    --    You must use a user with sufficient privileges to run the kubectl commands.
    -- 2) Determine the name of the redis master pod (<redis pod name>) in your environment. 
    --    This is part of the command in step 5. To determine the redis master pod name run the following command: 
    --    kubectl exec -n <namespace> redis-ha-server-0 /usr/local/bin/redis-cli info | grep role
    --    where <namespace> is the namespace used by Cloud Pak for Data (typically either wkc or zen).
    --    If the response is: role:master, then redis-ha-server-0 is the redis master pod.
    --    If the response is: role:slave, then redis-ha-server-1 is the redis master pod.
    -- 3) Copy and paste this script into a file named "script.lua".  
    -- 4) Copy the file to the node that you used in step 1.
    -- 5) Run the script on the node as described below. The script returns a list of keys found that need to be removed.  
    --    If you see errors when running the script, verify that the redis pod name and the namespace in the command match your environment.
    --
    -- Note: It is recommended that you first run the script in PREVIEW mode before cleaning up the locks. 
    --       If no keys are reported in the output, there is no need for clean-up.
    --  
    ---
    -- TO RUN THE SCRIPT IN PREVIEW MODE:
    --
    -- kubectl -n <namespace> exec -i <redis pod name> -- /usr/local/bin/redis-cli --eval /dev/stdin < script.lua
    --
    -- TO RUN THE SCRIPT AND CLEAN UP THE LOCKS
    --
    -- kubectl -n <namespace> exec -i <redis pod name> -- /usr/local/bin/redis-cli --eval /dev/stdin , true < script.lua
    --
    -- Note:
    -- <redis pod name> is the name of the redis master pod that was determined in step 2 above.
    -- <namespace> is the namespace used by IBM Cloud Pak for Data (typically either wkc or zen).
    --
    local matches = redis.call('keys', 'WKC-CATALOG-ENV-wkc-*com.ibm.catalog.utils.synchronization.SynchronizationUtils*');
    local keys_removed={};
    local removed_idx = 1;
    local one_year_in_seconds = 31556952;
    local one_day_in_seconds = 86400;
    for _,key in ipairs(matches) do
        local types_table = redis.call('TYPE', key);
        local key_type = types_table['ok'];
        local ttl = tonumber(redis.call('TTL', key));
        local lock_age = one_year_in_seconds - ttl;
        -- In newer code, TTL is 30 seconds by default
        -- To be safe, don't remove any locks with a TTL less than a day to avoid removing
        -- all locks created after the default TTL was changed
        if (key_type == 'hash' and ttl > one_day_in_seconds and lock_age > 3600) then
           if (ARGV[1] == 'true') then 
              redis.call('DEL', key);
              redis.call('PUBLISH', 'redisson_lock__channel:{' .. key .. '}', 0);
              keys_removed[removed_idx] = 'Removed key: ' .. key;
           else
              keys_removed[removed_idx] = 'Found key: ' .. key;
           end;
           removed_idx = removed_idx + 1;
        end;
    end;
    return keys_removed;
  4. Copy the file to the node that you used in step 1.
  5. Run the script on the node as described below. The script returns a list of keys found that need to be removed.  If you see errors when running the script, verify that the redis pod name and the namespace in the command match your environment.
Note: It is recommended that you first run the script in PREVIEW mode before cleaning up the locks. If no keys are reported in the output, there is no need for clean-up.
Commands to run the script:
  • In PREVIEW mode:
    kubectl -n <namespace> exec -i <redis pod name> -- /usr/local/bin/redis-cli --eval /dev/stdin < script.lua
  • In CLEAN UP mode to clean up the locks:
    kubectl -n <namespace> exec -i <redis pod name> -- /usr/local/bin/redis-cli --eval /dev/stdin , true < script.lua
Where:
  • <redis pod name> is the name of the redis master pod that was determined in step 2 above.
  • <namespace> is the namespace used by Cloud Pak for Data (typically either wkc or zen).
Command examples:

The following commands show how to run the file script.lua using the namespace zen and the redis master pod name redis-ha-server-1:
  • In PREVIEW mode:
    kubectl -n zen exec -i redis-ha-server-1 -- /usr/local/bin/redis-cli --eval /dev/stdin < script.lua
  • In CLEAN UP mode:
    kubectl -n zen exec -i redis-ha-server-1 -- /usr/local/bin/redis-cli --eval /dev/stdin , true < script.lua

Document Location

Worldwide

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSHGYS","label":"IBM Cloud Pak for Data"},"Component":"IBM Watson Knowledge Catalog","Platform":[{"code":"PF040","label":"RedHat OpenShift"}],"Version":"2.5.0","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Product Synonym

InfoSphere Information Governance Catalog; IGC; information assets; default catalog; Watson Knowledge Catalog; WKC; Cloud Pak for Data; CPD; cp4d;

Document Information

Modified date:
09 June 2020

UID

ibm12801163