IBM Support

CHCCLP Tutorials

General Page

This is a summary of the IBM Data Replication Community Wiki CHCCLP Tutorial pages.
Table of Contents 

CHCCLP - Initial Configuration

CHCCLP - Simple Table Mapping

CHCCLP - Multi-user Mode

CHCCLP - Table Mapping Detail

Add Multiple Table mapping to IBM Data Replication using Embedded CHCCLP

Add Multiple Table mapping to IBM Data Replication using Embedded CHCCLP

Collect Performance Stats of Subscriptions from CHCCLP and turn them into usable real time reports

CHCCLP - Initial Configuration

Prerequisite for using this tutorial is that you already have Access Server, Management Console and IIDR CDC Source/Target instance already installed and configured.

We will start with using CHCCLP interactive mode.  This is the simplest mode to utilize, and is used as the first step when you want to build a script for saving.

1. If you do not have an Access Server user yet, create an Access Server user by using dmcreateuser command.

./dmcreateuser -?
Creates a new user.

DMCREATEUSER username fullname description password role manager changePassword passwordNeverExpires [-accessserver hostname port adminuser adminpassword]

  username         Name of the user.
  fullname         Full name for the user.
  description      Description for the user.
  password         Password for the user.
  role             One of SYSADMIN, ADMIN, OPERATOR or MONITOR (case insensitive).
  manager          User has access manager privileges (TRUE/FALSE, case insensitive).
  changePassword   Password must be changed on first login (TRUE/FALSE, case insensitive).
  passwordNeverExpires Password never expires (TRUE/FALSE, case insensitive).

To script user creation using the command line interface, see the command
"add access user" in CHCCLP for more information.

$ ./dmcreateuser admin admin admin ******** SYSADMIN TRUE FALSE TRUE

2. Execute chcclp from <Access_Server>/bin directory.

$ ./chcclp
Command Line Processor for IBM InfoSphere Data Replication 11.3.3 [Build 5258]
(c) Copyright IBM Corporation 2013, 2015

You can execute IBM InfoSphere Data Replication commands from the command
prompt. Use double quotes around parameter values that contain special
characters or syntax keywords, or around empty strings. If the parameter value
contains double quotes, use single quotes around the value. Commands must be
terminated with ; and can span multiple lines.

For help on the available commands or for a specific command, type:

  help;
  help "connect server";
  help "11.3";

To turn on verbose output, type:

  set verbose;

To exit interactive mode, type exit; or quit;

Repl >

3. Connect to Access Server

Repl > connect server hostname localhost port 10101 username admin password ********;

4. Define new Access Server user as follow. This user will be used by CHCCLP. It can be done by dmcreateuser command as Step1.

Repl > add access user name admin_c password ******** manager yes changepassword no;

5. Add DataStores with connection information of your own instances. You can use a different name.

Repl > add access datastore name cdc1 hostname localhost port 10901;
Repl > add access datastore name cdc2 hostname localhost port 10902;

6. Add connection for Access Server user to DataStores

Repl > add access connection username admin_c datastore cdc1 dbuser khjang dbpassword ********;
Repl > add access connection username admin_c datastore cdc2 dbuser khjang dbpassword ********;

7. Disconnect Access Server connection and reconnect by using newly created Access Server user

Repl > disconnect server;
Repl > connect server hostname localhost port 10101 username admin_c password ********;

8. Check connection definition for the Access Server user.

Repl > list access connections username admin_c;

DATASTORE        HOSTNAME         PORT
---------------- ---------------- ----------------
cdc1             localhost        10901
cdc2             localhost        10902

9. Connect to DataStores then check DataStore properties.

Repl > connect datastore name cdc1 context source;
Repl > show datastore name cdc1;

PROPERTY                    VALUE
--------------------------- ---------------------------
Name:                       cdc1
Description:
Host Name:                  localhost
Port:                       10901
Version:                    V11R3M3T2BIIDR_Release_19
Platform:                   Java VM
Database:                   DB2 for LUW
Type:                       Dual
Multi-User Configuration:   No
Table Identification:       Schema (owner), table name

Repl > connect datastore name cdc2 context target;
Repl > show datastore name cdc2;

PROPERTY                    VALUE
--------------------------- ---------------------------
Name:                       cdc2
Description:
Host Name:                  localhost
Port:                       10902
Version:                    V11R3M3T2BIIDR_Release_19
Platform:                   Java VM
Database:                   DB2 for LUW
Type:                       Dual
Multi-User Configuration:   No
Table Identification:       Schema (owner), table name

10. Disconnect and exit chcclp

Repl > disconnect server;
Repl > exit;

Now you are ready to configure replication!

CHCCLP - Simple Table Mapping

The next step is to create a very simple table mapping and test replication.

1. Create a simple table with same name and structure to BOTH source and target database. For example,

CREATE TABLE CLPTEST_01 (COL1 VARCHAR(20) PRIMARY KEY NOT NULL, COL2 VARCHAR(20))

2. Insert a record to source table only. It will be used to validate refresh before mirror.

INSERT INTO CLTPTEST_01 VALUES ('a','a')

3. Execute chcclp then connect to Access Server and both DataStores.

$ ./chcclp
Command Line Processor for IBM InfoSphere Data Replication 11.3.3 [Build 5258]
(c) Copyright IBM Corporation 2013, 2015

You can execute IBM InfoSphere Data Replication commands from the command
prompt. Use double quotes around parameter values that contain special
characters or syntax keywords, or around empty strings. If the parameter value
contains double quotes, use single quotes around the value. Commands must be
terminated with ; and can span multiple lines.

For help on the available commands or for a specific command, type:

  help;
  help "connect server";
  help "11.3";

To turn on verbose output, type:

  set verbose;

To exit interactive mode, type exit; or quit;

Repl > connect server hostname localhost port 10101 username admin_c password ********;
Repl > connect datastore name cdc1 context source;
Repl > connect datastore name cdc2 context target;

4. Create a subscription.

Repl > add subscription name CLPTST01;

NOTICE : First 8 digits of subscription name must be UNIQUE for a source DataStore.

To work with an existing subscription, you need to use select subscription name <NAME>.

Note: You also need to consider Subscription Locking if Multiuser Configuration is enabled in the DataStore. This is covered in a follow on section.

5. Define table mapping

Repl > add table mapping sourceschema KHJANG sourcetable CLPTEST_01 targetschema KHJANG targettable CLPTEST_01;

NOTICE : Schema and Table names are case sensitive. Be careful if your SQL interpreter is not case sensitive.

6. Start subscription and check status. Refresh monitoring result may not

Repl > start mirroring;
Repl > monitor subscription refresh;

Refresh history for subscription CLPTST01. Statistics requested at: Jan 14, 2016
11:40:44 PM

SOURCE TABLE        START TIME                 END TIME                   SOURCE ROWS      APPLIED ROWS
------------------- -------------------------- -------------------------- ---------------- ----------------
KHJANG.CLPTEST_01   Jan 14, 2016 11:40:41 PM   Jan 14, 2016 11:40:42 PM   0                0

Repl > monitor subscription activity;

Activity metrics for subscription CLPTST01 since mirroring started. Statistics
requested at: Jan 14, 2016 11:40:52 PM

METRIC                            TOTAL
--------------------------------- ----------------
Source
  Bytes
    Source database bytes         426
    Source engine bytes           88
    Source communications bytes   558
  Operations
    Source pre-filter inserts     0
    Source pre-filter updates     0
    Source pre-filter deletes     0
    Source post-filter inserts    0
    Source post-filter updates    0
    Source post-filter deletes    0
Target
  Bytes
    Target communications bytes   476
    Target engine bytes           0
    Target database bytes         0
  Operations
    Target "source" inserts       0
    Target "source" updates       0
    Target "source" deletes       0
    Target apply inserts          0
    Target apply updates          0
    Target apply deletes          0

7. Select TARGET table to check result of refresh (initial load by CDC)

SELECT * FROM CLPTEST_01;

COL1                 COL2
-------------------- --------------------
a                    a

  1 record(s) selected.

8. Execute few transactions to SOURCE table.

INSERT INTO CLPTEST_01 VALUES ('b','b');
UPDATE CLPTEST_01 SET COL2='z' WHERE COL1='a';
SELECT * FROM CLPTEST_01;

COL1                 COL2
-------------------- --------------------
a                    z
b                    b

  2 record(s) selected.

9. Check Activity

Repl > monitor subscription activity;

Activity metrics for subscription CLPTST01 since mirroring started. Statistics
requested at: Jan 14, 2016 11:47:50 PM

METRIC                            TOTAL
--------------------------------- ----------------
Source
  Bytes
    Source database bytes         876
    Source engine bytes           2,618
    Source communications bytes   5,144
  Operations
    Source pre-filter inserts     1
    Source pre-filter updates     1

    Source pre-filter deletes     0
    Source post-filter inserts    1
    Source post-filter updates    1
    Source post-filter deletes    0
Target
  Bytes
    Target communications bytes   5,226
    Target engine bytes           17
    Target database bytes         17
  Operations
    Target "source" inserts       1
    Target "source" updates       1
    Target "source" deletes       0
    Target apply inserts          1
    Target apply updates          1
    Target apply deletes          0

10. Select Target table to validate that CDC works fine.

SELECT * FROM CLPTEST_01;

COL1                 COL2
-------------------- --------------------
a                    z
b                    b

  2 record(s) selected.

11. Stop replication and exit chcclp command prompt.

Repl > end replication;
Repl > exit;

CHCCLP - Multi-user Mode

IIDR CDC provides a Multiuser Configuration capability for subscription level editing.   It is applicable at the DataStore level.

When an Access Server user locks a subscription, this subscription can only be edited by the user who locked it.

In other words, subscription edit is enabled for a given user if Multiuser Configuration is disabled or the subscription is locked by the user.

1. Execute chcclp and connect to Access Server by using admin_c user.

Repl > connect server hostname localhost port 10101 username admin_c password ********;

2. Check DataStore properties and modify Multiuser to Yes. Recheck DataStore properties.

Repl > show access datastore name cdc1;

PROPERTY                   VALUE
-------------------------- --------------------------
Name:                      cdc1
Description:
Host Name:                 localhost
Port:                      10901
Version:                   V11R3M3T2BIIDR_Release_19
Platform:                  Java VM
Database:                  JDBC
Type:                      Dual
Multiuser:                 No

Repl > modify access datastore name cdc1 multiuser yes;
Repl > show access datastore name cdc1;

PROPERTY                   VALUE
-------------------------- --------------------------
Name:                      cdc1
Description:
Host Name:                 localhost
Port:                      10901
Version:                   V11R3M3T2BIIDR_Release_19
Platform:                  Java VM
Database:                  JDBC
Type:                      Dual
Multiuser:                 Yes

3. Do the same to the second (target) DataStore.

Repl > modify access datastore name cdc2 multiuser yes;

4. Check if admin user is assigned to DataStores. If not assigned yet assign it.

Repl > list access connections username admin;

DATASTORE        HOSTNAME         PORT
---------------- ---------------- ----------------
cdc1             localhost        10901
cdc2             localhost        10902

If the result is similar to the above, the user is already assigned to the DataStores.

Repl > list access connections username admin;

DATASTORE        HOSTNAME         PORT
---------------- ---------------- ----------------

If result is empty similar as above, user is not assigned to DataStores yet. You need to run following to assign user to DataStore.


Repl > add access connection username admin datastore cdc1 dbuser khjang dbpassword ********;
Repl > add access connection username admin datastore cdc2 dbuser khjang dbpassword ********;

Now both DataStores are available to both admin_mc and admin user.

6. Connect to DataStores and check subscription. It's not locked. Now try to modify description of the subscription. Your attempt will fail because it's not locked by admin_c user.

Repl > connect datastore name cdc1 context source;
Repl > connect datastore name cdc2 context target;
Repl > select subscription name CLPTST01;
Repl > show subscription;

PROPERTY                   VALUE
-------------------------- --------------------------
Name:                      CLPTST01
Description:
Source Datastore:          cdc1
Target Datastore:          cdc2
Source ID:                 CLPTST01
TCP Host:                  Auto Select
Firewall Port:
Persistency:               No
Transferable Work:         Target
Propagation Control:
Maintain History:          Target
Minimize Network Load:     No
Locked By User:
Repl > modify subscription description "Lock Test";
[ERR2317]: The subscription must be locked before the command can execute.

7. Lock the subscription and try to modify it. Now it works because the subscription is locked by admin_c

Repl > lock subscription;
Repl > modify subscription description "Lock Test";
Repl > show subscription;

PROPERTY                   VALUE
-------------------------- --------------------------
Name:                      CLPTST01
Description:               Lock Test
Source Datastore:          cdc1
Target Datastore:          cdc2
Source ID:                 CLPTST01
TCP Host:                  Auto Select
Firewall Port:
Persistency:               No
Transferable Work:         Target
Propagation Control:
Maintain History:          Target
Minimize Network Load:     No
Locked By User:            admin_c

In general, the Access Server user who locks a subscription can unlock it after modification.  Additionally, an Access Server user with System Administrator Role can force unlock a subscription that is locked by other user. Let's try it in the next steps:

8. Exit and restart chcclp then connect to Access Server by using admin user. Connect to DataStores also.

Repl > exit;

Repl > connect server hostname localhost port 10101 username admin password ********;
Repl > connect datastore name cdc1 context source;
Repl > connect datastore name cdc2 context target;

9. Try to unlock the subscription and check it. It works because admin user has System Administrator role.

Repl > select subscription name CLPTST01;
Repl > unlock subscription;
Repl > show subscription;

PROPERTY                   VALUE
-------------------------- --------------------------
Name:                      CLPTST01
Description:               Lock Test
Source Datastore:          cdc1
Target Datastore:          cdc2
Source ID:                 CLPTST01
TCP Host:                  Auto Select
Firewall Port:
Persistency:               No
Transferable Work:         Target
Propagation Control:
Maintain History:          Target
Minimize Network Load:     No
Locked By User:
Repl > show access user name admin;

PROPERTY                   VALUE
-------------------------- --------------------------

INFORMATION
  Name:                    admin
  Full Name:               admin
  Description:             admin

ACCESS
  Role:                    System Administrator
  Manager:                 Yes

STATUS
  Account Disabled:        No
  Account Locked:          No
  Password Change:         No
  Password Expires:        No

10. Now any user can lock the subscription for edit.

Step 9 ~ 10 is only to show FORCE unlock capability by System Administrator user.

11 Modify Multiuser to false and check "Locked By User" is removed from result of "show subscription".

Repl > modify access datastore name cdc1 multiuser false;
Repl > modify access datastore name cdc2 multiuser false;
Repl > connect datastore name cdc1 context source;
Repl > connect datastore name cdc2 context target;
Repl > select subscription name CLPTST01;
Repl > show subscription;

PROPERTY                   VALUE
-------------------------- --------------------------
Name:                      CLPTST01
Description:               Lock Test
Source Datastore:          cdc1
Target Datastore:          cdc2
Source ID:                 CLPTST01
TCP Host:                  Auto Select
Firewall Port:
Persistency:               No
Transferable Work:         Target
Propagation Control:
Maintain History:          Target
Minimize Network Load:     No

CHCCLP - Table Mapping Detail

IIDR CDC provides various customization of table mappings.   The following mapping customizations will be covered in this section:


Column Mapping

1. Create a simple table in the SOURCE database. For example,

CREATE TABLE CLPTEST_04 (COL1 VARCHAR(20) PRIMARY KEY NOT NULL, COL2 VARCHAR(20));

2. Create a simple table in the TARGET database with at least one addition varchar(20) column. For example,

CREATE TABLE CLPTEST_04 (COL1 VARCHAR(20) PRIMARY KEY NOT NULL, COL2 VARCHAR(20), COL3 VARCHAR(20));

3. Create CHILD table in the SOURCE database then insert a record.

CREATE TABLE CHILD (ID VARCHAR(20) PRIMARY KEY NOT NULL, PCOL1 VARCHAR(20), RESULT VARCHAR(20));

INSERT INTO CHILD VALUES ('1','a','SOURCE');

4. Execute chcclp command then create new subscription and create table mapping as follows:

Repl > connect server hostname localhost port 10101 username admin_c password ********;
Repl > connect datastore name cdc1 context source;
Repl > connect datastore name cdc2 context target;
Repl > add subscription name CLPTST04;
Repl > add table mapping sourceSchema KHJANG sourceTable CLPTEST_04 targetSchema KHJANG targetTable CLPTEST_04 ;
Repl > list column mappings;

Column mappings for table mapping KHJANG.CLPTEST_04 - KHJANG.CLPTEST_04.

SOURCE           TARGET COLUMN    INITIAL VALUE
---------------- ---------------- ----------------
COL1             COL1
COL2             COL2
                 COL3             NULL

NOTICE 1 : Reserved words are case-insensitive but all others such as name of db objects and  name of subscription or mapping are case-sensitive

NOTICE 2 : If you exit  and re-execute chcclp after this step,  you will need to select subscription and table as follows:

Repl > connect server hostname localhost port 10101 username admin_c password ********;
Repl > connect datastore name cdc1 context source;
Repl > connect datastore name cdc2 context target;
Repl > select subscription name CLPTST04;
Repl > select table mapping sourceTable CLPTEST_04;

5. Execute chcclp command as follows and modify parameters appropriately based on your environment.

%GETCOL function is used to select additional data from other tables based on defined key columns.

It will return value of RESULT column in CHILD table where POCL1 in CHILD is same as value OF COL1 with default value is NULL.

Make sure to modify expression to your own SOURCE database schema name instead of using KHJANG.

Repl > add derived column name D_COL3 datatype VARCHAR length 20

expression "%GETCOL(RESULT,'KHJANG.CHILD',NULL,PCOL1,COL1)";

Repl > map column targetColumn COL3 value D_COL3;

Repl > list column mappings;

Column mappings for table mapping KHJANG.CLPTEST_04 - KHJANG.CLPTEST_04.

SOURCE           TARGET COLUMN    INITIAL VALUE
---------------- ---------------- ----------------
COL1             COL1
COL2             COL2
D_COL3           COL3

Repl > show derived column name D_COL3;

PROPERTY                   VALUE
-------------------------- -----------------------------------------------
Name:                      D_COL3
Description:
Data Type:                 VARCHAR
Length:                    20
Scale:                     0
Nullable:                  false
Evaluation Frequency:      after
Expression:                %GETCOL(RESULT,'KHJANG.CHILD',NULL,PCOL1,COL1)

Now column mapping for COL3 from Derived Column D_COL3 is added.

6. Now start mirroring and check subscription status;

Repl > start mirroring;
Repl > monitor replication filter subscription name CLPTST04;

Monitors the replication state and latency of subscriptions. Statistics
requested at: Jan 28, 2016 4:03:58 AM

SUBSCRIPTION     STATE            SCHEDULED END    LATENCY THRESHOLD   SOURCE DATASTORE   TARGET DATASTORE
---------------- ---------------- ---------------- ------------------- ------------------ ------------------
CLPTST04         Mirror Continuous                                     cdc1               cdc2

Status can be "Starting" instead of "Mirror Continuous" because "start mirroring" is run asynchronously.

7. Insert few records to SOURCE table

INSERT INTO CLPTEST_04 VALUES ('a','a');

INSERT INTO CLPTEST_04 VALUES ('b','b');

8. Select TARGET table. You can see result of %GETCOL. Remember that the %GETCOL result came from a query of a SOURCE table.

SELECT * FROM CLPTEST_04;

COL1                 COL2                 COL3
-------------------- -------------------- ------------------------------
a                    a                    SOURCE
b                    b                    -

  2 record(s) selected.

9. End replication.

Repl > end replication;

Repl > monitor replication filter subscription name CLPTST04;

Monitors the replication state and latency of subscriptions. Statistics
requested at: Jan 28, 2016 9:45:52 PM

SUBSCRIPTION     STATE            SCHEDULED END    LATENCY THRESHOLD   SOURCE DATASTORE   TARGET DATASTORE
---------------- ---------------- ---------------- ------------------- ------------------ ------------------
CLPTST04         Inactive                                              cdc1               cdc2

10. Create CHILD table in TARGET database then insert a record.

CREATE TABLE CHILD (ID VARCHAR(20) PRIMARY KEY NOT NULL, PCOL1 VARCHAR(20), RESULT VARCHAR(20));

INSERT INTO CHILD VALUES ('1','C','TARGET');

11. Map COL3 again by using expression. Make sure to change your own database schema name of TARGET instead  of KHJANG.

Repl > select table mapping sourceTable CLPTEST_04;
Repl > map column targetColumn COL3 type expression
value "%GETCOL(RESULT,'KHJANG.CHILD',NULL,PCOL1,COL1)";
Repl > list column mappings;

Column mappings for table mapping KHJANG.CLPTEST_04 - KHJANG.CLPTEST_04.

SOURCE                                           TARGET COLUMN    INITIAL VALUE
------------------------------------------------ ---------------- ----------------
COL1                                             COL1
COL2                                             COL2
%GETCOL(RESULT,'KHJANG.CHILD',NULL,PCOL1,COL1)   COL3

12. Now start mirroring and check subscription status

Repl > start mirroring;
Repl > monitor replication filter subscription name CLPTST04;

Monitors the replication state and latency of subscriptions. Statistics
requested at: Jan 28, 2016 9:54:36 AM

SUBSCRIPTION     STATE            SCHEDULED END    LATENCY THRESHOLD   SOURCE DATASTORE   TARGET DATASTORE
---------------- ---------------- ---------------- ------------------- ------------------ ------------------
CLPTST04         Mirror Continuous                                     cdc1               cdc2

Status can be "Starting" instead of "Mirror Continuous" because "start mirroring" is run asynchronously.

13. Insert few records to SOURCE table

INSERT INTO CLPTEST_04 VALUES ('c','c');

INSERT INTO CLPTEST_04 VALUES ('d','d');

14. Select target table. You can see result of %GETCOL. Remember that it's come from TARGET table.

SELECT * FROM CLPTEST_04;

COL1                 COL2                 COL3
-------------------- -------------------- ------------------------------
a                    a                    SOURCE
b                    b                    -
c                    c                    TARGET
d                    d                    -

  4 record(s) selected.

As tested, Derived Column is run from SOURCE but Expression is run from TARGET.

Please reference Knowledge Center to learn more detailed about Derived Expressions.

15. Delete all records from SOURCE table. Records in TARGET table will be deleted by CDC also.

DELETE FROM CLPTEST_04;

16. End replication.

Repl > end replication;

Repl > monitor replication filter subscription name CLPTST04;

Monitors the replication state and latency of subscriptions. Statistics
requested at: Jan 28, 2016 9:55:52 PM

SUBSCRIPTION     STATE            SCHEDULED END    LATENCY THRESHOLD   SOURCE DATASTORE   TARGET DATASTORE
---------------- ---------------- ---------------- ------------------- ------------------ ------------------
CLPTST04         Inactive                                              cdc1               cdc2

Filtering


17. If you had exited chcclp, execute chcclp again then do the following.

Repl > connect server hostname localhost port 10101 username admin_c password ********;
Repl > connect datastore name cdc1 context source;
Repl > connect datastore name cdc2 context target;
Repl > select subscription name CLPTST04;

18. Check context and select table mapping if it's not selected

Repl > show context;

CONTEXT            CURRENT OBJECT
------------------ -----------------
Access Server      localhost@10101
Source Datastore   cdc1
Target Datastore   cdc2
Subscription       CLPTST04

Repl > select table mapping sourceTable CLPTEST_04;

Repl > show context;

CONTEXT            CURRENT OBJECT
------------------ ---------------------------------------
Access Server      localhost@10101
Source Datastore   cdc1
Target Datastore   cdc2
Subscription       CLPTST04
Table Mapping      KHJANG.CLPTEST_04 - KHJANG.CLPTEST_04

18. Add row filtering the check table mapping detail.

Repl > modify table mapping rowFilter "COL2 = 'Z'" select false;
Repl > show table mapping;

PROPERTY                      VALUE
----------------------------- --------------------------
Source Table:                 KHJANG.CLPTEST_04
Target Table:                 KHJANG.CLPTEST_04
Mapping Type:                 Standard
Method:                       Mirror
Prevent Recursion:            No
Status:                       Active

TARGET KEY
  Index Mode:                 Auto Detect

REFRESH
  Row Subset Refresh:
  Source WHERE clause:
  Target WHERE clause:

ROW-FILTERING
  Row-filtering Expression:   COL2 = 'Z'
  Select/Omit Rows:           omit

CONFLICTS
  Conflict Detection Columns:
  Conflict Resolution Method: None
  Value Comparison Column:
  User Exit (with Path):

With this filtering configuration, records that are matching this condition will be not replicated to target.

19. Now start mirroring and check subscription status

Repl > start mirroring;
Repl > monitor replication filter subscription name CLPTST04;

Monitors the replication state and latency of subscriptions. Statistics
requested at: Jan 31, 2016 11:29:16 PM

SUBSCRIPTION     STATE               SCHEDULED END    LATENCY THRESHOLD   SOURCE DATASTORE   TARGET DATASTORE
---------------- ------------------- ---------------- ------------------- ------------------ ------------------
CLPTST04         Mirror Continuous                                        cdc1               cdc2            


20. Insert following records to the SOURCE table

INSERT INTO CLPTEST_04 VALUES ('1','A');

INSERT INTO CLPTEST_04 VALUES ('2','Z');

21. Check records from the TARGET table. Check record with COL2='Z' is not replicated to TARGET

SELECT * FROM CLPTEST_04;

COL1                 COL2                 COL3
-------------------- -------------------- ------------------------------
1                    A                    -

  1 record(s) selected.

22. Update a record from the SOURCE table. Now this record will be inserted (instead of update) to TARGET.

UPDATE CLPTEST_04 SET COL2='B' WHERE COL1='2';

23. Check record from the TARGET table. Now you can see record with COL1='2' that was filtered at Step 6.

SELECT * FROM CLPTEST_04;

COL1                 COL2                 COL3
-------------------- -------------------- ------------------------------
1                    A                    -
2                    B                    -

  2 record(s) selected.

24. Delete all records from SOURCE table. Records in TARGET table will be deleted by CDC also.

DELETE FROM CLPTEST_04;

25. End replication.

Repl > end replication;

Repl > monitor replication filter subscription name CLPTST04;

Monitors the replication state and latency of subscriptions. Statistics

requested at: Jan 31, 2016 11:32:43 PM

SUBSCRIPTION     STATE            SCHEDULED END    LATENCY THRESHOLD   SOURCE DATASTORE   TARGET DATASTORE

---------------- ---------------- ---------------- ------------------- ------------------ ------------------

CLPTST04         Inactive                                              cdc1               cdc2

Translation

26. If you exit chcclp, execute chcclp again then do following.

Repl > connect server hostname localhost port 10101 username admin_c password ********;
Repl > connect datastore name cdc1 context source;
Repl > connect datastore name cdc2 context target;
Repl > select subscription name CLPTST04;

27. Check context and select table mapping if it's not selected.

Repl > show context;

CONTEXT            CURRENT OBJECT
------------------ -----------------
Access Server      localhost@10101
Source Datastore   cdc1
Target Datastore   cdc2
Subscription       CLPTST04

Repl > select table mapping sourceTable CLPTEST_04;

Repl > show context;

CONTEXT            CURRENT OBJECT
------------------ ---------------------------------------
Access Server      localhost@10101
Source Datastore   cdc1
Target Datastore   cdc2
Subscription       CLPTST04
Table Mapping      KHJANG.CLPTEST_04 - KHJANG.CLPTEST_04

28. Add data translation for COL2 as follow. It will translate M to 1 and F to 2 for COL2.

Repl > add data translation targetColumn COL2 before M after 1;
Repl > add data translation targetColumn COL2 before F after 2;
Repl > show data translation;

Data translation for table mapping KHJANG.CLPTEST_04 - KHJANG.CLPTEST_04.

TARGET COLUMN    BEFORE           AFTER
---------------- ---------------- ----------------
COL2             M                1
COL2             F                2
 

29. Now start mirroring and check subscription status

Repl > start mirroring;
Repl > monitor replication filter subscription name CLPTST04;

Monitors the replication state and latency of subscriptions. Statistics
requested at:  Jan 31, 2016 11:41:58 PM

SUBSCRIPTION     STATE               SCHEDULED END    LATENCY THRESHOLD   SOURCE DATASTORE   TARGET DATASTORE
---------------- ------------------- ---------------- ------------------- ------------------ ------------------
CLPTST04         Mirror Continuous                                        cdc1               cdc2            

30. Insert following records to SOURCE table

INSERT INTO CLPTEST_04 VALUES ('1','M');

INSERT INTO CLPTEST_04 VALUES ('2','F');

INSERT INTO CLPTEST_04 VALUES ('3','A');

31. Check records from TARGET table. Check translation is worked as defined.

SELECT * FROM CLPTEST_04;

COL1                 COL2                 COL3
-------------------- -------------------- ------------------------------
3                    A                    -
1                    1                    -
2                    2                    -

32. Delete all records from SOURCE table. Records in TARGET table will be deleted by CDC also.

DELETE FROM CLPTEST_04;

Conflicts
 

33. To test default behavior about data conflicts, delete all data then  insert a record to TARGET table first.

INSERT INTO CLPTEST_04 VALUES ('1','A',NULL);

34. Now insert same record to SOURCE table. Subscription will be Failed with error.

INSERT INTO CLPTEST_04 VALUES ('1','A');

35. Check status of subscription. Status may changed to Failed.

Repl > monitor replication filter subscription name CLPTST04;

Monitors the replication state and latency of subscriptions. Statistics
requested at: Jan 31, 2016 11:51:05 PM

SUBSCRIPTION     STATE            SCHEDULED END    LATENCY THRESHOLD   SOURCE DATASTORE   TARGET DATASTORE
---------------- ---------------- ---------------- ------------------- ------------------ ------------------
CLPTST04         Failed                                                cdc1               cdc2

36. Now try to check subscription level event logs for source first.

Repl > list subscription events type source count 10;

ROW   EVENT ID   TYPE          TIME                       OBJECT     MESSAGE
----- ---------- ------------- -------------------------- ---------- --------------------------------------------------------------------------------------------------------------------------
1     93         Information   Jan 31, 2016 11:48:15 PM   CLPTST04   *** IBM InfoSphere Data Replication is ending.
2     1465       Error         Jan 31, 2016 11:48:15 PM   CLPTST04   --- Subscription CLPTST04 is terminating abnormally.
3     1714       Error         Jan 31, 2016 11:48:15 PM   CLPTST04   IBM InfoSphere Data Replication to CLPTST04 is initiating shutdown due to failure on the target system. See the previ...
4     90         Error         Jan 31, 2016 11:48:14 PM   CLPTST04   IBM InfoSphere Data Replication terminating due to target system failure.
5     2922       Information   Jan 31, 2016 11:41:55 PM   CLPTST04   Subscription CLPTST04 has started using the single scrape staging store. Subscription bookmark: Journal name 0 Journa...
6     44         Information   Jan 31, 2016 11:41:55 PM   CLPTST04   Mirroring has been initiated for table KHJANG.CLPTEST_04.
7     1521       Information   Jan 31, 2016 11:41:54 PM   CLPTST04   Communication to CLPTST04 on target host localhost is starting on the Data channel.
8     9750       Information   Jan 31, 2016 11:41:54 PM   CLPTST04   Comms connected to the target host: localhost port: 10,902 successfully.
9     1463       Information   Jan 31, 2016 11:41:54 PM   CLPTST04   +++ Subscription CLPTST04 is starting in Continuous Mirroring mode.
10    1561       Information   Jan 31, 2016 11:41:54 PM   CLPTST04   The target reported time zone America/New_York.  IBM InfoSphere Data Replication will use source time zone America/Ne...

37. Source event log said that replication was failed by target system failure. Now check subscription level event logs for target.

Repl > list subscription events type target count 10;

ROW   EVENT ID   TYPE          TIME                       OBJECT     MESSAGE
----- ---------- ------------- -------------------------- ---------- --------------------------------------------------------------------------------------------------------------------------
1     95         Information   Jan 31, 2016 11:48:14 PM   CLPTST04   IBM InfoSphere Data Replication Communications are ending.
2     1714       Error         Jan 31, 2016 11:48:14 PM   CLPTST04   IBM InfoSphere Data Replication to CLPTST04 is initiating shutdown due to failure on the target system. See the previ...
3     1475       Error         Jan 31, 2016 11:48:12 PM   CLPTST04   Subscription CLPTST04 is terminating abnormally.
4     315        Error         Jan 31, 2016 11:48:12 PM   CLPTST04   Failed to add a row to table KHJANG.CLPTEST_04. COL1: [1] .  Database operation failed.  A SQL exception has occurred...
5     6673       Information   Jan 31, 2016 11:41:57 PM   CLPTST04   IBM InfoSphere Data Replication will commit on source transaction boundaries.
6     1473       Information   Jan 31, 2016 11:41:54 PM   CLPTST04   Subscription CLPTST04 is starting in Continuous Mirroring mode.
7     1531       Information   Jan 31, 2016 11:41:54 PM   CLPTST04   Communication with CLPTST04 was successfully started on the Data channel.
8     1560       Information   Jan 31, 2016 11:41:54 PM   CLPTST04   The source reported time zone America/New_York.  IBM InfoSphere Data Replication will use source time zone America/Ne...
9     94         Information   Jan 31, 2016 11:41:54 PM   CLPTST04   +++ IBM InfoSphere Data Replication was initiated by source system CLPTST04.
10    1532       Information   Jan 31, 2016 11:41:53 PM   CLPTST04   Sign on from CLPTST04 was successful.

38. Root cause of failure looks like described in ROW 4. Check detail event of row 4.

Repl > show subscription event details row 4;

EVENT ID:           315
TYPE:               Error
TIME:               Jan 31, 2016 11:48:12 PM

MESSAGE:
Failed to add a row to table KHJANG.CLPTEST_04. COL1: [1] .
Database operation failed.
A SQL exception has occurred. The SQL error code is '-803'. The SQL state is: 23505.
The error message is:
One or more values in the INSERT statement, UPDATE statement, or foreign key update
caused by a DELETE statement are not valid because the primary key, unique constraint
or unique index identified by "1" constrains table "KHJANG.CLPTEST_04" from having
duplicate values for the index key.. SQLCODE=-803, SQLSTATE=23505, DRIVER=4.19.26

Event messages are categorized with 5 different area.

  • Subscription (Source/Target) - Subscription level event. Event related with table level replication is also stored in here.
  • Single Scrape (Source Only) - Scraper event.
  • Datastore (Source/Target) - Datastore(instance, dmts64 process) level event.

NOTICE : It's important to check the event log using the right category to identify the root cause of a replication error.

39. If you exit chcclp, execute chcclp again then do following.

Repl > connect server hostname localhost port 10101 username admin_c password ********;
Repl > connect datastore name cdc1 context source;
Repl > connect datastore name cdc2 context target;
Repl > select subscription name CLPTST04;

40. Check context and select table mapping if it's not selected.

Repl > show context;

CONTEXT            CURRENT OBJECT
------------------ -----------------
Access Server      localhost@10101
Source Datastore   cdc1
Target Datastore   cdc2
Subscription       CLPTST04

Repl > select table mapping sourceTable CLPTEST_04;

Repl > show context;

CONTEXT            CURRENT OBJECT
------------------ ---------------------------------------
Access Server      localhost@10101
Source Datastore   cdc1
Target Datastore   cdc2
Subscription       CLPTST04
Table Mapping      KHJANG.CLPTEST_04 - KHJANG.CLPTEST_04

41. Now configure Conflict Resolution as source win and check table mapping detail

Repl > modify table mapping detectionColumns COL1, resolutionMethod sourceWins;
Repl > show table mapping;

PROPERTY                      VALUE
----------------------------- --------------------------
Source Table:                 KHJANG.CLPTEST_04
Target Table:                 KHJANG.CLPTEST_04
Mapping Type:                 Standard
Method:                       Mirror
Prevent Recursion:            No
Status:                       Active

TARGET KEY
  Index Mode:                 Auto Detect

REFRESH
  Row Subset Refresh:
  Source WHERE clause:
  Target WHERE clause:

ROW-FILTERING
  Row-filtering Expression:   COL2 = 'Z'
  Select/Omit Rows:           omit

CONFLICTS
  Conflict Detection Columns: COL1
  Conflict Resolution Method: Source Wins
  Value Comparison Column:
  User Exit (with Path):

42. Now start mirroring and check subscription status

Repl > start mirroring;
Repl > monitor replication filter subscription name CLPTST04;

Monitors the replication state and latency of subscriptions. Statistics
requested at: Feb 1, 2016 1:06:03 AM

SUBSCRIPTION     STATE               SCHEDULED END    LATENCY THRESHOLD   SOURCE DATASTORE   TARGET DATASTORE
---------------- ------------------- ---------------- ------------------- ------------------ ------------------
CLPTST04         Mirror Continuous                                        cdc1               cdc2            

48. Insert a record to TARGET

INSERT INTO CLPTEST_04 VALUES ('2','B',NULL);

49. Insert a record to SOURCE

INSERT INTO CLPTEST_04 VALUES ('2','A');

50. Select TARGET table and validate Conflict Resolution is works as Source Wins.

SELECT * FROM CLPTEST_04;

COL1                 COL2                 COL3
-------------------- -------------------- ------------------------------
2                    A                    -
1                    A                    -

51. Optionally check TS_CONFAUD table from TARGET. Audit records of Conflict Resolution are stored in this table. 2 records are in there with previous operations.

Operation

52. End replication.

Repl > end replication;

Repl > monitor replication filter subscription name CLPTST04;

Monitors the replication state and latency of subscriptions. Statistics

requested at: Feb 1, 2016 1:16:23 AM

SUBSCRIPTION     STATE            SCHEDULED END    LATENCY THRESHOLD   SOURCE DATASTORE   TARGET DATASTORE

---------------- ---------------- ---------------- ------------------- ------------------ ------------------

CLPTST04         Inactive                                              cdc1               cdc2

53. Check context and select table mapping if it's not selected.

Repl > show context;

CONTEXT            CURRENT OBJECT
------------------ -----------------
Access Server      localhost@10101
Source Datastore   cdc1
Target Datastore   cdc2
Subscription       CLPTST04

Repl > select table mapping sourceTable CLPTEST_04;

Repl > show context;

CONTEXT            CURRENT OBJECT
------------------ ---------------------------------------
Access Server      localhost@10101
Source Datastore   cdc1
Target Datastore   cdc2
Subscription       CLPTST04
Table Mapping      KHJANG.CLPTEST_04 - KHJANG.CLPTEST_04

53. Configuration operation for delete to do not delete.

Repl > modify operations onDelete doNotDelete;
Repl > show operations;

Row and table-level operations for table mapping KHJANG.CLPTEST_04 - KHJANG.CLPTEST_04.

PROPERTY                   VALUE
-------------------------- --------------------------

ROW-LEVEL OPERATIONS
  On Insert:               Insert Row
  On Update:               Update Row
  On Delete:               Do Not Delete

TABLE-LEVEL OPERATIONS
  On Clear/Truncate:       Delete All

NOTICE : Available options can be different by mapping type or target agent. Try help "show modify operations"; to check available operations choice per mapping type.

54. Now start mirroring and check subscription status

Repl > start mirroring;
Repl > monitor replication filter subscription name CLPTST04;

Monitors the replication state and latency of subscriptions. Statistics
requested at: Feb 1, 2016 1:23:24 AM

SUBSCRIPTION     STATE               SCHEDULED END    LATENCY THRESHOLD   SOURCE DATASTORE   TARGET DATASTORE
---------------- ------------------- ---------------- ------------------- ------------------ ------------------
CLPTST04         Mirror Continuous                                        cdc1               cdc2            

55. Run few SQLs include delete one.

56. Validate DELETE operation is not replicated to TARGET.

57. Stop subscription and delete it (optional)

Now try to configure all configuration in this section by using CHCCLP batch mode. Values within %% are replaced by value of <NAME>:<VALUE> format argument.

58. Download chcclp04.txt file and execute as follow. This script will create new subscription CLPBAT04. Must use FULL PATH to file name.

$ ./chcclp -f /script/chcclp04.txt HOSTNAME:localhost PORT:10101 USERNAME:admin_c PASSWORD:********
Repl > chcclp session set to cdc;
Repl > connect server hostname localhost password ******** port 10101 username admin_c;
Repl > connect datastore name cdc1 context source;
Repl > connect datastore name cdc2 context target;
Repl > add subscription name CLPBAT04;
Repl > add table mapping targetSchema KHJANG targetTable CLPTEST_04 sourceTable CLPTEST_04 sourceSchema KHJANG                                           ;
Repl > map column targetColumn COL3 type expression value "%GETCOL(RESULT,'KHJANG.CHILD',NULL,PCOL1,COL1)";
Repl > modify table mapping select false rowFilter "COL2 = 'Z'";
Repl > modify table mapping detectionColumns COL1 resolutionMethod sourceWins;
Repl > add data translation before M targetColumn COL2 after 1;
Repl > add data translation before F targetColumn COL2 after 2;
Repl > show table mapping;

PROPERTY                      VALUE
----------------------------- --------------------------
Source Table:                 KHJANG.CLPTEST_04
Target Table:                 KHJANG.CLPTEST_04
Mapping Type:                 Standard
Method:                       Mirror
Prevent Recursion:            No
Status:                       Refresh

TARGET KEY
  Index Mode:                 Auto Detect

REFRESH
  Row Subset Refresh:
  Source WHERE clause:
  Target WHERE clause:

ROW-FILTERING
  Row-filtering Expression:   COL2 = 'Z'
  Select/Omit Rows:           omit

CONFLICTS
  Conflict Detection Columns: COL1
  Conflict Resolution Method: Source Wins
  Value Comparison Column:
  User Exit (with Path):

Repl > list column mappings;

Column mappings for table mapping KHJANG.CLPTEST_04 - KHJANG.CLPTEST_04.

SOURCE                                           TARGET COLUMN    INITIAL VALUE
------------------------------------------------ ---------------- ----------------
COL1                                             COL1
COL2                                             COL2
%GETCOL(RESULT,'KHJANG.CHILD',NULL,PCOL1,COL1)   COL3

Repl > show data translation;

Data translation for table mapping KHJANG.CLPTEST_04 - KHJANG.CLPTEST_04.

TARGET COLUMN    BEFORE           AFTER
---------------- ---------------- ----------------
COL2             M                1
COL2             F                2

Repl > modify operations onDelete doNotDelete;
Repl > show operations;

Row and table-level operations for table mapping KHJANG.CLPTEST_04 - KHJANG.CLPTEST_04.

PROPERTY                   VALUE
-------------------------- --------------------------

ROW-LEVEL OPERATIONS
  On Insert:               Insert Row
  On Update:               Update Row
  On Delete:               Do Not Delete

TABLE-LEVEL OPERATIONS
  On Clear/Truncate:       Delete All

Repl > disconnect server;
Repl > exit;

Add Multiple Table mapping to IBM Data Replication using Embedded CHCCLP

Introduction

CHCCLP which is the command line interface for IBM Data Replication Change Data Capture starting version 10.2 allows users to perform operations that users used to perform in Management Console GUI. It provides rich set of commands to manage replication and perform operations like add table mapping, perform access server functions , monitor replication etc. User's can use CHCCLP in three different ways, 1. Interactive Mode 2. Batch Mode and 3. Embedded Mode.

Interactive mode allows users to perform one operation at a time and cannot be used for handling multiple operations in single batch, where as Batch mode and embedded mode allows users to perform operations in complex loops and in batch mode. If you are a JAVA programmer or some one who knows JAVA then you can use CHCCLP in Embedded mode . With the sophistication of programmable interface and simplicity of CHCCLP commands it helps users to perform complex operations in Data Replication. The below example shows list of commands to add a table mapping in interactive mode and subsequent section shows how to perform multiple table mappings in Embedded mode.

Add a table mapping in interactive mode

1. Initialize the chcclp environment from <Management Console_HOME>/bin/chcclp.bat

2. Once into the CHCCLP mode you can execute the below set of commands to add a table mapping

connect server hostname <hostname> port <portno> username <username> password <password>;

connect datastore name <source_datastore> context source;

connect datastore name <target_datastore> context target;

add subscription name <subscription_name>;   # remove this line in case you want to add an existing subscription

select subscription name <subscription_name>;

add table mapping <source schema> <source table> <target schema> <target table>;

....      # repeat add table mapping for every table mapping that you want to add to the subscription

disconnect server;

3. For getting help on syntax  type help to list all available functions and type help "function name" for sample example

As you might have noticed in the sample above, it is a tedious process to add multiple table mapping when there are large number of tables to be mapped. This is when Batch mode and Embedded mode comes for the rescue. The below section shows some of the basics on how to use CHCCLP in Embedded mode and how to write JAVA code for Adding multiple table mapping and get subscription details and their latency information.

Using CHCCLP in Embedded Mode

In either of the three modes of using CHCCLP, the basic syntax of the commands remains the same. Its just how we use them along with other scripting or programmable logic helps us to achieve greater flexibility.

Embedded mode of executing CHCCLP is a way to run CHCCLP on your JAVA environment which has access to MC and Access Server. The basic steps to setup an embedded mode environment for CHCCLP are ( Eclipse IDE is used as JAVA environment )

1. Create a JAVA Project in eclipse

image

2. Configure the build path of the project and add jar files . At the bare minimum you would require to add api.jar and chcclp.jar which can be obtained from <Access Server_HOME>/lib directory. You may need to add other jar files from this directory depending on your program. The picture on the left side shows Project called EmbeddedCHCCLP and its Referenced Libraries

3. After the setup of the Project , add a class that will run the CHCCLP in embedded mode. The above picture shows a JAVA Class with name EmbeddedCHCCLP

4. To execute the CHCCLP scripts within the JAVA code , we will use object of Class EmbeddedScript, and to use it add the below imports

import com.ibm.replication.cdc.scripting.EmbeddedScript;
import com.ibm.replication.cdc.scripting.EmbeddedScriptException;

Example :

 EmbeddedScript script = new EmbeddedScript();

Result result = new Result();

try

{

 script.open();

script.execute("connect server hostname <hostname> port <portno> username <username> password <password>");

}

catch (EmbeddedScriptException e)
          {
             System.out.println(e.getResultCodeAndMessage());
          }

      5. Each script execution returns result of type Result, and there are different types of results that are returned. Depending on the type of result you can get the values printed on to the console.

Example: Commands like list subscriptions, monitor subscription filter datastore returns result of type ResultStringTable

To print first column of all the rows returned in the result

rowCount=((ResultStringTable)result).getRowCount();
                if (rowCount > 0) {            
                
                    for(int i=0;i<rowCount;i++) {
                        System.out.println(((ResultStringTable)result).getValueAt(i, 0));
                    }
                }

Add multiple table mapping using Embedded Mode

The below JAVA program uses a property file from which all the details for mapping are retrieved . Programming constructs are used for looping , value retrieving logic and CHCCLP commands are used to perform the core functionality of adding the table mapping.

config.properties

AS_HOSTNAME=localhost
AS_PORT=10101
AS_USERNAME=Demo
AS_PASSWORD=demo123
TOTAL_GROUPS=2 # denotes number of subscriptions to be created. Define one group for each subscription
#SOURCE DATASTORE ,TARGET DATASTORE,SOURCESCHEMA,TARGETSCHEMA,SUBSCRIPTION NAME
GROUP1=DEMO_SOURCE,DEMO_TARGET,CDCSOURCE,CDCTARGET,DEMOSUB1  
GROUP2=DEMO_SOURCE,DEMO_TARGET,CDCSOURCE,CDCTARGET,DEMOSUB2
#SUBSCRIPTIONNAME = SOURCE_TABLE_NAME,TARGET_TABLE_NAME|SOURCE_TABLE_NAME,TARGET_TABLE_NAME ...
DEMOSUB1=SRC1,TGT1|SRC2,TGT2|SRC3,TGT3
DEMOSUB2=SRC4,TGT4 |SRC5,TGT5

Sample output

Table Mapping Completed
List of subscriptions under DEMO_SOURCE - DEMO_TARGET and their latency details

===============================================================================

DEMO_SUB1
DEMOSUB1
DEMOSUB2
EMBED_SUB1
SUB1
Latency information displayed only for mirroring subscriptions

SUBSCRIPTION       LATECNY THRESHOLD       LATENCY(sec)

----------------------------------------------------

DEMOSUB1                                        8
EMBED_SUB1                                      5

Download complete JAVA code here

Collect Performance Stats of Subscriptions from CHCCLP and turn them into usable real time reports

Amit Kumar | Aug 26 2014 ‎ | 10,458 Views

In my previous blog entry, I had shown how to use CHCCLP (Command line scripting interface for Management Console) in embedded mode to perform complex mapping. This section shows how to collect Performance Statistics from CHCCLP in real time and persist them to database tables for future analysis and reporting.

Some of the important CHCCLP commands for collecting performance details are :

  1. monitor replication
  2. monitor subscription activity
  3. monitor subscription busy tables
  4. monitor subscription latency
  5. monitor subscription performance
  6. monitor subscription refresh
  7. monitor table performance
  8. export subscription performance snapshot
  9. list subscription performance metrics
  10. list table performance metrics

Collect performance details for a subscription

There are number of ways and different commands that can be used in combination to get the required performance metrics. One of the most useful command that gets all the details related to subscription performance is monitor subscription performance metricsID " <command separated list of metrics ID > ". Users can only select the list of metrics ID that they are interested to collect for.

Tip: To get list of metric ID use list subscription performance metrics to know metricID for corresponding performance metrics

For example , If users are interested to collect Source & Target Engine Bytes processed, Source Pre and post Insert / Update / Delete, Target "Source" Insert /Update / Delete, Target "Apply" Insert / Update / Delete and Latency use the below command

monitor subscription performance metricIDs "102,104,105,106,107,108,109,118,202,204,205,206,207,208,209"

Once performance stats are retrieved ,  it can be persisted to a table structure or a flat file for further analysis and processing of the data. Users can retrieve statistics in real time and generate reports using ETL tools and Reporting of BI tools.

To export all the performance stats including Bottleneck information along with all performance indicators of Source / Target use

export subscription performance snapshot count 10 wait 5 folder "<path_to_snapshot">

To know more about how to collect performance snapshot visit my Demo video here

Now lets look at a example code, the will collect some of the performance metrics and persist into a table structure in database and sample graphs that can be generated out of these details.

We will use a property file to get details of subscription, its source and target data store etc. Below is the structure of properties file that we will use for our code:

config.properties

# Access Server Details

AS_HOSTNAME=localhost
AS_PORT=10101
AS_USERNAME=Demo
AS_PASSWORD=demo123

# DB Connection Details
DB_NAME=<dbName>
DB_USER=CDCMETADATA
DB_PASSWORD=CDCMETADATA
DB_PORT=1521
DB_HOST=<hostname>

# Performance Stats table Details
PERFORMANCE_SCHEMA=CDCMETADATA
PERFORMANCE_TABLE=PERFORMANCE

# Subscription Details
TOTAL_GROUPS=1
#SOURCE DATASTORE ,TARGET DATASTORE,SOURCESCHEMA,TARGETSCHEMA,SUBSCRIPTION NAME
GROUP1=DEMO_SOURCE2,DEMO_TARGET2,CDCSOURCE2,CDCTARGET2,DEMOSUB1|DEMOSUB2
GROUP2=DEMO_SOURCE2,DEMO_TARGET2,CDCSOURCE2,CDCTARGET2,DEMOSUB3

We will use the below structure for performance statistics table. You may need to add / remove columns depending on the list of performance metrics that you need to collect.

SUBID VARCHAR2(15) NOT NULL,PERFDATE varchar2(15),PERFTIME varchar2(15),SRC_BYTES NUMBER(38),SRC_PRE_INS NUMBER(38),SRC_PRE_UP NUMBER(38),SRC_PRE_DEL NUMBER(38),SRC_POS_INS NUMBER(38),SRC_POS_UP NUMBER(38),SRC_POS_DEL NUMBER(38),LATENCY NUMBER(38),TGT_BYTES NUMBER(38),TGT_SRC_INS NUMBER(38),TGT_SRC_UP NUMBER(38),TGT_SRC_DEL NUMBER(38),TGT_INS NUMBER(38),TGT_UP NUMBER(38),TGT_DEL NUMBER(38)

After getting the list of subscriptions for which performance stats has to be collected, collect the metrics and add to the metrics list for persisting to DB

for(int i=0;i<subNameList.size();i++){
                        // executeScript(script,"SELECT subscription name " + subName);
                        script.execute("select subscription name " + subNameList.get(i));
                        script.execute("monitor subscription performance metricIDs \"102,104,105,106,107,108,109,118,202,204,205,206,207,208,209\"");
                        result = script.getResult();
                        for (int k=0;k<((ResultStringTable)result).getRowCount();k++) {    
                            
                            if (! ((ResultStringTable)result).getValueAt(k, 0).trim().equals("Source") && !((ResultStringTable)result).getValueAt(k, 0).trim().equals("Target") ) {
                                String value = ((ResultStringTable)result).getValueAt(k, 1).replaceFirst(",","");
                                metricValues.add(value);
                                
                            }
                            
                        }

In the above code, for loop checks for certain rows of the output which is required to ignore header information about replication component . The below snippet shows output of the monitor subscription performance command when run on CHCCLP command window, which needs to be processed so that it can be transformed to a structure that can be persisted.

Current performance metrics for subscription DEMOSUB1.
Statistics requested at: Aug 25, 2014 11:48:47 PM

METRIC                                              VALUE
--------------------------------------------------- -------------
Source
    Source Engine - Source engine bytes processed   478,132
    Source Engine - Source pre-filter inserts       0
    Source Engine - Source pre-filter updates       0
    Source Engine - Source pre-filter deletes       0
    Source Engine - Source post-filter inserts      0
    Source Engine - Source post-filter updates      0
    Source Engine - Source post-filter deletes      0
    Source Engine - Latency value (seconds)         0
Target
    Target Engine - Target engine bytes processed   0
    Target Engine - Target "source" inserts         0
    Target Engine - Target "source" updates         0
    Target Engine - Target "source" deletes         0
    Target Apply - Target apply inserts             0
    Target Apply - Target apply updates             0
    Target Apply - Target apply deletes             0

Download the sample JAVA code here . The sample code uses Oracle DB to persist the performance stats. Add all the required dependency jar files for Embedded CHCCLP from <ACCESS SERVER_HOME>/lib directory.

This java code can be invoked in a loop over a predefined count and interval to collect the details and persist onto the database continuously. Once there are some details in the performance stats table, these details can be consumed by any ETL tool or reporting tools for generating graphs as below.

imageimage    

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSTRGZ","label":"InfoSphere Data Replication"},"Component":"","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF012","label":"IBM i"},{"code":"PF016","label":"Linux"},{"code":"PF051","label":"Linux on IBM Z Systems"},{"code":"PF033","label":"Windows"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
13 November 2019

UID

ibm11105095