Lesson 3: Setting up Derby auto replication in XSLD

Now that you have two or more members in this cache member group, you can set up replication among these members.

Before you begin

Time should be synched across all the nodes, because this feature relies on Timestamps of the modified file.
Watch-dog service needs to be running to ensure automatic Derby replication functionality.
Derby primary and replica are placed on XSLD catalog members only. The number of XSLD catalog members on which Derby can be placed is also restricted to 3.
Terminology note: The following example code is not from IBM® Corporation. While IBM values the use of inclusive language, terms and code that are not within the direct influence of IBM are sometimes needed for user understanding. As other industry leaders join IBM in embracing the use of inclusive language, IBM will update the documentation to reflect those changes.

For example, with the default setting for catalog server for 2-members or 3-members environment, all the XSLD members are used for Derby replication. For 4 or more members, Derby is only enabled on the members with catalog server, such as 3 XSLD catalog members. By default, only 3 members have catalog servers. If the default configuration is overridden and 4-members are catalog servers, Derby is placed on any 3 catalog XSLD members. See the dbconfig.json file for a list of these members.
```
{"db_type":"derby","database":"wxsdb","db_driver":"org.apache.derby.jdbc.ClientDriver","password":"*****","slave_host":"localhost","port":"1527","catalog_hosts":"[member1.com, member2.com, member3.com]","dbMaintenanceBeforeTS":"0","db_host":"member1.com","user":"wxsadmin"}
```
Each XSLD member can update to the latest dbconfig.json file. The update works seamlessly if all the XSLD members are online.
Failover takes longer after a host is offline as opposed to when a Derby process is stopped or ended. For a shorter turn-around time for failover, run stopNetworkServer to stop the Derby process.
```
wxs_install_root/wxs/derby/scripts/stopNetworkServer.sh/bat
```
Do not run manual Derby replication with AutoDBReplication enabled. For control on which machines the primary and replica Derby are placed, either run certain failovers or use manual Derby replication, as Derby primary and replica are placed on any of the 3 catalog XSLD members.

About this task

For the existing cache member group, Derby is replicated after AutoDBReplication is enabled, by setting it to true with the cURL command or REST Admin Server UI.
The automatic Derby replication in failover scenario is, when primary Derby fails, the replica Derby is promoted to primary and a new replica Derby is placed on any of the 3 catalog XSLD members.
When Derby replication is unavailable, or there is no replica Derby, a new replica Derby is placed on any of the 3 catalog XSLD members.

If there are no catalog XSLD members online for replication, the primary Derby member waits and places a new Derby replica after any of the catalog XSLD members is online.

For example, in the three nodes scenario:

Events	Member 1	Member 2	Member 3
Start-up	Master Derby	-	-
Few minutes after enabling automatic Derby replication	Master Derby	Replica Derby	-
Member 1 unavailable	X	Master Derby	Replica Derby
Member 3 unavailable as well	X	Master Derby	X
All XSLD nodes available	Replica derby	Master Derby	X

There can be various other combination based on the different events and number of XSLD nodes, but the same principles apply.
Automatic Derby replication takes about 10-20 minutes for failover. The duration depends on a number of factors, including system performance and network.
The time taken during Derby replication cannot be further reduced because this is required to avoid split-brain scenarios.
Stop all members. Identify the primary and replica Derby members. Then, stop the members in the following order:
1. Stop non-Derby members. Non-Derby members are members apart from the ones listed as db_host and slave_host in the dbconfig.json file.
2. Stop replica members and wait until slave_host information in the primary member dbconfig.json file changes to localhost.
3. Stop the db_host primary Derby member.
```
{"db_type":"derby","database":"wxsdb","db_driver":"org.apache.derby.jdbc.ClientDriver","password":"*****","slave_host":"localhost","port":"1527","catalog_hosts":"[member1.com, member2.com, member3.com]","dbMaintenanceBeforeTS":"0","db_host":"member1.com","user":"wxsadmin"}
```
Following this process ensures that, after all the XSLD members are started together, XSLD can determine which dbconfig.json file is latest and can assign primary db_host and later place a replica DB on the next available catalog member. If all XSLD members are stopped or rebooted at the same time for any reason, manual intervention might be required. Edit the correct db_host and slave_host values in the dbconfig.json file and restart the members. The dbconfig.json file is located at wxs_install_root/wlp/usr/shared/resources/dbconfig.json.
While Derby Replication is going on , there will be some downtime for Derby(Since the Primary Derby server will restart itself as part of Derby replication process) . During this time XSLD UI , Rest data , Rest Admin server will not be accessible.

Procedure

Open the XSLD REST admin console for member1.

Point your browser to the https://<hostname.domain.com>:9445/ibm/api/explorer URL where <hostname.domain.com> represents the fully qualified host name of the cache member.
Note: The host name must be the fully qualified name of the cache member.
Click runtime-setting : Runtime setting management API.
PUT -> Name - AutoDBReplication Value – true / false. (Default - false)
It also can be set up with a cURL command.
```
curl -k -u xsadmin:xsadmin4Me! -X PUT -H "Content-Type:application/json" -d "{\"value\":\"true\"}" https://jsmith.abc.com:9445/wxsadmin/v1/runtimesetting/AutoDBReplication
```
Where
- "xsadmin:xsadmin4Me!" is your administrator user ID and password.
- true is your AUTODBReplication settings
- https://jsmith.abc.com:9445/wxsadmin/v1/runtimesetting/AutoDBReplication is the fully qualified host name and port number of the cache member group for which you want to enable remote logging. It also contains the context root of the Admin REST server, the REST call to enable logging, and the system property log.syslogServiceName that will pass the value to the remote system.

Results

After the task completes successfully, the automatic Derby replication is enabled.