Frequently asked questions about IBM Security Key Lifecycle Manager Multi-Master

Q) When IBM Security Key Lifecycle Manager servers are configured for multi-master setup, the data synchronization service automatically copies data from the primary server to the master servers at regular intervals. Why do I need to use the Backup and Restore feature of IBM Security Key Lifecycle Manager?
    A) As a precautionary measure to avoid possible data loss, use the Backup and Restore feature to back up data manually at regular intervals.

Q) I added a non-HADR IBM Security Key Lifecycle Manager server to the multi-master cluster. Why am I still seeing data from the local database instead of data from the primary database?
    A) IBM Security Key Lifecycle Manager data source might still be pointing to local database even after successfully adding the master to the cluster. After adding the master to the cluster, the data source in its WebSphere® Application Server must point the primary database.
    You can manually update the server details in the data source of the master server that you added by running the following steps.

1. Log on to WebSphere Integrated Solutions Console (https://localhost:9083/ibm/console/logon.jsp).
2. Click Resources > JDBC > Data sources > SKLM DataSource.
3. In the SKLM DataSource page, verify the value in the Server name field under Common and required data source properties section. If the host name of the local server is shown, update the value by specifying the host name of the primary server. If the value in the Server name field is already updated with the host name of the primary server, restart WebSphere Application Server and close the page. Else, run the following steps.
4. In the Additional Properties section of the SKLM DataSource page, click the WebSphere Application Server data source properties link.
5.  In the Advanced DB2 features section, verify the values in the Alternate server names field.
6.  Specify host name of the standby servers as a comma-separated list in the Alternate server names field.
7.  Specify port number of the standby servers as a comma-separated list in the Alternate port numbers field.
8.  Save the changes.
9.  Click Resources > JDBC > Data sources > SKLM scheduler XA Datasource.
10.  Repeat the steps from 3 - 8.
11.  Restart WebSphere Application Server.

Q) How do I check that synchronization is running between IBM Security Key Lifecycle Manager primary and standby servers?
    A) You can check whether synchronization is running between IBM Security Key Lifecycle Manager primary and standby servers by verifying the time difference between PRIMARY_LOG_TIME and STANDBY_LOG_TIME Run the following command from DB2 Command Window

#db2pd -d <SKLM_DBName> –hadr

For example,

The following output is displayed.

Database Member 0 -- Database SKLMDB30 -- Active -- Up 1 days 21:27:01 -- Date 2017-11-09-20.10.56.059000

                        HADR_ROLE = PRIMARY
                          REPLAY_TYPE = PHYSICAL
                        HADR_SYNCMODE = SYNC
                           STANDBY_ID = 1
                        LOG_STREAM_ID = 0
                           HADR_STATE = PEER
                           HADR_FLAGS = TCP_PROTOCOL
                  PRIMARY_MEMBER_HOST = WIN-DBA2ALEJOC8
                     PRIMARY_INSTANCE = SKLMDB30
                       PRIMARY_MEMBER = 0
                  STANDBY_MEMBER_HOST = WIN-VB479C09AG3
                     STANDBY_INSTANCE = SKLMDB30
                       STANDBY_MEMBER = 0
                  HADR_CONNECT_STATUS = CONNECTED
             HADR_CONNECT_STATUS_TIME = 11/08/2017 23:25:28.730219 (1510212328)
          HEARTBEAT_INTERVAL(seconds) = 30
                     HEARTBEAT_MISSED = 0
                   HEARTBEAT_EXPECTED = 2490
                HADR_TIMEOUT(seconds) = 120
        TIME_SINCE_LAST_RECV(seconds) = 3
             PEER_WAIT_LIMIT(seconds) = 0
           LOG_HADR_WAIT_CUR(seconds) = 0.000
    LOG_HADR_WAIT_RECENT_AVG(seconds) = 0.001541
   LOG_HADR_WAIT_ACCUMULATED(seconds) = 45.835
                  LOG_HADR_WAIT_COUNT = 19538
SOCK_SEND_BUF_REQUESTED,ACTUAL(bytes) = 0, 65536
SOCK_RECV_BUF_REQUESTED,ACTUAL(bytes) = 0, 65536
            PRIMARY_LOG_FILE,PAGE,POS = S0000003.LOG, 4891, 191226150
            STANDBY_LOG_FILE,PAGE,POS = S0000003.LOG, 4886, 191205494
                  HADR_LOG_GAP(bytes) = 0
     STANDBY_REPLAY_LOG_FILE,PAGE,POS = S0000003.LOG, 4886, 191205494
       STANDBY_RECV_REPLAY_GAP(bytes) = 0
                     PRIMARY_LOG_TIME = 11/09/2017 20:10:52.000000 (1510287052)
                     STANDBY_LOG_TIME = 11/09/2017 20:10:20.000000 (1510287020)
              STANDBY_REPLAY_LOG_TIME = 11/09/2017 20:10:20.000000 (1510287020)
         STANDBY_RECV_BUF_SIZE(pages) = 4298
             STANDBY_RECV_BUF_PERCENT = 0
           STANDBY_SPOOL_LIMIT(pages) = 380000
                STANDBY_SPOOL_PERCENT = 0
                   STANDBY_ERROR_TIME = NULL
                 PEER_WINDOW(seconds) = 0
             READS_ON_STANDBY_ENABLED = N

In the output, PRIMARY_LOG_TIME shows the time at which the DB2 Transactional logs are updated for Primary server. STANDBY_LOG_TIME shows the time at which the DB2 Transactional logs are updated for standby server. You can ignore the time difference in milliseconds.

Q) How do I view status of ports in the IBM Security Key Lifecycle Manager multi-master setup?
    A) IBM Security Key Lifecycle Manager GUI uses icons to represent port status on the multi-master pages. The following table shows the port status icons and their meanings.

    Table 1. Status icons and their meanings

Icon	Description
	Port is reachable and serving requests as per the specifications.
	Port is not reachable. Service on a specific port might be down. Refresh status by using the Refresh option on the UI page.

Q) How do I view DB2 HADR status in the IBM Security Key Lifecycle Manager multi-master setup?
    A) IBM Security Key Lifecycle Manager GUI uses icons to represent DB2 HADR status on the multi-master pages. The following table shows the DB2 HADR status icons and their meanings.
    Table 2. Status icons and their meanings

Icon	Description
	DB2 HADR is in running state. All the HADR servers are connected with each other.
	DB2 HADR is in running state, but at least one of the standby HADR servers is not reachable.
	DB2 HADR is down and non-functional.

Q) How to solve the error : "Incoming instance is NOT clean", while adding new master?

  A) The incoming master should be a clean system. You can solve this issue, by removing the entry : "config.keystore.ssl.certalias= " from the SKLMConfig.properties file of the incoming master server. Restart the WebSphere server.

Q) How to resolve the error: "DB2 credentials of the incoming master do NOT match with that of the present master", while adding new master?

 A) This can occur due to the following reasons:

•  The DB2 administrator passwords of both the servers are actually different.
• Not in MultiMaster Cluster - The DB2 administrator passwords were changed post install, and the Rest service to update DB2 password on a stand-alone IBM Security Key Lifecycle Manager master server was not executed.

https://www.ibm.com/support/knowledgecenter/en/SSWPVP_3.0.0.1/com.ibm.sklm.doc/reference/ref/ref_ic_rest_multimaster_update_db2_password_standalone_sklm.html

• In Multi-Master Cluster - The Rest service to update DB2 password on all the IBM Security Key Lifecycle Manager servers in the multi-master cluster was not executed.

https://www.ibm.com/support/knowledgecenter/en/SSWPVP_3.0.0.1/com.ibm.sklm.doc/reference/ref/ref_ic_rest_multimaster_update_db2_password_multiple_sklm.html