Diagnosis and treatment of a heartbeat failure in a DSI cluster

If a Decision Server Insights cluster stops processing events or stops communicating with clients, add the -Xverbosegclog JVM parameter to the jvm.options file in your runtime servers. You can then check in the logs to see if the garbage collection (GC) pause times are longer than 30 seconds. If the GC times are longer than 30 seconds, update the configuration files to include properties to change the heartbeat policy and the XIO timeouts to match the heartbeat value.

How to change the heartbeat policy

1. First of all, create a file named catalogServer.properties.
2. Add the following properties to the file (catalogServer.properties):

   heartBeatFrequencyLevel=10
   xioTimeout=90
   xioReadTimeout=90
   xioWriteTimeout=90

3. Copy and paste the file into the root folder ($DSI_HOME/runtime/wlp/usr/servers/catalog_name) of each catalog server.
4. In the server.xml file of each catalog server, edit the xsServer element to add the file. The following example uses the file named catalogServer.properties.
   
5. Create a file named runtimeServer.properties.
6. Add the following properties to the file (runtimeServer.properties):

   xioTimeout=90
   xioReadTimeout=90
   xioWriteTimeout=90

7. Copy and paste the file runtimeServer.properties into the root folder of each runtime server.
8. Add a reference to the properties file in the xsServer element of the server.xml files.
   
9. Create a file named client.properties, and add the line: xioRequestTimeout=90000.
10. Copy and paste the file client.properties into the root folder of each runtime server.
11. Declare the file in the jvm.options file in each runtime server: -Dobjectgrid.client.props=client.properties
12. Create a new properties file named objectGridClient.properties, and add the following properties:

    xioRequestTimeout=90000
    xioTimeout=90
    xioReadTimeout=90
    xioWriteTimeout=90

13. Copy and paste the file objectGridClient.properties into the root folder of each connectivity (inbound and outbound) server.
14. Declare the file in the jvm.options file for each connectivity server: -Dobjectgrid.client.props=objectGridClient.properties
15. Finally, restart the catalogs, runtime servers, and connectivity servers in this order after making these changes.

Related information

• Troubleshooting
• Configuring Insight Server
• Creating, starting, and stopping servers