IBM Support

QRadar: How to use ha_diagnosis to troubleshoot high avalibility issues

How To


Summary

ha_diagnosis is a summary utility that completes a series of tests to output a summary of high availability appliance checks to the administrator.

Steps

Run ha_diagnosis
  1. SSH into your QRadar console (primary host).
  2. Run ha_diagnosis by using the following command:
    /opt/qradar/ha/bin/ha_diagnosis
    Example of a successful output:
    HA manager is running
    Currently, You are on HA primary.
    Check the HA State
     > Currently, local HA state reaches ACTIVE state
     > Currently, remote HA state reaches STANDBY state
    Check the HA heartbeat      [OK]
    Checking HA Virtual IP
     > HA Virtual Interface is UP
    Checking QRadar Services        [OK]
    Checking HA Mount
     > HA Mount service is running
    Checking HA DRBD
     > Local DRBD Role is primary
     > HA DRBD Connection Status is Connected
    Checking DRBD configuration files       [OK]
    Checking 'drbdadm show-gi store' fields     [OK]
    Check the hidden token      [OK]
     
    Diagnosis Summary:
     > All the HA check is PASSED    [OK]
  3. SSH into your secondary host.
  4. Run ha_diagnosis.sh by using the following command:
    /opt/qradar/ha/bin/ha_diagnosis
    Example of a successful output:
    HA manager is running
    Currently, You are on HA secondary.
    Check the HA State
    > Currently, local HA state reaches STANDBY state
    > Currently, remote HA state reaches ACTIVE state
    Check the HA heartbeat [OK]
    Checking HA Virtual IP
    > HA Virtual Interface is DOWN
    Checking QRadar Services [OK]
    Checking HA Mount
    > HA Mount service is not running
    Checking HA DRBD
    > Local DRBD Role is secondary
    > HA DRBD Connection Status is Connected
    Checking DRBD configuration files [OK]
    Checking 'drbdadm show-gi store' fields [OK]
    Check the hidden token [OK]
     
    Diagnosis Summary:
    > All the HA check is PASSED [OK]

    Result
    If you encounter earlier failures, such as missing a ha.conf file, the script stops early. Observe the output and follow the troubleshooting steps provided. If you need assistance, contact support and share the output of this script.

Understanding the output

The ha_diagnosis completes the following checks. Not all checks are run on every system, and if you do not use verbose mode, some successful outputs are hidden.
Check Description Notes
HA manager is running Ensures the HA manager is running by using /opt/qradar/ha/bin/ha If the HA manager is not running, contact support.
Currently, You are on HA <Role:Primary|Secondary> Determines the current system role by running /opt/qradar/ha/bin/ha cstate If both systems have the same role, contact support.
Check the HA State Determines the role of the appliance by running  /opt/qradar/ha/bin/ha cstate If you are on the primary host, and the local state is STANDBY, you can set the primary to the ACTIVE state.
Check the HA heartbeat Determines the heartbeat count (HBC) between hosts by running /opt/qradar/ha/bin/ha cstate If a heartbeat is not registered in the timeout period, the host is considered unavailable. The default heartbeat timeout it 30 seconds, which can be adjusted in the advanced cluster settings.
Checking HA Virtual IP Confirms the virtual IP by running /opt/qradar/ha/init.d/ha_ipaddr status When you create an HA cluster, the cluster virtual IP address takes the IP address of the primary HA host. See the HA Configuration Guide for more information.
Checking QRadar Services Checks the status of QRadar services such as hostservices, hostcontext, and Tomcat If any services are stopped, try starting them by using systemctl.
Checking HA Mount Checks that the HA file systems are mounted by running /opt/qradar/ha/init.d/ha_mount The HA mount service is used if you configured offboard storage. Otherwise, expect the service not to be running.
Checking HA DRBD Checks the HA Distributed Replication Block Device (DRBD) by running /opt/qradar/ha/init.d/hha_drbd status. Determines and DRBD connection state with cat /proc/drbd If you see both hosts are StandAlone, or one is StandAlone and the other is WFConnection, you might be in a split brain state and must contact support.
Checking DRBD configuration files Compares the DRBD configuration files between the local and remote hosts and ensures they aren't truncated If the DRBD state, files, or fields return an error, use this article to understand the error.
Checking 'drbdadm show-gi store' fields Confirms that data consistency and status of the fields by running drdadm show-gi store. If the DRBD state, files, or fields return an error, use this article to understand the error.
Check the hidden token Looks for hidden files and patching failures in /opt/qradar/ha After a failed patch, follow this procedure to recover the system.
Checking HA Gluster Filesystem Status Checks the HA Gluster Fileystem Status to ensure the glusterd daemon is running and the peer is connected If the DRBD state, files, or fields return an error, use this article to understand the error.

Options

  • -a Apply all the checks even if prerequisite checks fail
  • -S Check HA State from HA Manager
  • -s Check QRadar Service status
  • -f Check HA mount status
  • -d Check HA DRBD status
  • -i Check HA virtual interface status
  • -t Check HA hidden tokens
  • -V Verbose output, which displays all passed checks
  • -c Skip all the check if in synchronization state
  • -v Display revision information
  • -h Displays information about these flag options

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
23 November 2022

UID

ibm16828547