Troubleshooting
Problem
Symptom
Cause
Split Brain
Elastic search has a cluster of nodes with a single master node. When there is a communication issue between the master node and the other cluster nodes, then another node can assume the master status, causing a split brain.
Diagnosing The Problem
How to identify a split brain?
- Number of analytics events is reduced.
- Inconsistent behavior of the displayed results
Resolving The Problem
You can check the nodes in split brain by running:
kubectl get pods
Exec into one of the storage pods:
kubectl exec -it <POD_NAME> — bash
curl_es _cluster/stats?pretty | grep "_nodes" -A 3
"_nodes": {
"total": 3,
"successful": 3,
"failed": 0
Check the number of successful nodes and make sure that matches the number of storage pods. If the nodes are fewer than the pods, then there is a split brain.
To resolve a split brain restart the pod that is by itself, or has less members in the cluster. For example if there are 3 pods then check each pod for:
curl_es _cat/nodes
Which ever pod shows less number of nodes, restart that pod.
There will be some data loss during this activity. With a container-based install (v2018 and v10) the chances of split brain happening is very low.
Related Information
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
01 November 2021
UID
ibm16496817