How To
Summary
When the disk space reaches 95% used Elasticsearch has a protective function that locks the indices stopping new data from being written to them. This is to stop Elasticsearch from using any further disk causing the disk to become exhausted.
This document details what can be done to unlock the indices.
Objective
- Backups running
- The rolling of log files over night
- General increase in database or attachment size
- Collection of logs in directories where logs do not roll
In /var/log/elasticsearch/elasticsearch.log you see messages such as
[2019-08-06T11:22:54,950][WARN ][o.e.c.r.a.DiskThresholdMonitor] [VQNpg25] flood stage disk watermark [95%] exceeded on [VQNpg25iQlK6wVi_5uMvYA][VQNpg25][/var/lib/elasticsearch/nodes/0] free: 2.8gb[4.8%], all indices on this node will marked read-only
In /usr/share/co3/logs/client.log you see this message when data is added to the database but it cannot be indexed
11:24:50.831 [http-nio-443-exec-1] ERROR com.co3.search.ElasticSearchManager - Failed to update object 36 of index artifact com.co3.search.exception.ElasticSearchQueryException: ElasticSearch operation PUT on https://localhost:9200/artifact/_doc/36 failed with response 403: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];.
at com.co3.search.ElasticSearchClient$ClientHelper.execute(ElasticSearchClient.java:246)
at com.co3.search.ElasticSearchClient.put(ElasticSearchClient.java:333)
at com.co3.search.ElasticSearchManager.updateData(ElasticSearchManager.java:306)
at com.co3.search.ElasticSearchManager.sendToElasticUpdateData(ElasticSearchManager.java:881)
at com.co3.search.ElasticSearchUpdater.onObjectsCommitted(ElasticSearchUpdater.java:98)
Steps
v35 and higher
sudo systemctl stop restrict-elastic
Get the state of the indices
sudo curl -k -X GET "http://localhost:9200/incident/_settings?pretty"
{
"incident" : {
"settings" : {
"index" : {
"mapping" : {
"total_fields" : {
"limit" : "2500"
}
},
"number_of_shards" : "5",
"blocks" : {
"read_only_allow_delete" : "true"
},
"provided_name" : "incident",
"max_result_window" : "10000",
"creation_date" : "1598333552470",
"number_of_replicas" : "1",
"uuid" : "1hFWIVHHS5ebgUrPUP3gOg",
"version" : {
"created" : "6020499"
}
}
}
}
}
sudo curl -k -X PUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d ' {"index.blocks.read_only_allow_delete": null} '
sudo curl -k -X GET "http://localhost:9200/incident/_settings?pretty"
These commands will reinitiate a full reindex after a restart of IBM Resilient.
sudo resutil configset -key elastic_server.init_schema -bvalue true
sudo systemctl restart resilient
sudo systemctl start restrict-elastic
v33 to v35
sudo -i
cd /crypt/certs
resutil keyvaultget -name elastic_client_keystore
Keep the password safe somewhere
keytool -importkeystore -srckeystore elastic-client-keystore -destkeystore elastic-client-keystore.p12 -deststoretype PKCS12
Use the password from the keyvaultget command for all three prompts
openssl pkcs12 -in elastic-client-keystore.p12 -nodes -nocerts -out client-key.pem -password pass:********
MAC verified OK
Use the password from above
cat client-key.pem elastic-client.pem > client-all.pem
Focusing on the incident index.
curl -E ./client-all.pem -k -X GET 'https://localhost:9200/incident/_settings?pretty'
{
"incident" : {
"settings" : {
"index" : {
"mapping" : {
"total_fields" : {
"limit" : "2500"
}
},
"number_of_shards" : "5",
"blocks" : {
"read_only_allow_delete" : "true"
},
"provided_name" : "incident",
"max_result_window" : "10000",
"creation_date" : "1563787508818",
"number_of_replicas" : "1",
"uuid" : "DK7wqtgHRIW8mATznR3BiQ",
"version" : {
"created" : "6020499"
}
}
}
}
}
Notice "read_only_allow_delete" : "true"
The following command removes the lock for ALL indices
curl -E ./client-all.pem -k -X PUT -H "Content-Type: application/json" https://localhost:9200/_all/_settings -d ' {"index.blocks.read_only_allow_delete": null} '
Checking the incident index again
curl -E ./client-all.pem -k -X GET 'https://localhost:9200/incident/_settings?pretty'
{
"incident" : {
"settings" : {
"index" : {
"mapping" : {
"total_fields" : {
"limit" : "2500"
}
},
"number_of_shards" : "5",
"provided_name" : "incident",
"max_result_window" : "10000",
"creation_date" : "1563787508818",
"number_of_replicas" : "1",
"uuid" : "DK7wqtgHRIW8mATznR3BiQ",
"version" : {
"created" : "6020499"
}
}
}
}
}
Notice that "read_only_allow_delete" : "true" is not present.
You can check all indices by running
curl -E ./client-all.pem -k -X GET 'https://localhost:9200/_all/_settings?pretty'
From this point onwards new data is indexed but data that was created while the indices were locked is not so the following is required to rebuild the indices.
These commands will reinitiate a full reindex after a restart of IBM Resilient.
sudo resutil configset -key elastic_server.init_schema -bvalue true
sudo systemctl restart resilient
Search begins working again as soon as you execute the commands, but until the index is fully rebuilt the results might be incomplete. The amount of time it takes to rebuild the index depends on the size of your data. Data is indexed from newest to oldest so newer incidents are available in the index sooner than older ones.
Additional Information
- /usr/share/co3/logs/client_access_log<YYYY-MM-DD>.log
- /var/log/elasticsearch/
- /tmp/
- /home/resadmin/
- /crypt/backups/
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
19 April 2021
UID
ibm11160026