IBM Support

Elasticsearch indices are locked after a shortage of disk space

How To


Summary

When the disk space reaches 95% used Elasticsearch has a protective function that locks the indices stopping new data from being written to them. This is to stop Elasticsearch from using any further disk causing the disk to become exhausted.

This document details what can be done to unlock the indices.

Objective

While the indices are locked new data is not indexed and thus is not searchable. Users might see inconsistent search results with new incidents or updates not returned when searched for.
Some of the reasons for the shortage of disk could be related to
  • Backups running
  • The rolling of log files over night
  • General increase in database or attachment size
  • Collection of logs in directories where logs do not roll

In /var/log/elasticsearch/elasticsearch.log you see messages such as

[2019-08-06T11:22:54,950][WARN ][o.e.c.r.a.DiskThresholdMonitor] [VQNpg25] flood stage disk watermark [95%] exceeded on [VQNpg25iQlK6wVi_5uMvYA][VQNpg25][/var/lib/elasticsearch/nodes/0] free: 2.8gb[4.8%], all indices on this node will marked read-only

In /usr/share/co3/logs/client.log you see this message when data is added to the database but it cannot be indexed

11:24:50.831 [http-nio-443-exec-1] ERROR com.co3.search.ElasticSearchManager - Failed to update object 36 of index artifact  com.co3.search.exception.ElasticSearchQueryException: ElasticSearch operation PUT on https://localhost:9200/artifact/_doc/36 failed with response 403: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];.   
at com.co3.search.ElasticSearchClient$ClientHelper.execute(ElasticSearchClient.java:246)   
at com.co3.search.ElasticSearchClient.put(ElasticSearchClient.java:333)   
at com.co3.search.ElasticSearchManager.updateData(ElasticSearchManager.java:306)   
at com.co3.search.ElasticSearchManager.sendToElasticUpdateData(ElasticSearchManager.java:881)   
at com.co3.search.ElasticSearchUpdater.onObjectsCommitted(ElasticSearchUpdater.java:98)

Steps

v35 and higher

Remove the restriction
sudo systemctl stop restrict-elastic

Get the state of the indices
sudo curl -k -X GET "http://localhost:9200/incident/_settings?pretty"
Look out for "read_only_allow_delete" : "true".
{
  "incident" : {
    "settings" : {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "2500"
          }
        },
        "number_of_shards" : "5",
        "blocks" : {
          "read_only_allow_delete" : "true"
        },
        "provided_name" : "incident",
        "max_result_window" : "10000",
        "creation_date" : "1598333552470",
        "number_of_replicas" : "1",
        "uuid" : "1hFWIVHHS5ebgUrPUP3gOg",
        "version" : {
          "created" : "6020499"
        }
      }
    }
  }
}
This command removes the lock for ALL indices

sudo curl -k -X PUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d ' {"index.blocks.read_only_allow_delete": null} '
Get the state of the indices
sudo curl -k -X GET "http://localhost:9200/incident/_settings?pretty"
"read_only_allow_delete" : "true" should now not be present.
From this point onwards new data is indexed but data that was created while the indices were locked is not so the following is required to rebuild the indices.

These commands will reinitiate a full reindex after a restart of IBM Resilient.
sudo resutil configset -key elastic_server.init_schema -bvalue true
sudo systemctl restart resilient
sudo systemctl start restrict-elastic
Search begins working again as soon as you execute the commands, but until the index is fully rebuilt the results might be incomplete. The amount of time it takes to rebuild the index depends on the size of your data.  Data is indexed from newest to oldest so newer incidents are available in the index sooner than older ones.

v33 to v35

sudo -i
cd /crypt/certs
resutil keyvaultget -name elastic_client_keystore

Keep the password safe somewhere

keytool -importkeystore -srckeystore elastic-client-keystore -destkeystore elastic-client-keystore.p12 -deststoretype PKCS12

Use the password from the keyvaultget command for all three prompts

openssl pkcs12 -in elastic-client-keystore.p12 -nodes -nocerts -out client-key.pem -password pass:********
 MAC verified OK

Use the password from above

cat client-key.pem elastic-client.pem > client-all.pem

Focusing on the incident index.

curl -E ./client-all.pem -k -X GET 'https://localhost:9200/incident/_settings?pretty'
{
  "incident" : {
    "settings" : {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "2500"
          }
        },
        "number_of_shards" : "5",
        "blocks" : {
          "read_only_allow_delete" : "true"
        },
        "provided_name" : "incident",
        "max_result_window" : "10000",
        "creation_date" : "1563787508818",
        "number_of_replicas" : "1",
        "uuid" : "DK7wqtgHRIW8mATznR3BiQ",
        "version" : {
          "created" : "6020499"
        }
      }
    }
  }
}

Notice "read_only_allow_delete" : "true"

The following command removes the lock for ALL indices

curl -E ./client-all.pem -k -X PUT -H "Content-Type: application/json" https://localhost:9200/_all/_settings -d ' {"index.blocks.read_only_allow_delete": null} '

Checking the incident index again

curl -E ./client-all.pem -k -X GET 'https://localhost:9200/incident/_settings?pretty'
{
  "incident" : {
    "settings" : {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "2500"
          }
        },
        "number_of_shards" : "5",
        "provided_name" : "incident",
        "max_result_window" : "10000",
        "creation_date" : "1563787508818",
        "number_of_replicas" : "1",
        "uuid" : "DK7wqtgHRIW8mATznR3BiQ",
        "version" : {
          "created" : "6020499"
        }
      }
    }
  }
}

Notice that "read_only_allow_delete" : "true" is not present.

You can check all indices by running

curl -E ./client-all.pem -k -X GET 'https://localhost:9200/_all/_settings?pretty'

From this point onwards new data is indexed but data that was created while the indices were locked is not so the following is required to rebuild the indices.

These commands will reinitiate a full reindex after a restart of IBM Resilient.

sudo resutil configset -key elastic_server.init_schema -bvalue true
sudo systemctl restart resilient

Search begins working again as soon as you execute the commands, but until the index is fully rebuilt the results might be incomplete. The amount of time it takes to rebuild the index depends on the size of your data.  Data is indexed from newest to oldest so newer incidents are available in the index sooner than older ones.

Additional Information

This could happen more than once. Asses your disk requirements, which depend on the usage of IBM Resilient and plan to extend or add extra disk.
Consider removing unnecessary files
  • /usr/share/co3/logs/client_access_log<YYYY-MM-DD>.log
  • /var/log/elasticsearch/
  • /tmp/
  • /home/resadmin/
  • /crypt/backups/

Document Location

Worldwide

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSIP9Q","label":"IBM Security SOAR"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
19 April 2021

UID

ibm11160026

Page Feedback