
Elasticsearch considers available disk space to calculate and allocate shard on that node. if there is less space left on disk, Elasticsearch put itself into read-only mode.
By default these setting are enabled in Elasticsearch.
cluster.routing.allocation.disk.threshold_enabled: By default its true and will enable following settings.
cluster.routing.allocation.disk.watermark.low: Default to 85%, which means, elastic search will not create more shards on the node with more than 85% disk space used.
cluster.routing.allocation.disk.watermark.high: Default to 90%, which means, Elasticsearch will try to move shard from node with 90% or more disk spaced used.
cluster.routing.allocation.disk.watermark.flood_stage: Default to 95%, which means, Elasticsearch will enforce read-only mode to all the index that has one or more shard on any of the node with 95% disk space used.
Solution:
Free up some disk space: If possible, free up disk spaced so that free space be more than 5%. After disk is freed up need to unlock read only access.
PUT /twitter/_settings
{
“index.blocks.read_only_allow_delete”: null
}
Disable or change settings: We can change watermark setting to low value, example of settings are as below.
PUT _cluster/settings
{
“transient”: {
“cluster.routing.allocation.disk.watermark.low”: “100gb”,
“cluster.routing.allocation.disk.watermark.high”: “50gb”,
“cluster.routing.allocation.disk.watermark.flood_stage”: “10gb”,
“cluster.info.update.interval”: “1m”
}
more information on this issue can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/disk-allocator.html