Elasticsearch: FORBIDDEN/12/index read-only / allow delete (api) error

Elasticsearch considers available disk space to calculate and allocate shard on that node. if there is less space left on disk, Elasticsearch put itself into read-only mode.
By default these setting are enabled in Elasticsearch.

cluster.routing.allocation.disk.threshold_enabled: By default its true and will enable following settings.

cluster.routing.allocation.disk.watermark.low: Default to 85%, which means, elastic search will not create more shards on the node with more than 85% disk space used.

cluster.routing.allocation.disk.watermark.high: Default to 90%, which means, Elasticsearch will try to move shard from node with 90% or more disk spaced used.

cluster.routing.allocation.disk.watermark.flood_stage: Default to 95%, which means, Elasticsearch will enforce read-only mode to all the index that has one or more shard on any of the node with 95% disk space used.

Solution:

Free up some disk space: If possible, free up disk spaced so that free space be more than 5%. After disk is freed up need to unlock read only access.

PUT /twitter/_settings
{
“index.blocks.read_only_allow_delete”: null
}

Disable or change settings: We can change watermark setting to low value, example of settings are as below.

PUT _cluster/settings

{

“transient”: {

“cluster.routing.allocation.disk.watermark.low”: “100gb”,

“cluster.routing.allocation.disk.watermark.high”: “50gb”,

“cluster.routing.allocation.disk.watermark.flood_stage”: “10gb”,

“cluster.info.update.interval”: “1m”

}

more information on this issue can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/disk-allocator.html

Leave a comment