Elasticsearch: FORBIDDEN/12/index read-only / allow delete (api) error

Elasticsearch considers available disk space to calculate and allocate shard on that node. if there is less space left on disk, Elasticsearch put itself into read-only mode.
By default these setting are enabled in Elasticsearch.

cluster.routing.allocation.disk.threshold_enabled: By default its true and will enable following settings.

cluster.routing.allocation.disk.watermark.low: Default to 85%, which means, elastic search will not create more shards on the node with more than 85% disk space used.

cluster.routing.allocation.disk.watermark.high: Default to 90%, which means, Elasticsearch will try to move shard from node with 90% or more disk spaced used.

cluster.routing.allocation.disk.watermark.flood_stage: Default to 95%, which means, Elasticsearch will enforce read-only mode to all the index that has one or more shard on any of the node with 95% disk space used.

Solution:

Free up some disk space: If possible, free up disk spaced so that free space be more than 5%. After disk is freed up need to unlock read only access.

PUT /twitter/_settings
{
“index.blocks.read_only_allow_delete”: null
}

Disable or change settings: We can change watermark setting to low value, example of settings are as below.

PUT _cluster/settings

{

“transient”: {

“cluster.routing.allocation.disk.watermark.low”: “100gb”,

“cluster.routing.allocation.disk.watermark.high”: “50gb”,

“cluster.routing.allocation.disk.watermark.flood_stage”: “10gb”,

“cluster.info.update.interval”: “1m”

}

more information on this issue can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/disk-allocator.html

AWS: CloudFront 502 bad gateway error

Usually 502 error happens when Cloudfront unable to reach origin endpoint. Some of the common reasons are:

Backend is busy: Origin endpoint ( ELB -> EC2) is busy or load is higher than given threshold. If this is happening reguarly, its better to get more instance in auto scaling group or increase existing instance for larger capacity.

WAF rule: Some request might get blocked by WAF rule. So check cloudFront logs to verfiy, if needed modify your application or disable WAF rule ( not recommended).

DNS: If Endpoint has custom DNS, make sure DNS is reachable.

EC2 instances: All EC2 instances in origin endpoint are down or unhealthy.