Logstash [logstash.outputs.elasticsearch][main] retrying failed action with response code: 403

问题描述：
今天发现当天的索引在ES中并没有创建，logstash中不停的报错：

[2021-05-24T05:47:51,904][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2021-05-24T05:47:51,904][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}

索引变成了只读

解决办法：

集群需要在每个 es 节点服务器上执行

# 含密码
curl -XPUT -H 'Content-Type: application/json' --user 账号:密码 http://es服务器ip:port/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

# 不含密码
curl -XPUT -H 'Content-Type: application/json' http://es服务器ip:port/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

Disk-based shard allocation | Elasticsearch Guide [7.6] | Elastichttps://www.elastic.co/guide/en/elasticsearch/reference/7.6/disk-allocator.html