解决LogStash报错：FORBIDDEN/12/index read-only / allow delete (api)

一、现象

在Kibana上一直查询不到索引最新插入的数据，通过排查发现

Logstash 的日志(/logs/logstash-plain.log) 大量报错：FORBIDDEN/12/index read-only / allow delete (api)

[2020-10-27T15:35:12,119][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [index_user_info] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2020-10-27T15:35:12,119][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [index_user_info] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2020-10-27T15:35:12,119][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [index_user_info] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2020-10-27T15:35:12,119][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [index_user_info] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2020-10-27T15:35:12,119][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [index_user_info] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})

其中index_user_info 是我的ES索引名称！

 

二、原因：磁盘空间不够导致

查阅资料，发现ES说明文档中有写明：当ES数据所在目录磁盘空间使用率超过90%后，ES将修改为只读状态，所以初步判断是磁盘空间不足导致ES不允许写入。

df -h 查看磁盘使用，发现97%、

 

三、解决办法

首先清理磁盘空间

# 1、进入根目录
cd /

# 2、查看哪个文件夹占用空间最大
du -h --max-depth=1


# 3、之后再进入那个占用大的文件夹，继续执行
du -h --max-depth=1

# 4、一步步排查到占用空间最大的目录

# 5、确认并删除大文件

其次在kibana上执行如下命令

PUT /_all/_settings
{
  "index.blocks.read_only_allow_delete": null
  
}

 

或者不使用Kibana，使用curl

curl -XPUT -H "Content-Type: application/json" http://192.168.0.162:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'