一、现象
在Kibana上一直查询不到索引最新插入的数据,通过排查发现
Logstash 的日志(/logs/logstash-plain.log) 大量报错:FORBIDDEN/12/index read-only / allow delete (api)
[2020-10-27T15:35:12,119][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [index_user_info] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2020-10-27T15:35:12,119][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [index_user_info] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2020-10-27T15:35:12,119][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [index_user_info] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2020-10-27T15:35:12,119][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [index_user_info] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2020-10-27T15:35:12,119][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [index_user_info] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
其中index_user_info 是我的ES索引名称!
二、原因:磁盘空间不够导致
查阅资料,发现ES说明文档中有写明:当ES数据所在目录磁盘空间使用率超过90%后,ES将修改为只读状态,所以初步判断是磁盘空间不足导致ES不允许写入。
df -h 查看磁盘使用,发现97%、
三、解决办法
首先清理磁盘空间
# 1、进入根目录
cd /
# 2、查看哪个文件夹占用空间最大
du -h --max-depth=1
# 3、之后再进入那个占用大的文件夹,继续执行
du -h --max-depth=1
# 4、一步步排查到占用空间最大的目录
# 5、确认并删除大文件
其次在kibana上执行如下命令
PUT /_all/_settings
{
"index.blocks.read_only_allow_delete": null
}
或者不使用Kibana,使用curl
curl -XPUT -H "Content-Type: application/json" http://192.168.0.162:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'