ELK 日常问题

ES磁盘情况：可以通过自带的监控查看磁盘占用情况，以及jvm情况

# 获取所有索引：es机器上，如果是集群，任一es上即可
curl -XGET 'http://192.168.0.56:9200/_cat/indices/?v'|awk '{print $3}'|sort
# 删除索引，如监控的索引，可根据情况，编写定期清理的脚本，比如保存一个月的日志等
curl -XDELETE 'http://192.168.0.56:9200/.monitoring-es-6-2019.04.01'

#!/bin/bash
# 获取所有索引
curl -XGET 'http://192.168.0.57:9200/_cat/indices/?v'|awk '{print $3}'|sort > /root/es.txt
# 删除索引，如监控索引
index_array=(nginx-test nginx-beta nginx-prod tomcat-uc)
for index in ${index_array[@]}
do
    LOG_NAME=`cat /root/es.txt|grep $index|head -1`
    curl -XDELETE http://192.168.0.57:9200/$LOG_NAME
done

出现的问题

1.一旦在存储超过95％的磁盘中的节点上分配了一个或多个分片的任何索引，该索引将被强制进入只读模式

https://www.aityp.com/%E8%A7%A3%E5%86%B3elasticsearch%E7%B4%A2%E5%BC%95%E5%8F%AA%E8%AF%BB/

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

 

2.index has exceeded [1000000] - maximum allowed to be analyzed for highlighting
For large texts, indexing with offsets or term vectors is recommended!

详细的出错内容是这样：

　　　　{"type":"illegal_argument_exception","reason":"The length of [message] field of [l60ZgW0Bv9XMTlnX27A_] doc of [syslog] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!”}}

错误原因：索引偏移量默认是100000，超过了
解决方法：https://www.cnblogs.com/zhanchenjin/p/11672900.html

 

3.circuit_breaking_exception', '[parent] Data too large, data for [<http_request>] would be [246901928/235.4mb], which is larger than the limit of [246546432/235.1mb]
 

详细的出错内容是这样：

　　　　elasticsearch.exceptions.TransportError: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large, data for [<http_request>] would be [246901928/235.4mb], which is larger than the limit of [246546432/235.1mb], real usage: [246901768/235.4mb], new bytes reserved: [160/160b], usages [request=0/0b, fielddata=11733/11.4kb, in_flight_requests=160/160b, accounting=6120593/5.8mb]')

 　　

　　错误原因：

　　　　堆内存不够当前查询加载数据所以会报 https://github.com/docker-library/elasticsearch/issues/98

 　　解决方案：

• 提高堆栈内存

　　　　在宿主机执行：sudo sysctl -w vm.max_map_count=262144

　　　　docker增加命令参数设置java的虚拟机初始化堆栈大小1G，和最大堆栈大小3G