起因:今天,发现管理后台的日志报表中的日志没有数据
于是查看管理日志的服务pluslog服务
{"level":"error","time":"2023-12-28T15:01:10.354+0800","msg":"es InsertLog error","err":"buldRequest Error {\"resp\":\"{\\\"took\\\":1,\\\"errors\\\":true,\\\
"items\\\":[{\\\"index\\\":{\\\"_index\\\":\\\"search_doc_log_v1_2023_12\\\",\\\"_type\\\":\\\"_doc\\\",\\\"_id\\\":\\\"79587380\\\",\\\"status\\\":
400,\\\"error\\\":{\\\"type\\\":\\\"illegal_argument_exception\\\",\\\"reason\\\":\\\"Validation Failed: 1: this action would add [10] total shards, but this
cluster currently has [2992]/[3000] maximum shards open;\\\"}}}]}\"}","es name":"7","log.ID":"9584444920"}
报错显示es错误,查看es服务健康状态
kubectl exec -it es1 -c elasticsearch -- curl -u sgadmin:xxxx '127.0.0.1:9200/_cat/health?v'
发现es分片已经2992,报错显示2992/3000,明显是分片不足了,修改最大分片数设置为90万
问题解决
kubectl exec -it es1 -c elasticsearch -- curl -XPUT -u sgadmin:xxxx 'http://127.0.0.1:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{"persistent":{"cluster":{"max_shards_per_node":900000}}}'