es 错误处理(this cluster currently has [3002]/[3000] maximum shards open)

问题：ES 又莫名其妙宕机了

查看 Elasticsearch 日志如下

[2022-03-22T09:00:02,430][WARN ][o.e.x.m.e.l.LocalExporter] [aibee-devops-es03] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: [.monitoring-es-7-2022.03.22] IndexCreationException[failed to create index [.monitoring-es-7-2022.03.22]]; nested: ValidationException[Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [3002]/[3000] maximum shards open;];
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:125) ~[x-pack-monitoring-7.1.1.jar:7.1.1]

说明：ElasticSearch 7.x 开始，集群默认的分片数是 1000 个

根据错误提示显然是分片数限制，无法再增加新的分片，导致 ElasticSearch down 掉了

我们可以通过 ElasticSearch API 来增大分片数限制

curl \
-X PUT \
-H "Content-Type:application/json" \
-u elastic:password \
-d '{
        "transient":{
            "cluster":{
                "max_shards_per_node":90000
            }
        }
    }'  
'http://localhost:9200/_cluster/settings'

通过 Kibana 来修改（推荐）

# 临时增加分片数（重启es会设置时效）
PUT /_cluster/settings
{
  "transient": {
    "cluster": {
      "max_shards_per_node":900000
    }
  }
}

# 永久增加分片数（推荐）
PUT /_cluster/settings
{
  "persistent": {
    "cluster": {
      "max_shards_per_node":900000
    }
  }
}

查看分片数

GET /_cluster/settings?pretty

结果

{
  "persistent" : {
    "cluster" : {
      "max_shards_per_node" : "900000"
    },
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : {
    "cluster" : {
      "max_shards_per_node" : "900000"
    }
  }
}

参考

https://www.elastic.co/guide/en/elasticsearch/reference/7.1/modules-cluster.html
https://www.elastic.co/guide/en/elasticsearch/reference/7.1/cluster-update-settings.html