Elasticsearch运行过程中常见错误

一、安装成功后无法查询

排除步骤：

• 排查elasticsearch，查看是否正常运行，有无报错信息
• 如果elasticsearch运行正常，重启正常，但依然无法访问9200端口，需确认防火墙问题
• 防火墙开放端口后使用下面命令查看elasticsearch状态：
  curl -XGET 'http://localhost:9200/_cluster/health?pretty'
  1. 查看该条命令执行后的输出结果，如果输出curl连接有问题，则elasticsearch未启动起来，需进一步看日志确认启动异常问题。
  2. 如果输出为status：red，则说明elasticsearch运行起来了，但状态不可用，需要排查日志确认原因。
  3. 如果输出为status：yellow或green，则表示elasticsearch运行正常
• 启动正常，状态也正常，依然无法连接查询，可以执行一下
  curl -XGET 127.0.0.1:9200/index/_mapping # index为你环境设置的索引
  如果得到的结果是error，"index_not_found_exception"则表示今天的索引未生成，需要进一步排查filebeat和logstash是否正常。

二、运行一段时间后出现频繁挂死的情况，重启后正常，过段时间继续挂死

报错如下：

[2019-06-24T16:28:17,441][WARN ][o.e.m.j.JvmGcMonitorService][gc][old][189106][34072] duration [12.7m], collections [54]/[12.7m], total [12.7m]/[3.3d], memory [3.8gb]->[3.8gb]/[3.8gb], all_pools {[young] [1gb]->[1gb]/[1gb]}{[survivor] [136.1mb]->[96.1mb]/[136.5mb]}{[old] [2.6gb]->[2.6gb]/[2.6gb]}
[2019-06-24T16:28:17,441][WARN ][o.e.m.j.JvmGcMonitorService][gc][189106] overhead, spent [12.7m] collecting in the last [12.7m]
[2019-06-24T16:28:17,466][ERROR][o.e.i.e.Engine           ]  [index][2] lucene commit failed
java.lang.OutOfMemoryError: Java heap space
    at org.apache.lucene.util.fst.BytesStore.writeByte(BytesStore.java:89) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
    at org.apache.lucene.util.fst.FST.<init>(FST.java:295) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
 ...

java.lang.OutOfMemoryError: Java heap space，内存溢出，此时需要修改jvm.options 中的内存大小，修改-Xms4g -Xmx4g两个值，修改完重启elasticsearch即可。

三、查询elasticsearch报错

报错如下：

nested: NotSerializableExceptionWrapper[too_many_clauses: maxClauseCount is set to 1024];
    at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:329)
    at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:312)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:617)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:485)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:461)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)
    at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343)
    at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
    at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654)
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    ... 3 more
Caused by: NotSerializableExceptionWrapper[too_many_clauses: maxClauseCount is set to 1024]
    at org.apache.lucene.search.BooleanQuery$Builder.add(BooleanQuery.java:125)
    at org.apache.lucene.search.BooleanQuery$Builder.add(BooleanQuery.java:138)
    at org.elasticsearch.index.query.TermsQueryBuilder.handleTermsQuery(TermsQueryBuilder.java:453)
    at org.elasticsearch.index.query.TermsQueryBuilder.doToQuery(TermsQueryBuilder.java:414)
    at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
    at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:444)
    at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:418)
    at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
    at org.elasticsearch.index.query.QueryShardContext.lambda$toQuery$1(QueryShardContext.java:313)
    at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:325)

该报错是查询条件数过多导致的，elasticsearch默认查询条件限制是1024，建议优化查询条件语句，也可通过修改配置避免elasticsearch.yml 添加：
indices.query.bool.max_clause_count: 10240 #冒号后有空格，否则无法启动elasticsearch
该配置把查询条件数限制改大到10240，改完重启elasticsearch。

四、如果logstash设置的模板有经常变动的内容，则数据过多是elasticsearch会挂死：

java.lang.IllegalArgumentException: Limit of total fields [1000] in index [bh_2019-03-25] has been exceeded
        at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:604) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:420) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:336) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:268) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:311) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) [elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.10.jar:5.6.10]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]

查看之前的日志会发现，elasticsearch一直在update mapping，这个报错的原因是fields过多，超过了1000，虽然也可以通过配置修改解决，但频繁增加不同的key在一定程度上说明模板设置有问题，建议调整logstash的入库模板。