大家好:
最近发现,cdn日志,处理进es,但是,不能正确查询到结果,求解答。
cdn日志是每100行正常日志上报到服务器,所以,单行日志可能会很大,request_body中可能好多日志的集合,kv形式的。
es版本为:1.6.0,写入es的工具用的childe的hangout
怀疑是,es写入了单个term大于32k了,然后无法查询。
通过kibana页面查询,页面报错是:Courier Fetch: 2 of 4 shards failed.
查看es master的报错是:
[2015-12-18 16:21:08,696][DEBUG][action.search.type ] [bes01] [1305683] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [bes04][inet[/10.10.10.166:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: [logstash-es-img-2015.12.17][1]: query[filtered(did:02f715e2072fc7265b14f4470f565b2b)->BooleanFilter(+cache(@timestamp:[1450167667735 TO 1450426867736]))],from[0],size[298],sort[!]: Fetch Failed [Failed to highlight field [request_body]]
at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:133)
at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:133)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:504)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 43666
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:493)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:396)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:371)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:351)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getLeafContext(WeightedSpanTermExtractor.java:360)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:216)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
at org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
at org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:119)
... 10 more
Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 43666
at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:467)
... 20 more
[2015-12-18 16:21:08,766][DEBUG][action.search.type ] [bes01] [1307503] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [bes03][inet[/10.10.10.165:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: [logstash-es-img-2015.12.17][3]: query[filtered(did:02f715e2072fc7265b14f4470f565b2b)->BooleanFilter(+cache(@timestamp:[1450167667735 TO 1450426867736]))],from[0],size[298],sort[!]: Fetch Failed [Failed to highlight field [request_body]]
at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:133)
at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:133)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:504)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 42779
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:493)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:396)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:371)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:351)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getLeafContext(WeightedSpanTermExtractor.java:360)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:216)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
at org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
at org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:119)
... 10 more
Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 42779
at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:467)
... 20 more
[2015-12-18 16:21:09,507][DEBUG][action.search.type ] [bes01] [1307427] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [bes06][inet[/10.10.10.168:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: [logstash-es-img-2015.12.16][1]: query[filtered(did:02f715e2072fc7265b14f4470f565b2b)->BooleanFilter(+cache(@timestamp:[1450167667735 TO 1450426867736]))],from[0],size[288],sort[!]: Fetch Failed [Failed to highlight field [request_body]]
at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:133)
at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:133)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:504)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 42325
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:493)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:396)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:371)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:351)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getLeafContext(WeightedSpanTermExtractor.java:360)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:216)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
at org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
at org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:119)
... 10 more
Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 42325
at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:467)
... 20 more
[2015-12-18 16:21:09,547][DEBUG][action.search.type ] [bes01] [1305672] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [bes05][inet[/10.10.10.167:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: [logstash-es-img-2015.12.16][0]: query[filtered(did:02f715e2072fc7265b14f4470f565b2b)->BooleanFilter(+cache(@timestamp:[1450167667735 TO 1450426867736]))],from[0],size[288],sort[!]: Fetch Failed [Failed to highlight field [request_body]]
at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:133)
at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:133)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:504)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 44259
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:493)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:396)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:371)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:351)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getLeafContext(WeightedSpanTermExtractor.java:360)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:216)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
at org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
at org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:119)
... 10 more
Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 44259
at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:467)
... 20 more
[2015-12-18 16:21:09,587][DEBUG][action.search.type ] [bes01] [1305685] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [bes04][inet[/10.10.10.166:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: [logstash-es-img-2015.12.16][2]: query[filtered(did:02f715e2072fc7265b14f4470f565b2b)->BooleanFilter(+cache(@timestamp:[1450167667735 TO 1450426867736]))],from[0],size[288],sort[!]: Fetch Failed [Failed to highlight field [request_body]]
at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:133)
at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:133)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:504)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 32889
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:493)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:396)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:371)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:351)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getLeafContext(WeightedSpanTermExtractor.java:360)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:216)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
at org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
at org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:119)
... 10 more
Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 32889
at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:467)
... 20 more