java sql超过32k_查询cdn日志报错，单term日志大于32K，看报错是成功写入了es，但是，不能查询出来...

最新推荐文章于 2023-03-28 13:51:44 发布

弦歌缓缓

最新推荐文章于 2023-03-28 13:51:44 发布

阅读量221

点赞数

文章标签： java sql超过32k

本文链接：https://blog.csdn.net/weixin_28750421/article/details/114207841

版权

大家好：

最近发现，cdn日志，处理进es，但是，不能正确查询到结果，求解答。

cdn日志是每100行正常日志上报到服务器，所以，单行日志可能会很大，request_body中可能好多日志的集合，kv形式的。

es版本为：1.6.0，写入es的工具用的childe的hangout

怀疑是，es写入了单个term大于32k了，然后无法查询。

通过kibana页面查询，页面报错是：Courier Fetch: 2 of 4 shards failed.

查看es master的报错是：

[2015-12-18 16:21:08,696][DEBUG][action.search.type ] [bes01] [1305683] Failed to execute fetch phase

org.elasticsearch.transport.RemoteTransportException: [bes04][inet[/10.10.10.166:9300]][indices:data/read/search[phase/fetch/id]]

Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: [logstash-es-img-2015.12.17][1]: query[filtered(did:02f715e2072fc7265b14f4470f565b2b)->BooleanFilter(+cache(@timestamp:[1450167667735 TO 1450426867736]))],from[0],size[298],sort[!]: Fetch Failed [Failed to highlight field [request_body]]

at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:133)

at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:133)

at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194)

at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:504)

at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)

at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)

at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.RuntimeException: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 43666

at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:493)

at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:396)

at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:371)

at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:351)

at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getLeafContext(WeightedSpanTermExtractor.java:360)

at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:216)

at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)

at org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)

at org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)

at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)

at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:119)

... 10 more

Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 43666

at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)

at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:467)

... 20 more

[2015-12-18 16:21:08,766][DEBUG][action.search.type ] [bes01] [1307503] Failed to execute fetch phase

org.elasticsearch.transport.RemoteTransportException: [bes03][inet[/10.10.10.165:9300]][indices:data/read/search[phase/fetch/id]]

Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: [logstash-es-img-2015.12.17][3]: query[filtered(did:02f715e2072fc7265b14f4470f565b2b)->BooleanFilter(+cache(@timestamp:[1450167667735 TO 1450426867736]))],from[0],size[298],sort[!]: Fetch Failed [Failed to highlight field [request_body]]