记一次Elasticsearch连接慢的故障

昨天上午接到研发通知,告知Elasticsearch连接不上,ES集群为1主3从,登录主节点查看进程在,9200端口已无监听,查看最后的日志为:

[2020-04-27T09:59:48,063][ERROR][i.n.u.c.D.rejectedExecution] [ES-001] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
    at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:867) ~[?:?]
    at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:328) ~[?:?]
    at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:321) ~[?:?]
    at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:778) ~[?:?]
    at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:768) ~[?:?]
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:432) ~[?:?]
    at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:112) ~[?:?]

这时重启ES主节点,主节点启动成功后,等待恢复分片,监控发现分片恢复很慢,集群连接也非常慢,集群共404个主分片,一上午才恢复了40%,后继续找原因,比如修改参数:

cluster.routing.allocation.node_concurrent_recoveries: 10    #默认为2

观察只是提高了分片并发数,速度还是一样慢,继续查看26号的日志发现很多如下一样的日志:

[2020-04-26T19:26:01,823][DEBUG][o.e.a.s.TransportSearchAction] [ES-001] [37660] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [ES-004][172.28.231.28:9300][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [37660]
[2020-04-26T19:26:01,835][DEBUG][o.e.a.s.TransportSearchAction] [ES-001] [161303] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [ES-002][172.28.231.25:9300][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [161303]
[2020-04-26T19:26:01,856][DEBUG][o.e.a.s.TransportSearchAction] [ES-001] [37649] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [ES-004][172.28.231.28:9300][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [37649]
[2020-04-27T02:00:48,750][DEBUG][o.e.a.s.TransportSearchAction] [ES-001] [78812] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [ES-003][172.28.231.17:9300][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [78812]

原因:

从节点每台机都重复启动了,端口只有一个在监听,但是进程有2个

解决:

杀掉所有ES进程,重启恢复正常

  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

hblicy

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值