kudu生产问题汇总

最新推荐文章于 2023-09-07 07:03:01 发布

远方有海，小样不乖

最新推荐文章于 2023-09-07 07:03:01 发布

阅读量574

点赞数

分类专栏：大数据文章标签：大数据 Powered by 金山文档

本文链接：https://blog.csdn.net/anguoan/article/details/129129095

版权

大数据专栏收录该内容

27 篇文章 0 订阅

订阅专栏

一.kudu内存限制问题导致写入数据失败

内存限制问题（Memory Limits）：
Service unavailable: Soft memory limit exceeded (at 105.52% of capacity)

问题出现原因：
Kudu有一个硬性和软性的内存限制。

硬存储器限制是Kudu进程允许使用的最大数量，由--memory_limit_hard_bytes标志控制。（我们本次是这种情况，之前是默认配置1GB（偏小导致数据无法写入），此次修改为20GB恢复正常写入）

软内存限制是由硬盘内存限制的一个百分比，由--memory_limit_soft_percentage标志控制，memory_limit_soft_percentage默认值为80％，决定进程在开始拒绝某些写入操作之前可能使用的内存量。
那么Kudu就会因为记忆背压而拒绝写入。这可能会导致写入超时。有几种方法可以缓解库杜的内存压力：

解决方案：
1.如果主机有更多的内存可用于Kudu，增加–memory_limit_hard_bytes。
2.通过增加磁盘数量或增加维护管理器线程（此线程负责将内存中的数据刷入磁盘）的数量，–maintenance_manager_num_threads。
通常，维护管理器线程与数据目录的建议比率是1：3。减少在应用程序端流向Kudu的写入量。

二.kudu client超时问题

2019-09-28 05:19:53.467 [pool-10-thread-1] ERROR c.a.o.canal.client.adapter.kudu.service.KuduSyncService - org.apache.kudu.client.NonRecoverableException: RPC can not complete before timeout: KuduRpc(method=GetTableSchema, tablet=Kudu Master, attempt=47, DeadlineTracker(timeout=30000, elapsed=29831), Traces: [0ms] sending RPC to server , [4ms] delaying RPC due to Illegal state: Not the leader. Local UUID: d46fb8e8eb65404e9a98cb02af727131, Consensus state: current_term: 6 config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7d5c6a9d684b14832fe9fd3179fd66" member_type: VOTER last_known_addr { host: "olapv06.nbike.songguo7.com.ucb" port: 7051 } } peers { permanent_uuid: "65229578233e41ee8e3289d98de9f316" member_type: VOTER
......
Caused by: org.apache.kudu.client.RecoverableException: Not the leader. Local UUID: d46fb8e8eb65404e9a98cb02af727131, Consensus state: current_term: 6 leader_uuid: "6f7d5c6a9d684b14832fe9fd3179fd66" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7d5c6a9d684b14832fe9fd3179fd66" member_type: VOTER last_known_addr { host: "olapv06.nbike.songguo7.com.ucb" port: 7051 } } peers { permanent_uuid: "65229578233e41ee8e3289d98de9f316" member_type: VOTER last_known_addr { host: "olapv07.nbike.songguo7.com.ucb" port: 7051 } } peers { permanent_uuid: "d46fb8e8eb65404e9a98cb02af727131" member_type: VOTER last_known_addr { host: "olapv08.nbike.songguo7.com.ucb" port: 7051 } } }
... 40 common frames omitted
2019-09-28 05:19:56.646 [pool-6-thread-1] ERROR c.a.o.canal.client.adapter.kudu.service.KuduSyncService - org.apache.kudu.client.NonRecoverableException: RPC can not complete before timeout: KuduRpc(method=GetTableSchema, tablet=Kudu Master, attempt=34, DeadlineTracker(timeout=30000, elapsed=28691), Traces: [0ms] querying master, [0ms] Sub rpc: ConnectToMaster sending RPC to server master-olapv06.nbike.songguo7.com.ucb:7051, [0ms] Sub rpc: ConnectToMaster sending RPC to server master-olapv07.nbike.songguo7.com.ucb:7051, [0ms] Sub rpc: ConnectToMaster sending RPC to server master-olapv08.nbike.songguo7.com.ucb:7051, [1ms] Sub rpc: ConnectToMaster received from server master-olapv06.nbike.songguo7.com.ucb:7051 response OK, [1ms] sending RPC to server , [1ms] Sub rpc: ConnectToMaster received from server master-olapv07.nbike.songguo7.com.ucb:

观察kudu master日志，发现client 这边超时的时候 master那边刚刚选举出新的leader, 怀疑选举的过程影响了client 的正常工作

三.kudu client 突然无法访问kudu server

2019-10-04 04:17:21.576 [New I/O worker #3] INFO org.apache.kudu.client.AsyncKuduClient - Removing server 4ac2005dce9847e58eb59704cf7f9311 from this tablet's cache d20c6cb2482a44bca84c06a67f0d4c8f
2019-10-04 10:27:27.828 [New I/O worker #3] ERROR org.apache.kudu.client.TabletClient - [Peer 1a831bc46172402bb1bfc19f6165a302] Unexpected exception from downstream on [id: 0xf81fc375, /10.9.86.193:37228 => olapv07.nbike.songguo7.com.ucb/10.100.125.111:7050]
java.lang.RuntimeException: Could not deserialize the response, incompatible RPC? Error is: step
at org.apache.kudu.client.KuduRpc.readProtobuf(KuduRpc.java:383)
at org.apache.kudu.client.Negotiator.parseSaslMsgResponse(Negotiator.java:282)
at org.apache.kudu.client.Negotiator.handleResponse(Negotiator.java:235)
at org.apache.kudu.client.Negotiator.messageReceived(Negotiator.java:229)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.apache.kudu.client.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.apache.kudu.client.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

解决方案：https://gitter.im/cloudera/kudu/archives/2018/01/05