一.kudu内存限制问题导致写入数据失败
内存限制问题(Memory Limits):
Service unavailable: Soft memory limit exceeded (at 105.52% of capacity)
问题出现原因:
Kudu有一个硬性和软性的内存限制。
硬存储器限制是Kudu进程允许使用的最大数量,由--memory_limit_hard_bytes标志控制。(我们本次是这种情况,之前是默认配置1GB(偏小导致数据无法写入),此次修改为20GB恢复正常写入)
软内存限制是由硬盘内存限制的一个百分比,由--memory_limit_soft_percentage标志控制,memory_limit_soft_percentage默认值为80%,决定进程在开始拒绝某些写入操作之前可能使用的内存量。
那么Kudu就会因为记忆背压而拒绝写入。这可能会导致写入超时。有几种方法可以缓解库杜的内存压力:
解决方案:
1.如果主机有更多的内存可用于Kudu,增加–memory_limit_hard_bytes。
2.通过增加磁盘数量或增加维护管理器线程(此线程负责将内存中的数据刷入磁盘)的数量,–maintenance_manager_num_threads。
通常,维护管理器线程与数据目录的建议比率是1:3。减少在应用程序端流向Kudu的写入量。
二.kudu client超时问题
2019-09-28 05:19:53.467 [pool-10-thread-1] ERROR c.a.o.canal.client.adapter.kudu.service.KuduSyncService - org.apache.kudu.client.NonRecoverableException: RPC can not complete before timeout: KuduRpc(method=GetTableSchema, tablet=Kudu Master, attempt=47, DeadlineTracker(timeout=30000, elapsed=29831), Traces: [0ms] sending RPC to server , [4ms] delaying RPC due to Illegal state: Not the leader. Local UUID: d46fb8e8eb65404e9a98cb02af727131, Consensus state: current_term: 6 config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7d5c6a9d684b14832fe9fd3179fd66" member_type: VOTER last_known_addr { host: "olapv06.nbike.songguo7.com.ucb" port: 7051 } } peers { permanent_uuid: "65229578233e41ee8e3289d98de9f316" member_type: VOTER
......
Caused by: org.apache.kudu.client.RecoverableException: Not the leader. Local UUID: d46fb8e8eb65404e9a98cb02af727131, Consensus state: current_term: 6 leader_uuid: "6f7d5c6a9d684b14832fe9fd3179fd66" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7d5c6a9d684b14832fe9fd3179fd66" member_type: VOTER last_known_addr { host: "olapv06.nbike.songguo7.com.ucb" port: 7051 } } peers { permanent_uuid: "65229578233e41ee8e3289d98de9f316" member_type: VOTER last_known_addr { host: "olapv07.nbike.songguo7.com.ucb" port: 7051 } } peers { permanent_uuid: "d46fb8e8eb65404e9a98cb02af727131" member_type: VOTER last_known_addr { host: "olapv08.nbike.songguo7.com.ucb" port: 7051 } } }
... 40 common frames omitted
2019-09-28 05:19:56.646 [pool-6-thread-1] ERROR c.a.o.canal.client.adapter.kudu.service.KuduSyncService - org.apache.kudu.client.NonRecoverableException: RPC can not complete before timeout: KuduRpc(method=GetTableSchema, tablet=Kudu Master, attempt=34, DeadlineTracker(timeout=30000, elapsed=28691), Traces: [0ms] querying master, [0ms] Sub rpc: ConnectToMaster sending RPC to server master-olapv06.nbike.songguo7.com.ucb:7051, [0ms] Sub rpc: ConnectToMaster sending RPC to server master-olapv07.nbike.songguo7.com.ucb:7051, [0ms] Sub rpc: ConnectToMaster sending RPC to server master-olapv08.nbike.songguo7.com.ucb:7051, [1ms] Sub rpc: ConnectToMaster received from server master-olapv06.nbike.songguo7.com.ucb:7051 response OK, [1ms] sending RPC to server , [1ms] Sub rpc: ConnectToMaster received from server master-olapv07.nbike.songguo7.com.ucb:
观察kudu master日志,发现client 这边超时的时候 master那边刚刚选举出新的leader, 怀疑选举的过程影响了client 的正常工作
三.kudu client 突然无法访问kudu server
2019-10-04 04:17:21.576 [New I/O worker #3] INFO org.apache.kudu.client.AsyncKuduClient - Removing server 4ac2005dce9847e58eb59704cf7f9311 from this tablet's cache d20c6cb2482a44bca84c06a67f0d4c8f
2019-10-04 10:27:27.828 [New I/O worker #3] ERROR org.apache.kudu.client.TabletClient - [Peer 1a831bc46172402bb1bfc19f6165a302] Unexpected exception from downstream on [id: 0xf81fc375, /10.9.86.193:37228 => olapv07.nbike.songguo7.com.ucb/10.100.125.111:7050]
java.lang.RuntimeException: Could not deserialize the response, incompatible RPC? Error is: step
at org.apache.kudu.client.KuduRpc.readProtobuf(KuduRpc.java:383)
at org.apache.kudu.client.Negotiator.parseSaslMsgResponse(Negotiator.java:282)
at org.apache.kudu.client.Negotiator.handleResponse(Negotiator.java:235)
at org.apache.kudu.client.Negotiator.messageReceived(Negotiator.java:229)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.apache.kudu.client.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.apache.kudu.client.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
解决方案:https://gitter.im/cloudera/kudu/archives/2018/01/05
![](https://i-blog.csdnimg.cn/blog_migrate/c2c313290887cca4d7dc6737f1f5cdad.png)
观察kudu日志与上链接中的日志类似,替换canal客户端中的kudu-client版本至1.5.0
四.关于kudu tablet重启慢的问题解决办法
![](https://i-blog.csdnimg.cn/blog_migrate/32dba1b687ded5054fcc98b1c607ad02.png)
五.kudu DeltaMemStores 内存无法回收问题
https://issues.apache.org/jira/browse/KUDU-2238
![](https://i-blog.csdnimg.cn/blog_migrate/cfbaf42071947f14372d864467cad266.png)
![](https://i-blog.csdnimg.cn/blog_migrate/0a97dbfc069299a48cd3dd4a29e07dab.png)