当flume接收kafka消息过多会导致如下错误:
Exception in thread "PollableSourceRunner-KafkaSource-s1" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:149)
at java.lang.StringCoding.decode(StringCoding.java:193)
at java.lang.String.<init>(String.java:426)
at java.lang.String.<init>(String.java:491)
at org.apache.kafka.common.serialization.StringDeserializer.deserialize(StringDeserializer.java:47)
at org.apache.kafka.common.serialization.StringDeserializer.deserialize(StringDeserializer.java:28)
at org.apache.kafka.common.serialization.ExtendedDeserializer$Wrapper.deserialize(ExtendedDeserializer.java:65)
at org.apache.kafka.common.serialization.ExtendedDeserializer$Wrapper.deserialize(ExtendedDeserializer.java:55)
at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:1038)
at org.apache.kafka.clients.consumer.internals.Fetcher.access$3300(Fetcher.java:110)
at org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.fetchRecords(Fetcher.java:1223)
at org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.access$1400(Fetcher.java:1072)
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:562)
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:523)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1230)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1187)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1154)
at org.apache.flume.source.kafka.KafkaSource.doProcess(KafkaSource.java:216)
at org.apache.flume.source.AbstractPollableSource.process(AbstractPollableSource.java:60)
at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133)
at java.lang.Thread.run(Thread.java:748)
[2020-04-22 21:52:50,985] WARN Sink failed to consume event. Attempting next sink if available. (org.apache.flume.sink.LoadBalancingSinkProcessor)
org.apache.flume.EventDeliveryException: Failed to send events
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:398)
at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:156)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: slave1, port: 52020 }: Failed to send batch
at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:310)
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:380)
... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: slave1, port: 52020 }: RPC request exception
at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:360)
at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:298)
... 4 more
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:352)
... 5 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.ArrayList.iterator(ArrayList.java:840)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:103)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:131)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:68)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
at org.apache.avro.ipc.specific.SpecificRequestor.writeRequest(SpecificRequestor.java:127)
at org.apache.avro.ipc.Requestor$Request.getBytes(Requestor.java:473)
at org.apache.avro.ipc.Requestor.request(Requestor.java:181)
at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:84)
at com.sun.proxy.$Proxy6.appendBatch(Unknown Source)
at org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:343)
at org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:339)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
[2020-04-22 21:52:53,746] INFO Rpc sink k1: Building RpcClient with hostname: slave1, port: 52020 (org.apache.flume.sink.AbstractRpcSink)
如下图所示:
或者:
Exception in thread "PollableSourceRunner-KafkaSource-s1" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "PollableSourceRunner-KafkaSource-s1"
Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.OutOfMemoryError: GC overhead limit exceeded
如下图所示:
出现以上这种情况主要是由于flume的jvm设置的最大内存默认为20M,只需要改一下这个值即可,如下图所示: