flume采集kafka数据到hdfs报如下错误

运行Flume没多久就报下面的异常：

2016-08-24 17:35:58,927 (Flume Thrift IPC Thread 8) [ERROR - org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:196)] Error while writing to required channel: org.apache.flume.channel.MemoryChannel{name: memoryChannel}
2016-08-24 17:35:59,332 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - kafka.utils.Logging$class.error(Logging.scala:97)] Failed to collate messages by topic, partition due to: GC overhead limit exceeded
2016-08-24 17:35:59,332 (Flume Thrift IPC Thread 8) [ERROR - org.apache.thrift.ProcessFunction.process(ProcessFunction.java:41)] Internal error processing appendBatch
java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.util.Arrays.copyOf(Arrays.java:3332)
    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
    at java.lang.StringBuffer.append(StringBuffer.java:272)
    at java.io.StringWriter.write(StringWriter.java:112)
    at java.io.PrintWriter.write(PrintWriter.java:456)
    at java.io.PrintWriter.write(PrintWriter.java:473)
    at java.io.PrintWriter.print(PrintWriter.java:603)
    at java.io.PrintWriter.println(PrintWriter.java:756)
    at java.lang.Throwable$WrappedPrintWriter.println(Throwable.java:764)
    at java.lang.Throwable.printStackTrace(Throwable.java:658)
    at java.lang.Throwable.printStackTrace(Throwable.java:721)
    at org.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:60)
    at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:87)
    at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:413)
    at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:313)
    at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
    at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
    at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
    at org.apache.log4j.Category.callAppenders(Category.java:206)
    at org.apache.log4j.Category.forcedLog(Category.java:391)
    at org.apache.log4j.Category.log(Category.java:856)
    at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:576)
    at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:196)
    at org.apache.flume.source.ThriftSource$ThriftSourceHandler.appendBatch(ThriftSource.java:457)
    at org.apache.flume.thrift.ThriftSourceProtocol$Processor$appendBatch.getResult(ThriftSourceProtocol.java:259)
    at org.apache.flume.thrift.ThriftSourceProtocol$Processor$appendBatch.getResult(ThriftSourceProtocol.java:247)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
    at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
    at org.apache.thrift.server.Invocation.run(Invocation.java:18)

显然是我采集的数据太大了，导致Flume的JVM内存不够用。

用 ps -aux | grep flume查找Flume进程也能看到Flume使用了多少内存。

解决方法：

1、 vim bin/flume-ng

　　在里面找到JAVA_OPTS="-Xmx20m"，它默认启动时最大可用内存为20，只要将其调大一点就可以了。

2、 或者在Flume conf目录下找到flume-env.sh.template文件

　　cp flume-env.sh.template flume-env.sh

　　vim flume-env.sh

　　把下面这句配置的注释删掉就可了

　　# export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"

需要cd /etc/flume-ng/conf

cp flume-env.sh.template flume-env.sh

修改export JAVA_OPTS调大即可！！！

当使用flume-ng进行日志采集的时候，如果日志文件很大，容易导致flume出现：

java.lang.OutOfMemoryError: Java heap space

这样的错误，这就需要调整flume相应的jvm启动参数。

修改 flume下的conf/flume-env.sh文件：

 export JAVA_OPTS="-Xms512m -Xmx1024m -Dcom.sun.management.jmxremote"

其中：
-Xms<size>        set initial Java heap size.........................
-Xmx<size>        set maximum Java heap size.........................

主要修改Xmx和Xms两个参数,可以根据OS内存的大小进行合理设置,一般一个flume agent 1G左右大小即可