Spark,Hadoop交流群,群QQ号:521066396,欢迎加入共同学习,一起进步~
1、问题
使用Flume遇到如下报错:
java.lang.OutOfMemoryError: GC overhead limit exceeded
或者:
java.lang.OutOfMemoryError: Java heap space
Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.OutOfMemoryError: Java heap space
2、解决方法
# Enviroment variables can be set here.
export JAVA_HOME=/usr/java/jdk1.8.0_51
# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
export JAVA_OPTS="-Xms8192m -Xmx8192m -Xss256k -Xmn2g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit"
注意:在启动 agent 的时候一定要带上 -c conf 选项,否则 flume-env.sh 里配置的环境变量不会被加载生效。
例如:
./flume-ng agent --name tier1 --conf ../etc/flume-ng/conf.dist/ --conf-file ../etc/flume-ng/conf.dist/kafkatohdfs.conf -f ../etc/flume-ng/conf.dist/flume-conf.properties