Flume-ng-1.4.0安装及运行遇到问题总结

1、到官方网上下载apache-flume-1.4.0-bin.tar.gz

2、解压安装包

    tar -zxvf apache-flume-1.4.0-bin.tar.gz

3、配置环境变量

export FLUME_HOME=/root/install/apache-flume-1.4.0-bin
export PATH=$PATH:$FLUME_HOME/bin

4、让配置文件生效

source /etc/profile

5、编写一个测试案例

（1）在$FLUME_HOME/conf/目录下新建文件example-conf.properties，其内容如下

<span style="font-size:18px;"><span style="font-size:18px;"># Describe the source 
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1</span></span>

（2）在conf目录下新建一个源文件file-test.txt并向其中写入数据

<span style="font-size:18px;"><span style="font-size:18px;">echo "hello world" >file-test.txt</span></span>

（3）启动agent代理

<span style="font-size:18px;"><span style="font-size:18px;">flume-ng agent -n a1 -f example-conf.properties</span></span>

（4）另开一个窗口启动avro-client客户端向agent代理发送数据(以下的localhost目前照这个例子还不能写成ip地址)

<span style="font-size:18px;"><span style="font-size:18px;">flume-ng avro-client -H localhost -p 44444 -F file-test.txt</span></span>

从上图输出结果可以看出avro-client客户端发来的数据已经被agent代理接收到，在本例中的配置中，设置的sink类型为logger，其输出结果会保存在日志中

问题 总结

1、出现如下错误，解决办法：把guava-10.0.1.jar需要换成guava-11.0.2.jar

<span style="font-size:18px;">Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoSuchMethodError: com.google.common.cache.CacheBuilder.build()Lcom/google/common/cache/Cache;
        at org.apache.hadoop.hdfs.DomainSocketFactory.<init>(DomainSocketFactory.java:45)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:517)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453)
</span>

2、出现如下错误，解决办法：把protobuf-java-2.4.1-shaded.jar换成protobuf-java-2.5.0.jar

<span style="font-size:18px;">Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoSuchMethodError: com.google.common.cache.CacheBuilder.build()Lcom/google/common/cache/Cache;
        at org.apache.hadoop.hdfs.DomainSocketFactory.<init>(DomainSocketFactory.java:45)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:517)
</span>

3、如果Hadoop群集是HA配置需要将core-site.xml和hdfs-site.xml文件复制到$FLUME_HOME/conf/目录下，这才能使用dfs.nameservices

4、配置动态目录时报错

<span style="font-size:18px;">java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null</span>

 解决办法：添加agent1.sinks.hdfssink1.hdfs.useLocalTimeStamp = true

<span style="font-size:18px;">agent1.sinks.hdfssink1.hdfs.path = hdfs://mycluster/flume-data/%y-%m-%d/%H%M
agent1.sinks.hdfssink1.hdfs.filePrefix = accesslog
agent1.sinks.hdfssink1.hdfs.useLocalTimeStamp = true</span>

5、相关参数说明

(1)以下3个参数搭配动态生成目录

<span style="font-size:18px;">agent1.sinks.sink2hdfs.hdfs.round = true
agent1.sinks.sink2hdfs.hdfs.roundValue = 60
agent1.sinks.sink2hdfs.hdfs.roundUnit = minute</span>

(2)以下3个参数搭配来确定输出到hdfs上的数据是什么格式(如，是否压缩)

<span style="font-size:18px;">agent1.sinks.sink2hdfs.hdfs.writeFormat = Text
agent1.sinks.sink2hdfs.hdfs.fileType = DataStream
#agent1.sinks.sink2hdfs.hdfs.codeC = gzip</span>

(3)以下的3个参数用来确定文件输出的目录结构及文件名称

<span style="font-size:18px;">agent1.sinks.sink2hdfs.hdfs.path = hdfs://mycluster/flume/%Y-%m-%d/%H
agent1.sinks.sink2hdfs.hdfs.filePrefix = consolidation-accesslog-%H-%M-%S
agent1.sinks.sink2hdfs.hdfs.useLocalTimeStamp = true</span>

(4)以下参数确定输出端文件系统类型

<span style="font-size:18px;">agent1.sinks.sink2hdfs.type = hdfs</span>

（5）关于Flume内存溢出的问题，此时会报各种莫名奇妙的异常，异常如下

<span style="font-size:18px;">org.apache.avro.AvroRuntimeException: Unknown datum type: java.lang.Exception: java.lang.OutOfMemoryError: GC overhead limit exceeded
14/10/17 09:56:06 WARN ipc.NettyServer: Unexpected exception from downstream.
java.lang.OutOfMemoryError: Java heap space
14/10/17 09:50:31 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
Exception in thread "pool-17-thread-6" java.lang.NoClassDefFoundError: Could not initialize class java.text.MessageFormat
EXCEPTION: java.lang.OutOfMemoryError: GC overhead limit exceeded)
java.lang.OutOfMemoryError: GC overhead limit exceeded
Error while writing to required channel: org.apache.flume.channel.MemoryChannel{name:
AvroRuntimeException: Excessively large list allocation request detected: 1398022191 items! Connection closed.
</span>

这个问题害得我同事通宵到第二天10点了还没搞定，网上查了很多资料都是说设置JVM参数，我们同事还一起修改flume-en.sh里的

# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
#JAVA_OPTS="-Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote" 这个参数，可是怎么改都不管用。都是没有明显的说出原因，还说修改什么limits.conf文件，什么解释都有很乱。

第二天我来上班接着排查：

解决办法:

<1>ps -aux|grep flume 查看刚才启动的flume进程，发现如下信息

<span style="font-size:18px;">/usr/jdk/bin/java -Xmx20m -Dflume.root.logger=INFO -cp conf</span>

<2>于是看一个flume-ng的python脚本，发现

<span style="font-size:18px;">JAVA_OPTS="-Xmx20m"</span>

问题就是出在这里了，然后我把flume-ng脚本里的这个值调大后，一切运行正常了。

<span style="font-size:18px;">JAVA_OPTS="-Xmx2048m"</span>

6.自己写的plugin不能直接放到plugin.d目录下，而是要自己子目录，我当时直接把自己写的jar放到plugin.d目录下还是找不到，所以又再建子目录。

<span style="font-size:18px;">[root@storm1 apache-flume-1.4.0-bin]# ls
bin  CHANGELOG  conf  DEVNOTES  docs  lib  LICENSE  NOTICE  plugins.d  README  RELEASE-NOTES  tools
[root@storm1 apache-flume-1.4.0-bin]# cd plugins.d/panguoyuan/lib/
[root@storm1 lib]# ls
flume-ng-ext.jar</span>

7.采集端的sink的type=avro，汇总端的source的type如果为netcat的话会报如下错误

解决办法：把source的type改为avro

<span style="font-size:18px;">2015-03-11 11:40:44,144 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
        at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:382)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: fk01, port: 44444 }: Failed to send batch
        at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:294)
        at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:366)
        ... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: fk01, port: 44444 }: Handshake timed out after 20000ms
        at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:338)
        at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:282)
        ... 4 more
Caused by: java.util.concurrent.TimeoutException
        at java.util.concurrent.FutureTask.get(FutureTask.java:201)
        at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:336)
        ... 5 more
2015-03-11 11:40:49,147 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:205)] Rpc sink avro_sink: Building RpcClient with hostname: 10.58.22.219, port: 44444</span>

8、flume输出到hdfs之前需要对指定目录授权：hadoop dfs -chmod -R 777 /