1、到官方网上下载apache-flume-1.4.0-bin.tar.gz
2、解压安装包
tar -zxvf apache-flume-1.4.0-bin.tar.gz
3、配置环境变量
export FLUME_HOME=/root/install/apache-flume-1.4.0-bin
export PATH=$PATH:$FLUME_HOME/bin
4、让配置文件生效
source /etc/profile
5、编写一个测试案例
(1)在$FLUME_HOME/conf/目录下新建文件example-conf.properties,其内容如下
- <span style="font-size:18px;"><span style="font-size:18px;"># Describe the source
- a1.sources = r1
- a1.sinks = k1
- a1.channels = c1
- # Describe/configure the source
- a1.sources.r1.type = avro
- a1.sources.r1.bind = localhost
- a1.sources.r1.port = 44444
- # Describe the sink
- a1.sinks.k1.type = logger
- # Use a channel which buffers events in memory
- a1.channels.c1.type = memory
- a1.channels.c1.capacity = 1000
- a1.channels.c1.transactionCapacity = 100
- # Bind the source and sink to the channel
- a1.sources.r1.channels = c1
- a1.sinks.k1.channel = c1</span></span>
- <span style="font-size:18px;"><span style="font-size:18px;">echo "hello world" >file-test.txt</span></span>
- <span style="font-size:18px;"><span style="font-size:18px;">flume-ng agent -n a1 -f example-conf.properties</span></span>
(4)另开一个窗口启动avro-client客户端向agent代理发送数据(以下的localhost目前照这个例子还不能写成ip地址)
- <span style="font-size:18px;"><span style="font-size:18px;">flume-ng avro-client -H localhost -p 44444 -F file-test.txt</span></span>
从上图输出结果可以看出avro-client客户端发来的数据已经被agent代理接收到,在本例中的配置中,设置的sink类型为logger,其输出结果会保存在日志中
问题 总结
1、出现如下错误,解决办法:把guava-10.0.1.jar需要换成guava-11.0.2.jar
- <span style="font-size:18px;">Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoSuchMethodError: com.google.common.cache.CacheBuilder.build()Lcom/google/common/cache/Cache;
- at org.apache.hadoop.hdfs.DomainSocketFactory.<init>(DomainSocketFactory.java:45)
- at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:517)
- at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453)
- </span>
- <span style="font-size:18px;">Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoSuchMethodError: com.google.common.cache.CacheBuilder.build()Lcom/google/common/cache/Cache;
- at org.apache.hadoop.hdfs.DomainSocketFactory.<init>(DomainSocketFactory.java:45)
- at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:517)
- </span>
4、配置动态目录时报错
- <span style="font-size:18px;">java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null</span>
- <span style="font-size:18px;">agent1.sinks.hdfssink1.hdfs.path = hdfs://mycluster/flume-data/%y-%m-%d/%H%M
- agent1.sinks.hdfssink1.hdfs.filePrefix = accesslog
- agent1.sinks.hdfssink1.hdfs.useLocalTimeStamp = true</span>
5、相关参数说明
(1)以下3个参数搭配动态生成目录
- <span style="font-size:18px;">agent1.sinks.sink2hdfs.hdfs.round = true
- agent1.sinks.sink2hdfs.hdfs.roundValue = 60
- agent1.sinks.sink2hdfs.hdfs.roundUnit = minute</span>
- <span style="font-size:18px;">agent1.sinks.sink2hdfs.hdfs.writeFormat = Text
- agent1.sinks.sink2hdfs.hdfs.fileType = DataStream
- #agent1.sinks.sink2hdfs.hdfs.codeC = gzip</span>
- <span style="font-size:18px;">agent1.sinks.sink2hdfs.hdfs.path = hdfs://mycluster/flume/%Y-%m-%d/%H
- agent1.sinks.sink2hdfs.hdfs.filePrefix = consolidation-accesslog-%H-%M-%S
- agent1.sinks.sink2hdfs.hdfs.useLocalTimeStamp = true</span>
- <span style="font-size:18px;">agent1.sinks.sink2hdfs.type = hdfs</span>
(5)关于Flume内存溢出的问题,此时会报各种莫名奇妙的异常,异常如下
- <span style="font-size:18px;">org.apache.avro.AvroRuntimeException: Unknown datum type: java.lang.Exception: java.lang.OutOfMemoryError: GC overhead limit exceeded
- 14/10/17 09:56:06 WARN ipc.NettyServer: Unexpected exception from downstream.
- java.lang.OutOfMemoryError: Java heap space
- 14/10/17 09:50:31 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
- java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
- Exception in thread "pool-17-thread-6" java.lang.NoClassDefFoundError: Could not initialize class java.text.MessageFormat
- EXCEPTION: java.lang.OutOfMemoryError: GC overhead limit exceeded)
- java.lang.OutOfMemoryError: GC overhead limit exceeded
- Error while writing to required channel: org.apache.flume.channel.MemoryChannel{name:
- AvroRuntimeException: Excessively large list allocation request detected: 1398022191 items! Connection closed.
- </span>
# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
#JAVA_OPTS="-Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote" 这个参数,可是怎么改都不管用。都是没有明显的说出原因,还说修改什么limits.conf文件,什么解释都有很乱。
第二天我来上班接着排查:
解决办法:
<1>ps -aux|grep flume 查看刚才启动的flume进程,发现如下信息
- <span style="font-size:18px;">/usr/jdk/bin/java -Xmx20m -Dflume.root.logger=INFO -cp conf</span>
- <span style="font-size:18px;">JAVA_OPTS="-Xmx20m"</span>
- <span style="font-size:18px;">JAVA_OPTS="-Xmx2048m"</span>
6.自己写的plugin不能直接放到plugin.d目录下,而是要自己子目录,我当时直接把自己写的jar放到plugin.d目录下还是找不到,所以又再建子目录。
- <span style="font-size:18px;">[root@storm1 apache-flume-1.4.0-bin]# ls
- bin CHANGELOG conf DEVNOTES docs lib LICENSE NOTICE plugins.d README RELEASE-NOTES tools
- [root@storm1 apache-flume-1.4.0-bin]# cd plugins.d/panguoyuan/lib/
- [root@storm1 lib]# ls
- flume-ng-ext.jar</span>
7.采集端的sink的type=avro,汇总端的source的type如果为netcat的话会报如下错误
解决办法:把source的type改为avro
- <span style="font-size:18px;">2015-03-11 11:40:44,144 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
- org.apache.flume.EventDeliveryException: Failed to send events
- at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:382)
- at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
- at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
- at java.lang.Thread.run(Thread.java:744)
- Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: fk01, port: 44444 }: Failed to send batch
- at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:294)
- at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:366)
- ... 3 more
- Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: fk01, port: 44444 }: Handshake timed out after 20000ms
- at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:338)
- at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:282)
- ... 4 more
- Caused by: java.util.concurrent.TimeoutException
- at java.util.concurrent.FutureTask.get(FutureTask.java:201)
- at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:336)
- ... 5 more
- 2015-03-11 11:40:49,147 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:205)] Rpc sink avro_sink: Building RpcClient with hostname: 10.58.22.219, port: 44444</span>