Flume单机安装并且测试

本文使用的是：apache-flume-1.5.0-bin.tar.gz

解压

[sparkadmin@hadoop4 ~]$ tar -zxvf apache-flume-1.5.0-bin.tar.gz

进行外关联

[sparkadmin@hadoop4 ~]$ ln -s apache-flume-1.5.0-bin flume

[sparkadmin@hadoop4 ~]$ cd flume/conf/
[sparkadmin@hadoop4 conf]$ ll
total 16
-rw-r--r-- 1 sparkadmin sparkadmin 1661 Mar 29  2014 flume-conf.properties.template
-rw-r--r-- 1 sparkadmin sparkadmin 1197 Mar 29  2014 flume-env.sh.template
-rw-r--r-- 1 sparkadmin sparkadmin 3063 Mar 29  2014 log4j.properties

[sparkadmin@hadoop4 conf]$ touch example.conf
[sparkadmin@hadoop4 conf]$ vi example.conf

#命名此代理上的组件

a1.sources=r1
a1.sinks=k1
a1.channels=c1


#source 完成对日志数据的收集

a1.sources.r1.type=netcat
a1.sources.r1.bind=hadoop4
a1.sources.r1.port=44444


#channel 主要提供一个队列的功能，对source提供的数据进行简单的缓存。

#它是一个内存缓冲队列，容量是1000条数据，满100条就会被sink处理掉
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100


#sink 取出channel中的数据，进行相应的存储
a1.sinks.k1.type=logger


#Bind the source and sink to the channel
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1

Netcat source 在某一端口上进行侦听，它将每一行文字变成一个事件源，也就是数据是基于换行符分隔。

它打开一个指定端口，侦听数据将每一行文字变成Flume事件，并通过连接通道发送。

运行：
[sparkadmin@hadoop4 flume]$ bin/flume-ng agent --conf conf --conf-file conf/example.conf --name a1 -Dflume.root.logger=INFO,console

.......................................

2016-09-28 10:19:57,329 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:164)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/172.28.18.237:44444]

表示启动成功。

打开另一个相同的窗口
[sparkadmin@hadoop4 ~]$ yum install telnet -y
[sparkadmin@hadoop4 ~]$ telnet hadoop4 44444
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Hello world

OK

this is a good word and my work is very intersting
OK

回到刚才安装flume的窗口上，将显示
2016-09-28 10:21:07,326 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 48 65 6C 6C 6F 20 77 6F 72 6C 64 0D             Hello world. }

2016-09-28 10:21:07,326 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 74 68 69 73 20 69 73 20 61 20 67 6F 6F 64 20 77this is a good w }

由此，使用netcat作为source的功能即演示成功了。

当然，我们发现了一个问题，明明在telnet中发送的数据是：this is a good word and my work is very intersting，接收到的数据却是this is a good w。数据不完整。

在源码 flume-ng-core 下面的 org.apache.flume.sink

在LoggerSink.java中：
if (event != null) {
        if (logger.isInfoEnabled()) {
          logger.info("Event: " + EventHelper.dumpEvent(event));
        }
}


我们去看EventHelper.java的dumpEvent方法：
private static final int DEFAULT_MAX_BYTES = 16;

需改源码这个数字，然后编译源码，编译源码可以参考我的博客《Maven编译Flume-ng 1.5.0》。

比如：我修改为

private static final int DEFAULT_MAX_BYTES = 30;

然后编译源码后验证：

[sparkadmin@hadoop4 ~]$ telnet hadoop4 44444
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

this is a good word and my work is very intersting
OK

回到刚才安装flume的窗口上，将显示

2016-09-28 20:55:18,634 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 74 68 69 73 20 69 73 20 61 20 67 6F 6F 64 20 77 this is a good w
00000010 6F 72 64 20 61 6E 64 20 6D 79 20 77 6F 72       ord and my wor }

显示出30个字符。