Flume NG 例子

flmue NG 应用案例：
==========================1 hello world例子(netcat)=================
<1>编写配置文件, 如 conf/flume-conf.properties文件：
# The configuration file needs to define the sources, the channels and the sinks.
# Sources, channels and sinks are defined per agent, in this case called 'a1'
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# For each one of the sources, the type is defined
a1.sources.r1.type = netcat
a1.sources.r1.bind = 192.168.0.180
a1.sources.r1.port = 44444


#The channel can be defined as follows.
a1.sources.r1.channels = c1
# Each sink's type must be defined
a1.sinks.k1.type = logger


#Specify the channel the sink should use
a1.sinks.k1.channel = c1


# Each channel's type is defined.
a1.channels.c1.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


<2>启动flume agent
/bin/flume-ng agent -c conf -f flume-conf.properties -name a1 -Dflume.root.logger=INFO,console


<3>telnet去输入点数据
huangpeng@huangpeng-ThinkPad-X260:~/Downloads$ telnet 192.168.0.180 44444
Trying 192.168.0.180...
Connected to 192.168.0.180.
Escape character is '^]'.
a
OK
hello
OK


此时有结果在打印台输出:
....
....
17/03/20 02:45:13 INFO node.AbstractConfigurationProvider: Created channel c1
17/03/20 02:45:13 INFO source.DefaultSourceFactory: Creating instance of source r1, type netcat
17/03/20 02:45:13 INFO sink.DefaultSinkFactory: Creating instance of sink: k1, type: logger
17/03/20 02:45:13 INFO node.AbstractConfigurationProvider: Channel c1 connected to [r1, k1]
17/03/20 02:45:13 INFO node.Application: Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:r1,state:IDLE} }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@106f26bb counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
17/03/20 02:45:13 INFO node.Application: Starting Channel c1
17/03/20 02:45:14 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
17/03/20 02:45:14 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: c1 started
17/03/20 02:45:14 INFO node.Application: Starting Sink k1
17/03/20 02:45:14 INFO node.Application: Starting Source r1
17/03/20 02:45:14 INFO source.NetcatSource: Source starting
17/03/20 02:45:14 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/192.168.0.180:44444]
17/03/20 02:45:36 INFO sink.LoggerSink: Event: { headers:{} body: 61 0D                                           a. }
17/03/20 02:45:40 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 0D                               hello. }


17/03/20 02:48:03 INFO sink.LoggerSink: Event: { headers:{} body: 61 61 61 61 0D                                  aaaa. }






==========================2 单节点写入hdfs(Exec模式)=================
<1>编写脚本不断生成数据重定向到本地文件
#!/bin/bash


while true
do
        echo '11111111111' >> /home/hadoop/software/apache-flume-1.7.0-bin/tmp/data.log
        sleep 1
done


<2>编写配置文件
hadoop@master:~/software/apache-flume-1.7.0-bin/conf$ cat flume-conf2.properties 
a1.sources = r1
a1.sinks = k1
a1.channels = c1




a1.sources.r1.type = exec
a1.sources.r1.channels = c1
a1.sources.r1.command = tail -F /home/hadoop/software/apache-flume-1.7.0-bin/tmp/data.log




a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path = hdfs://master:8020/flume/%Y%m%d/%H%M/
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 2
a1.sinks.k1.hdfs.roundUnit = minute
a1.sinks.k1.hdfs.rollSize = 4000000
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.batchSize = 10
a1.sinks.k1.hdfs.rollInterval = 3
a1.sinks.k1.hdfs.rollSize = 1024
a1.sinks.k1.hdfs.useLocalTimeStamp = true


a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


<3>启动
hadoop@master:~/software/apache-flume-1.7.0-bin/conf$ ../bin/flume-ng agent -c conf -f flume-conf2.properties -n a1 -Dflume.root.logger=INFO,console




==========================3 双节点并联写入hdfs(Exec模式)=================
和2一样，只是有个地方要注意
(1) -c conf 是指定flume的配置文件所在地，有时候需要写绝对路径才能识别
(2)在第二台节点安装flume跑测试时，发现错误异常，提示我某些java包路径找不到，查资料发现http://linuxsogood.org/1491.html


总结一下,安装完flume之后,需要从hadoop中拷贝过来的jar包有以下
commons-configuration-1.6.jar
hadoop-auth-2.5.2.jar
hadoop-common-2.5.2.jar
hadoop-hdfs-2.5.2.jar
hadoop-mapreduce-client-core-2.5.2.jar


<1>节点一配置文件和启动命令
../bin/flume-ng agent -c conf -f flume-conf2.properties -n a1 -Dflume.root.logger=INFO,console


hadoop@master:~/software/apache-flume-1.7.0-bin/conf$ cat flume-conf2.properties 
a1.sources = r1
a1.sinks = k1
a1.channels = c1




a1.sources.r1.type = exec
a1.sources.r1.channels = c1
a1.sources.r1.command = tail -F /home/hadoop/software/apache-flume-1.7.0-bin/tmp/data.log




a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path = hdfs://master:8020/flume/%Y%m%d/%H%M/
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 2
a1.sinks.k1.hdfs.roundUnit = minute
a1.sinks.k1.hdfs.rollSize = 4000000
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.batchSize = 10
a1.sinks.k1.hdfs.rollInterval = 3
a1.sinks.k1.hdfs.rollSize = 1024
a1.sinks.k1.hdfs.useLocalTimeStamp = true


a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


<2>节点二配置文件和启动命令
../bin/flume-ng agent -c ~/software/apache-flume-1.7.0-bin/conf -f flume-conf2.properties -name a1 -Dflume.root.logger=INFO,console


hadoop@slaver01:~/software/apache-flume-1.7.0-bin/conf$ cat flume-conf2.properties 
a1.sources = r1
a1.sinks = k1
a1.channels = c1




a1.sources.r1.type = exec
a1.sources.r1.channels = c1
a1.sources.r1.command = tail -F /home/hadoop/software/apache-flume-1.7.0-bin/tmp/data.log




a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path = hdfs://192.168.0.180:8020/flume/%Y%m%d/%H%M/
a1.sinks.k1.hdfs.filePrefix = events2-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 2
a1.sinks.k1.hdfs.roundUnit = minute
a1.sinks.k1.hdfs.rollSize = 4000000
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.batchSize = 10
a1.sinks.k1.hdfs.rollInterval = 3
a1.sinks.k1.hdfs.rollSize = 1024
a1.sinks.k1.hdfs.useLocalTimeStamp = true


a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


<3>hdfs结果展示
hadoop@master:~/software/hadoop-2.5.2/bin$ ./hadoop fs -ls /flume/20170321/0812
Found 11 items
-rw-r--r--   3 hadoop supergroup         72 2017-03-21 08:12 /flume/20170321/0812/events-.1490055121929
-rw-r--r--   3 hadoop supergroup        120 2017-03-21 08:12 /flume/20170321/0812/events-.1490055121930
-rw-r--r--   3 hadoop supergroup         96 2017-03-21 08:12 /flume/20170321/0812/events-.1490055121931
-rw-r--r--   3 hadoop supergroup         24 2017-03-21 08:12 /flume/20170321/0812/events-.1490055133937
-rw-r--r--   3 hadoop supergroup        120 2017-03-21 08:12 /flume/20170321/0812/events-.1490055133938
-rw-r--r--   3 hadoop supergroup         54 2017-03-21 08:11 /flume/20170321/0812/events2-.1490055121197
-rw-r--r--   3 hadoop supergroup         36 2017-03-21 08:11 /flume/20170321/0812/events2-.1490055127200
-rw-r--r--   3 hadoop supergroup         45 2017-03-21 08:12 /flume/20170321/0812/events2-.1490055127201
-rw-r--r--   3 hadoop supergroup         45 2017-03-21 08:12 /flume/20170321/0812/events2-.1490055136204
-rw-r--r--   3 hadoop supergroup          9 2017-03-21 08:12 /flume/20170321/0812/events2-.1490055136205

-rw-r--r--   3 hadoop supergroup         54 2017-03-21 08:12 /flume/20170321/0812/events2-.1490055142210

==========================4 双节点串联写入hdfs(Exec模式)=================
机器A通过avro sink写入机器B avro source， 机器B写入hdfs


<1>机器A配置以及启动命令
 ../bin/flume-ng agent -c ~/software/apache-flume-1.7.0-bin/conf -f source2avro.properties -n a1 -Dflume.root.logger=INFO,console
hadoop@master:~/software/apache-flume-1.7.0-bin/conf$ cat source2avro.properties 
a1.sources = r1
a1.sinks = k1
a1.channels = c1




a1.sources.r1.type = exec
a1.sources.r1.channels = c1
a1.sources.r1.command = tail -F /home/hadoop/software/apache-flume-1.7.0-bin/tmp/data.log


a1.sinks.k1.type = avro
a1.sinks.k1.hostname = 192.168.0.181
a1.sinks.k1.port = 44445
a1.sinks.k1.channel = c1


a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


<2>机器B配置以及启动命令
../bin/flume-ng agent -c ~/software/apache-flume-1.7.0-bin/conf -f avro2hdfs.properties -n a1 -Dflume.root.logger=INFO,console
hadoop@slaver01:~/software/apache-flume-1.7.0-bin/conf$ cat avro2hdfs.properties 
a1.sources = r1
a1.sinks = k1
a1.channels = c1




a1.sources.r1.type = avro
a1.sources.r1.bind = 192.168.0.181
a1.sources.r1.port = 44445
a1.sources.r1.channels = c1


a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path = hdfs://192.168.0.180:8020/flume2/%Y%m%d/%H%M/
a1.sinks.k1.hdfs.filePrefix = avro-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 2
a1.sinks.k1.hdfs.roundUnit = minute
a1.sinks.k1.hdfs.rollSize = 4000000
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.batchSize = 10
a1.sinks.k1.hdfs.rollInterval = 3
a1.sinks.k1.hdfs.rollSize = 1024
a1.sinks.k1.hdfs.useLocalTimeStamp = true


a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


<3>hdfs结果
hadoop@slaver02:~/software/hadoop-2.5.2/bin$ ./hadoop fs -ls /flume2/20170322/0134
Found 4 items
-rw-r--r--   3 hadoop supergroup        120 2017-03-22 01:34 /flume2/20170322/0134/avro-.1490117668758
-rw-r--r--   3 hadoop supergroup         36 2017-03-22 01:34 /flume2/20170322/0134/avro-.1490117668759
-rw-r--r--   3 hadoop supergroup        120 2017-03-22 01:34 /flume2/20170322/0134/avro-.1490117690603
-rw-r--r--   3 hadoop supergroup         36 2017-03-22 01:34 /flume2/20170322/0134/avro-.1490117690604