Flume-应用案例（二）

###Spool###
监测目录下的新增文件，并将文件中的内容读取出来。

• 编写spooldir source配置文件

>echo spooldir.conf
>vim spooledir.conf
#defined name of source/channel/sink
a1.sources=r1
a1.channels=c1
a1.sinks=k1

#configure spooldir source
a1.sources.r1.type=spooldir
a1.sources.r1.spoolDir=/usr/develop-fm/flume1.7.0/temp
a1.sources.r1.fileHeader=true

#configure logger sink
a1.sinks.k1.type=logger

#configure memory channel
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100

#bind source and sink to channel
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1

• 启动spool agent

>./flume-ng agent  -f ../conf/spooldir.conf -n a1 -Dflume.root.logger=INFO,console

• 向目录中导入新文件

>cp hql.txt /usr/develop-fm/flume1.7.0/temp/

• 查看控制台

 16/11/06 23:46:36 INFO sink.LoggerSink: Event: { headers:{file=/usr/develop-fm/flume1.7.0/temp/hql.txt} body: 70 75 74 20 27 75 73 65 72 27 2C 27 4D 61 73 61 put 'user','Masa }
16/11/06 23:46:36 INFO sink.LoggerSink: Event: { headers:{file=/usr/develop-fm/flume1.7.0/temp/hql.txt} body: 70 75 74 20 27 75 73 65 72 27 2C 27 4D 61 72 69 put 'user','Mari }
16/11/06 23:46:36 INFO sink.LoggerSink: Event: { headers:{file=/usr/develop-fm/flume1.7.0/temp/hql.txt} body: 70 75 74 20 27 75 73 65 72 27 2C 27 52 6F 62 65 put 'user','Robe }
16/11/06 23:46:36 INFO sink.LoggerSink: Event: { headers:{file=/usr/develop-fm/flume1.7.0/temp/hql.txt} body: }
16/11/06 23:46:36 INFO sink.LoggerSink: Event: { headers:{file=/usr/develop-fm/flume1.7.0/temp/hql.txt} body: 70 75 74 20 27 75 73 65 72 27 2C 27 59 76 6F 6E put 'user','Yvon }
16/11/06 23:46:36 INFO sink.LoggerSink: Event: { headers:{file=/usr/develop-fm/flume1.7.0/temp/hql.txt} body: 70 75 74 20 27 75 73 65 72 27 2C 27 4D 61 73 61 put 'user','Masa }
16/11/06 23:46:36 INFO sink.LoggerSink: Event: { headers:{file=/usr/develop-fm/flume1.7.0/temp/hql.txt} body: 70 75 74 20 27 75 73 65 72 27 2C 27 4D 61 72 69 put 'user','Mari }
16/11/06 23:46:36 INFO sink.LoggerSink: Event: { headers:{file=/usr/develop-fm/flume1.7.0/temp/hql.txt} body: 70 75 74 20 27 75 73 65 72 27 2C 27 52 6F 62 65 put 'user','Robe }
16/11/06 23:46:36 INFO sink.LoggerSink: Event: { headers:{file=/usr/develop-fm/flume1.7.0/temp/hql.txt} body: }

Exec

从一个给定命令的输出获取数据源，通常情况下是tail

• 编写exec source配置文件

#defined name of source/channel/sink
a1.sources=r1
a1.channels=c1
a1.sinks=k1

#configure exec source
a1.sources.r1.type=exec
a1.sources.r1.command=tail -F /usr/develop-fm/flume1.7.0/temp/exectest.log

#configure memory channel
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100

#configure sink
a1.sinks.k1.type=logger

#bind source and sink to channel
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1

• 启动agent

>./flume-ng agent  -f ../conf/exec.conf -n a1 -Dflume.root.logger=INFO,console

• 向文件中写入数据

>echo "aaaaa" >> exectest.log

• 查看控制台

16/11/07 00:01:20 INFO sink.LoggerSink: Event: { headers:{} body: 61 61 61 61                                     aaaa }
16/11/07 00:01:20 INFO sink.LoggerSink: Event: { headers:{} body: 61 61 61 61                                     aaaa }

SyslogTCP

• 编写SyslogTCP source配置文件

>echo >> syslog.conf
>vim syslog.conf
#defined name of source/channel/sink
a1.sources=r1
a1.channels=c1
a1.sinks=k1

#configure syslogtcp source
a1.sources.r1.type=syslogtcp
a1.sources.r1.host=hbase
a1.sources.r1.port=5140

#configure channel
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100

#sink
a1.sinks.k1.type = logger

a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1

• 启动agent

>./flume-ng agent  -f ../conf/syslog.conf -n a1 -Dflume.root.logger=INFO,console

• 向端口发送数据

>echo "hello syslogtcp" | nc hbase 5140

• 查看控制台

6/11/07 00:23:30 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 68 65 6C 6C 6F 20 73 79 73 6C 6F 67 20 74 63 70 hello syslog tcp }