flume收集日志的几种sources的收集实例

实例1 ：类型avro，在flume的conf里面创建一个avro.conf用来测试，如下：
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.channels = c1
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动flume

/usr/flume/apache-flume-1.6.0-bin/bin/flume-ng agent --conf /usr/flume/apache-flume-1.6.0-bin/conf/ --conf-file /usr/flume/apache-flume-1.6.0-bin/conf/avro.conf --name a1 -Dflume.root.logger=INFO,console

开启仍外一个窗口，穿件一个测试文件，写入一些测试信息，然后透过avro发送给flume

 echo "hellow word" > /var/log/27.log

/usr/flume/apache-flume-1.6.0-bin/bin/flume-ng avro-client -c /usr/flume/apache-flume-1.6.0-bin/conf/ -H localhost -p 44444 -F /var/log/27.log

结果如图：

实例2：Spool监测配置的目录下新增的文件，并将文件中的数据读取出来。需要注意两点：1spool目录下的文件不可以再打开编辑，2spool目录下不可包含相应的子目录

spool.conf

a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = spooldir
a1.sources.r1.channels = c1
a1.sources.r1.spoolDir = /var/log/ppp
a1.sources.r1.fileHeader = true
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动flume

/usr/flume/apache-flume-1.6.0-bin/bin/flume-ng agent --conf /usr/flume/apache-flume-1.6.0-bin/conf/ --conf-file /usr/flume/apache-flume-1.6.0-bin/conf/spool.conf --name a1 -Dflume.root.logger=INFO,console

监控的目录是 /var/log/ppp

echo "hellow word" > /var/log/ppp/3.log

结果

实例3：EXEC执行一个给定的命令获得输出的源,如果要使用tail命令，必选使得file足够大才能看到输出内容
exec.conf

a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
#a1.sources.r1.type = exec
#a1.sources.r1.channels = c1
#a1.sources.r1.command = tail -F /var/log/flume.log

a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444


# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动flume

/usr/flume/apache-flume-1.6.0-bin/bin/flume-ng agent --conf /usr/flume/apache-flume-1.6.0-bin/conf/ --conf-file /usr/flume/apache-flume-1.6.0-bin/conf/exec.conf --name a1 -Dflume.root.logger=INFO,console

往文件里面写内容：

for i in {1..100};do echo "exec tail$i" >> /var/log/flume.log;echo $i;sleep 0.1;done

结果：

实例4：Syslogtcp监听TCP的端口做为数据源
Syslogtcp.conf

a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.port = 80
a1.sources.r1.host = localhost
a1.sources.r1.channels = c1
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动flume

/usr/flume/apache-flume-1.6.0-bin/bin/flume-ng agent --conf /usr/flume/apache-flume-1.6.0-bin/conf/ --conf-file /usr/flume/apache-flume-1.6.0-bin/conf/Syslogtcp.conf --name a1 -Dflume.root.logger=INFO,console

往端口80发送内容：

echo "hello idoall.org syslog" | nc localhost 5140

结果：