Flume:A机器到B机器的日志收集配置

日志收集过程：

1）机器A上监控一个文件，当我们访问主站时会有用户行为日志记录到access.log
2）avro sink把新产生的日志输出到对应的avro source指定的hostname和port上
3）通过avro source对应的agent将我们的日志输出到控制台（kafka）

技术选型：

exec source + memory channel + avro sink	
avro source + memory channel + logger sink 

name:exec-memory-avro.conf

a1.sources = r1
a1.sinks = k1
a1.channels = c1

a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /root/zoujc/data
a1.sources.r1.shell = /bin/sh -c

a1.sinks.k1.type = avro
a1.sinks.k1.hostname = zoujc01
a1.sinks.k1.port = 44444

a1.channels.c1.type = memory

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

name:avro-memory-logger.conf

a2.sources = r1
a2.sinks = k1
a2.channels = c1

a2.sources.r1.type = avro
a2.sources.r1.bind = zoujc01
a2.sources.r1.port = 44444

a2.sinks.k1.type = logger

a2.channels.c1.type = memory

a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1

启动

#先启动avro-memory-logger.conf
flume-ng agent
–name a1
–conf $FLUME_HOME/conf
–conf-file $FLUME_HOME/conf/avro-memory-logger.conf \
-Dflume.root.logger=INFO,console

#再启动exec-memory-avro.conf
flume-ng agent
–name a1
–conf $FLUME_HOME/conf
–conf-file $FLUME_HOME/conf/exec-memory-avro.conf
-Dflume.root.logger=INFO,console

echo i am zoujc >> data.log