紧接我的上一篇博客 Flume应用案例之监控一个文件实时采集新增的数据输出到控制台
虚拟机啥的我先不搭了,就在一台服务器上演示
技术选型:
exec source + memory channel + avro sink
avro source + memory channel + logger sink
所以需要两个配置文件:
在conf目录下创建exec-memory-avro.conf和avro-memory-logger.conf文件
exec-memory-avro.conf
exec-memory-avro.sources = exec-source
exec-memory-avro.sinks = avro.sink
exec-memory-avro.channels = memory-channel
exec-memory-avro.sources.exec-source.type = exec
exec-memory-avro.sources.exec-source.command = tail -F /home/zq/data/data.log
exec-memory-avro.sources.exec-source.shell = /bin/sh -c
exec-memory-avro.sinks.avro.sink.type = avro
exec-memory-avro.sinks.avro.sink.hostname = zq
exec-memory-avro.sinks.avro.sink.port = 44444
exec-memory-avro.channels.memory-channel.type = memory
exec-memory-avro.sources.exec-source.channels = memory-channel
exec-memory-avro.sinks.avro.sink.channel = memory-channel
avro-memory-logger.conf
avro-memory-logger.sources = avro-source
avro-memory-logger.sinks = logger.sink
avro-memory-logger.channels = memory-channel
avro-memory-logger.sources.avro-source.type = avro
avro-memory-logger.sources.avro-source.bind = zq
avro-memory-logger.sources.avro-source.port = 44444
avro-memory-logger.sinks.logger.sink.type = logger
avro-memory-logger.channels.memory-channel.type = memory
avro-memory-logger.sources.avro-source.channels = memory-channel
avro-memory-logger.sinks.logger-sink.channel = memory-channel
测试:
1.新开一个控制台,进行ssh连接
2.在当前控制台先启动avro-memory-logger.conf,再启动exec-memory-avro.conf
flume-ng agent \
--name avro-memory-logger \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/avro-memory-logger.conf \
-Dflume.root.logger=INFO,console
flume-ng agent \
--name exec-memory-avro \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/exec-memory-avro.conf \
-Dflume.root.logger=INFO,console
再开一个终端,前两个中有一个会响应消息