agent选择
A机器 exec source + memory channel + avro sink
B机器 avro source + memory channel + logger sink
avro source: 监听avro端口,并且接收来自外部avro信息,
avro sink:一般用于跨节点传输,主要绑定数据移动目的地的ip和port
这里测试的时候准备两台服务器!
两台都要安装flume。
我这里用的是node1服务器和node2服务器
注意:要先启动node2,它是监听4444端口的,然后起node1才能往node2的44444端口发数据
在node2机器上创建agent配置文件
cd /app/flume/flume/conf
vi test-avro-memory-logger.conf
配置中avro-source.bind要用主机名,不能用localhost,亲试用localhost的话node1连接被拒绝
#agent avro-memory-logger
avro-memory-logger.sources = avro-source
avro-memory-logger.sinks = logger-sink
avro-memory-logger.channels = memory-channel
avro-memory-logger.sources.avro-source.type = avro
avro-memory-logger.sources.avro-source.bind = node2
avro-memory-logger.sources.avro-source.port = 44444
avro-memory-logger.sinks.logger-sink.type = logger
avro-memory-logger.channels.memory-channel.type = memory
avro-memory-logger.sources.avro-source.channels = memory-channel
avro-memory-logger.sinks.logger-sink.channel = memory-channel
启动node2的agent
cd /app/flume/flume
bin/flume-ng agent --name avro-memory-logger -c conf -f conf/test-avro-memory-logger.conf -Dflume.root.logger=INFO,console
这里一定要等node2的avro-source启动成功,已经监听了自己的44444端口,才能去启动node1,不然node1启动会被拒绝连接
在node1机器上创建agent配置文件
cd /app/flume/flume/conf
vi test-exec-memory-avro.conf
#agent exec-memory-avro
exec-memory-avro.sources = exec-source
exec-memory-avro.sinks = avro-sink
exec-memory-avro.channels = memory-channel
exec-memory-avro.sources.exec-source.type = exec
exec-memory-avro.sources.exec-source.command = tail -f /app/flume/testData/testData.log
exec-memory-avro.sources.exec-source.shell = /bin/sh -c
exec-memory-avro.sinks.avro-sink.type = avro
exec-memory-avro.sinks.avro-sink.hostname = node2
exec-memory-avro.sinks.avro-sink.port = 44444
exec-memory-avro.channels.memory-channel.type = memory
exec-memory-avro.sources.exec-source.channels = memory-channel
exec-memory-avro.sinks.avro-sink.channel = memory-channel
启动node1的agent
cd /app/flume/flume
bin/flume-ng agent --name exec-memory-avro -c conf -f conf/test-exec-memory-avro.conf -Dflume.root.logger=INFO,console
向node1的exec-source监听的文件中写数据
cd /app/flume/testData
echo hello exce again >> testData.log
查看node2中是否通过node1的avro sink接收到数据