flume到flume消息传递

环境：两台虚拟机（ 每台都有flume）

第一台slave作为消息的产生者

第二台master作为消息的接收者    IP（192.168.83.133）

原理：通过监听slave中文件的变化，获取变化信息，发送到另一台电脑。

 

1、slave上配置conf里面的example.conf(标红的注意下)

#Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /home/qq/pp/data/test.log
a1.sources.r1.channels = c1

# Describe the sink
##sink端的avro是一个数据发送者
a1.sinks = k1
##type设置成avro来设置发消息
a1.sinks.k1.type = avro
a1.sinks.k1.channel = c1
##下沉到hadoop2这台机器
a1.sinks.k1.hostname = 192.168.83.133
##下沉到mini2中的44444
a1.sinks.k1.port = 44444
a1.sinks.k1.batch-size = 2

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

2、master上配置conf里面的example.conf(标红的注意下)

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
##source中的avro组件是一个接收者服务
a1.sources.r1.type = avro
a1.sources.r1.channels = c1
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

3、向监听文件写入字符串(程序循环写入，不用手动修改test.log文件了)

[root@s1 # cd /home/qq/pp/data
[root@s1 home/qq/pp/data# while true
> do
> echo "fdssdgfgf" >> test.log

> sleep 1
> done

 

4、查看上面的程序是否执行

#cd /home/qq/pp/data

#tail -f test.log

显示如下即可：

 

5、打开消息接收者master的flume

进入flume安装目录，执行如下语句

bin/flume-ng agent -c conf -f conf/example.conf -n a1 -Dflume.root.logger=INFO,console

现在回打印出一些信息

6、启动slave的flume

进入flume安装目录，执行如下语句

bin/flume-ng agent -c conf -f conf/example.conf -n a1 -Dflume.root.logger=INFO,console

7、查看5）中的窗口会显示插入的字符串

 

 

 参考：https://blog.csdn.net/tototuzuoquan/article/details/73203241

 

转载于:https://www.cnblogs.com/51python/p/10961371.html