07案例-单数据源多出口-Sink组

本文介绍如何使用Flume进行多级数据传输,通过配置不同的Agent,将数据从源文件传输并同时存储到HDFS中。具体步骤包括创建配置文件、设置source、channel和sink,以及启动Flume Agent。

案例需求:
使用Flume-1监控文件变动,Flume-1将变动内容传递给Flume-2,
Flume-2负责存储到HDFS。同时Flume-1将变动内容传递给Flume-3,
Flume-3也负责存储到HDFS。

实现步骤:

1.准备工作
	在/root/appp/flume/job目录下创建group2文件夹
		mkdir group2
		cd group2/
2.创建flume-netcat-flume.conf
	(配置1个接收日志文件的source和1个channel、两个sink,
		分别输送给flume-flume-console1和flume-flume-console2。)
	touch flume-netcat-flume.conf
	vim flume-netcat-flume.conf
	
		# Name the components on this agent
		a1.sources = r1
		a1.channels = c1
		a1.sinkgroups = g1
		a1.sinks = k1 k2

		# Describe/configure the source
		a1.sources.r1.type = netcat
		a1.sources.r1.bind = localhost
		a1.sources.r1.port = 44444

		a1.sinkgroups.g1.processor.type = load_balance
		a1.sinkgroups.g1.processor.backoff = true
		a1.sinkgroups.g1.processor.selector = round_robin
		a1.sinkgroups.g1.processor.selector.maxTimeOut=10000

		# Describe the sink
		a1.sinks.k1.type = avro
		a1.sinks.k1.hostname = hadoop102
		a1.sinks.k1.port = 4141

		a1.sinks.k2.type = avro
		a1.sinks.k2.hostname = hadoop102
		a1.sinks.k2.port = 4142

		# Describe the channel
		a1.channels.c1.type = memory
		a1.channels.c1.capacity = 1000
		a1.channels.c1.transactionCapacity = 100

		# Bind the source and sink to the channel
		a1.sources.r1.channels = c1
		a1.sinkgroups.g1.sinks = k1 k2
		a1.sinks.k1.channel = c1
		a1.sinks.k2.channel = c1
3.创建flume-flume-console1.conf
	(配置上级Flume输出的Source,输出是到本地控制台)
	touch flume-flume-console1.conf
	vim flume-flume-console1.conf
		# Name the components on this agent
		a2.sources = r1
		a2.sinks = k1
		a2.channels = c1

		# Describe/configure the source
		a2.sources.r1.type = avro
		a2.sources.r1.bind = hadoop102
		a2.sources.r1.port = 4141

		# Describe the sink
		a2.sinks.k1.type = logger

		# Describe the channel
		a2.channels.c1.type = memory
		a2.channels.c1.capacity = 1000
		a2.channels.c1.transactionCapacity = 100

		# Bind the source and sink to the channel
		a2.sources.r1.channels = c1
		a2.sinks.k1.channel = c1
4.创建flume-flume-console2.conf
	(配置上级Flume输出的Source,输出是到本地控制台)
	touch flume-flume-console2.conf
	vim flume-flume-console2.conf
		# Name the components on this agent
		a3.sources = r1
		a3.sinks = k1
		a3.channels = c2

		# Describe/configure the source
		a3.sources.r1.type = avro
		a3.sources.r1.bind = hadoop102
		a3.sources.r1.port = 4142

		# Describe the sink
		a3.sinks.k1.type = logger

		# Describe the channel
		a3.channels.c2.type = memory
		a3.channels.c2.capacity = 1000
		a3.channels.c2.transactionCapacity = 100

		# Bind the source and sink to the channel
		a3.sources.r1.channels = c2
		a3.sinks.k1.channel = c2
5.执行配置文件
	(分别开启对应配置文件:flume-flume-console2,flume-flume-console1,flume-netcat-flume)
	bin/flume-ng agent --conf conf/ --name a3 
		--conf-file job/group2/flume-flume-console2.conf -Dflume.root.logger=INFO,console
		
	bin/flume-ng agent --conf conf/ --name a2 
		--conf-file job/group2/flume-flume-console1.conf -Dflume.root.logger=INFO,console
		
	bin/flume-ng agent --conf conf/ --name a1 
		--conf-file job/group2/flume-netcat-flume.conf
6.使用telnet工具向本机的44444端口发送内容
	telnet localhost 44444
7.查看Flume2及Flume3的控制台打印日志
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

hao难懂

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值