安装与测试

安装

1、下载tar包，可以使用wget +url或者到官方网站上下载tar包

2、解压到指定目录，使用tar命令

3、进入到配置文件中，进行简单的配置即可使用

即可进行使用

简单测试

进入到flume配置文件，配置flume.conf

a1.sources = r1

a1.sinks = k1

a1.channels = c1

#source

a1.sources.r1.type = netcat

a1.sources.r1.bind = localhost

a1.sources.r1.port  = 44444

#sink

a1.sinks.k1.type = logger

#channel

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

#bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

进入bin目录

./flume-ng agent --conf ../conf  --conf-file flume.conf --name a1 -Dflume.root.logger=INFO,console

启动之后，可以使用telnet进行数据写入

telnet localhost 44444

flume集群搭建

说明：2个agent，一个负责数据采集，然后提交给收集数据的agent，收集到数据之后打印出来

搭建：flume集群环境只需要将单机环境copy到多台电脑上即可，然后修改conf文件，使之负责不同的事情

数据采集端：

source：使用spooldir扫描文件获取资源

sink：avro sink

channel : memory

数据接收端：

source ：avro source 

sink : logger sink

channel : memory

数据采集端：push.conf

agent2.sources = tailsource-1

agent2.sinks = remotesink

agent2.channels = memoryChannel-1

agent2.sources.tailsource-1.type = spooldir

agent2.sources.tailsource-1.spoolDir = /tmp/logs

agents.sources.tailsource-1.channels = memoryChannel-1

agent2.channels.memoryChannel-1.type = memory

agent2.channels.memoryChannel-1.keep-alive = 10

agent2.channels.memoryChannel-1.capacity = 100000

agent2.channels.memoryChannel-1.transactionCapacity = 100000

agent2.sinks.remotesink.type = avro

agent2.sinks.remotesink.hostname = 192.168.1.110

agent2.sinks.remotesink.port = 44444

agent2.sinks.remotesink.channel = memoryChannel-1

数据接收端：pull.conf

agent1.sources = avrosrc 

agent1.channels = memoryChannel

agent1.sinks = loggersink

agent1.sources.avrosrc.type = avro

agent1.sources.avrosrc.bind = 192.168.1.110

agent1.sources.avrosrc.port  = 44444

agent1.sources.avrosrc.channels = memoryChannel

agent1.channels.memoryChannel.type = memory

agent1.channels.memoryChannel.keep-alive = 10

agent1.channels.memoryChannel.capacity = 100000

agent1.channels.memoryChannel.transactionCapacity = 100000

agent1.sinks.loggersink.type = logger

agent1.sinks.loggersink.channel = memoryChannel

注意：指定环境要安装avro（avro依赖cmake）、flume agent必须安装JDK1.6版本以上的支持