Flume是什么?
Flume是⼀种分布式,可靠且可⽤的服务,⽤于有效地收集,聚合和移动⼤量⽇志数据。Flume构建在⽇志流之上⼀个简单灵活的架构。它具有可靠的可靠性机制和许多故障转移和恢复机制,具有强⼤的容错性。使⽤Flume这套架构实现对⽇志流数据的实时在线分析。Flume⽀持在⽇志系统中定制各类数据发送⽅,⽤于收集数据;同时,Flume提供对数据进⾏简单处理,并写到各种数据接受⽅(可定制)的能⼒。当前Flume有两个版本Flume 0.9X版本的统称Flume-og,Flume1.X版本的统称Flume-ng。由于Flumeng经过重⼤重构,与Flume-og有很⼤不同,使⽤时请注意区分。本次课程使⽤的是apache-flume-1.9.0-bin.tar.gz
Flume安装
安装JDK 1.8+ 配置JAVA_HOME环境变量-略
[root@CentOS ~]# tar -zxf apache-flume-1.9.0-bin.tar.gz -C /usr/
[root@CentOS ~]# cd /usr/apache-flume-1.9.0-bin/
[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng version
Flume 1.9.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: d4fcab4f501d41597bc616921329a4339f73585e
Compiled by fszabo on Mon Dec 17 20:45:25 CET 2018
From source with checksum 35db629a3bda49d23e9b3690c80737f9
Agent配置模板
声明组件信息
.sources = .sinks =
.channels =
组件配置
.sources.. =
.channels.. =
.sinks.. =链接组件
.sources..channels = …
.sinks..channel =
测试部分:
[root@CentOS ~]# tar -zxf apache-flume-1.9.0-bin.tar.gz -C /usr/
[root@CentOS ~]# cd /usr/apache-flume-1.9.0-bin/
[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng version
Flume 1.9.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git Revision: d4fcab4f501d41597bc616921329a4339f73585e
Compiled by fszabo on Mon Dec 17 20:45:25 CET 2018
From source with checksum 35db629a3bda49d23e9b3690c80737f9
① example1.properties 单个Agent的配置,将该配置⽂件放置在flume安装⽬录下的conf⽬录下。
#声明基本组件 Source Channel Sink
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
#配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = netcat
a1.sources.s1.bind = CentOS
a1.sources.s1.port = 44444
配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = logger
#配置Channel通道,主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1
1、安装⼀下 yum -y install nmap-ncat ,这样⽅便后续的测试。 2、需要安装 yum -y install telnet ,⽅便做测试。
②启动a1 采集组件 [root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent
–conf conf/ --name a1 – conf-file conf/example1.properties -Dflume.root.logger=INFO,console
组件概述
Source-输⼊源
Avro Source:内部启动⼀个Avro 服务器,⽤于接收来⾃Avro Client的请求,并且将接收数据存储到Chanel中。
#声明基本组件 Source Channel Sink example2.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = avro
a1.sources.s1.bind = CentOS
a1.sources.s1.port = 44444
#配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = logger
#配置Channel通道,主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
#进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1
下面命令一定要写在一行上
[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/example2.properties -Dflume.root.logger=INFO,console
[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng avro-client --host CentOS --port 44444 --filename /root/t_emp
Exec Source:可以将指令在控制台输出采集过来。
# 声明基本组件 Source Channel Sink example3.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = exec
a1.sources.s1.command = tail -F /root/t_user
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = logger
# 配置Channel通道,主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1
下面命令必须在一行执行
root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent --conf conf/ --name a1 – conf-file conf/example3.properties -Dflume.root.logger=INFO,console
实时观察 t_user的动态变化
tail -f t_user
Spooling Directory Source:采集静态⽬录下,新增⽂本⽂件,采集完成后会修改⽂件后缀,但是不会删除采集的源⽂件,如果⽤户只想采集⼀次,可以修改该source默认⾏为。
# 声明基本组件 Source Channel Sink example4.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = spooldir
a1.sources.s1.spoolDir = /root/baizhi/spooldir
a1.sources.s1.fileHeader = true
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = logger
# 配置Channel通道,主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1
下面的命令必须写在一行上
[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent --conf conf/ --name a1 – conf-file conf/example4.properties -Dflume.root.logger=INFO,console
必须事先创建目录 否则报错
[root@Centos baizhi]# mkdir spooldir
Taildir Source:实时监测动态⽂本⾏的追加,并且记录采集的⽂件读取的位置了偏移量,即使下⼀
次再次采集,可以实现增量采集。
# 声明基本组件 Source Channel Sink example5.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = TAILDIR
a1.sources.s1.filegroups = g1 g2
a1.sources.s1.filegroups.g1 = /root/baizhi/.*\.sql$
a1.sources.s1.filegroups.g2 = /root/baizhi/.*\.txt$
a1.sources.s1.headers.g1.type = log
a1.sources.s1.headers.g2.type = java
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = logger
# 配置Channel通道,主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1
执行前确保/root/baizhi下面存在..sql$ 和 ..txt$
下面代码必须在一行执行
[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent --conf conf/ --name a1 – conf-file conf/example5.properties
-Dflume.root.logger=INFO,console
Kafka Source
数据从kafka中的生产者输入的 kafka source是kafka的消费者 消费到channel
# 声明基本组件 Source Channel Sink example9.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.s1.batchSize = 100
a1.sources.s1.batchDurationMillis = 2000
Sink-输出
Logger Sink :通常⽤于测试/调试⽬的。
File Roll Sink: 可以将采集的数据写⼊到本地⽂件
HDFS Sink: 可以将数据写⼊到HDFS⽂件系统。
a1.sources.s1.kafka.bootstrap.servers = Centos:9092
a1.sources.s1.kafka.topics = topic01
a1.sources.s1.kafka.consumer.group.id = g1
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = logger
# 配置Channel通道,主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1
注意事项
启动zookeeper
./bin/zkServer.sh start zoo.cfg
启动kafka
./bin/kafka-server-start.sh -daemon config/server.properties
生产
./bin/kafka-console-producer.sh
--broker-list CentOSA:9092,CentOSB:9092,CentOSC:9092
--topic topic01
a1.sources.s1.batchSize = 100 (批量写入通道的最大消息数)小于等于a1.channels.c1.transactionCapacity = 100
小于等于a1.channels.c1.capacity = 1000
Sink-输出
Logger Sink :通常⽤于测试/调试⽬的。
File Roll Sink: 可以将采集的数据写⼊到本地⽂件
创建文件example6.properties
# 声明基本组件 Source Channel Sink example6.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = netcat
a1.sources.s1.bind = Centos
a1.sources.s1.port = 44444
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = file_roll
a1.sinks.sk1.sink.directory = /root/baizhi/file_roll
a1.sinks.sk1.sink.rollInterval = 0
# 配置Channel通道,主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1
启动flume执行命令必须在一行
> [root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/example6.properties
HDFS Sink
可以将数据写⼊到HDFS⽂件系统
# 声明基本组件 Source Channel Sink example7.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = netcat
a1.sources.s1.bind = Centos
a1.sources.s1.port = 44444
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = hdfs
a1.sinks.sk1.hdfs.path = /flume-hdfs/%y-%m-%d
a1.sinks.sk1.hdfs.rollInterval = 0
a1.sinks.sk1.hdfs.rollSize = 0
a1.sinks.sk1.hdfs.rollCount = 0
a1.sinks.sk1.hdfs.useLocalTimeStamp = true
a1.sinks.sk1.hdfs.fileType = DataStream
Kafka Sink
将数据写⼊Kafka的Topic中
Avro Sink: 将数据写出给Avro Source
# 配置Channel通道,主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1
Kafka Sink
将数据写⼊Kafka的Topic中
# 声明基本组件 Source Channel Sink example8.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = netcat
a1.sources.s1.bind = Centos
a1.sources.s1.port = 44444
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.sk1.kafka.bootstrap.servers = Centos:9092
a1.sinks.sk1.kafka.topic = topic01
a1.sinks.sk1.kafka.flumeBatchSize = 20
a1.sinks.sk1.kafka.producer.acks = 1
a1.sinks.sk1.kafka.producer.linger.ms = 1
a1.sinks.sk1.kafka.producer.compression.type = snappy
# 配置Channel通道,主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1
Avro Sink: 将数据写出给Avro Source
# 声明基本组件 Source Channel Sink example10.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.s1.batchSize = 100
a1.sources.s1.batchDurationMillis = 2000
a1.sources.s1.kafka.bootstrap.servers = Centos:9092
a1.sources.s1.kafka.topics = topic01
a1.sources.s1.kafka.consumer.group.id = g1
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = avro
a1.sinks.sk1.hostname = Centos
a1.sinks.sk1.port = 44444
# 配置Channel通道,主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1
# 声明基本组件 Source Channel Sink example10.properties
a2.sources = s1
a2.sinks = sk1
a2.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a2.sources.s1.type = avro
a2.sources.s1.bind = Centos
a2.sources.s1.port = 44444
# 配置Sink组件,将接收数据打印在⽇志控制台
a2.sinks.sk1.type = file_roll
a2.sinks.sk1.sink.directory = /root/baizhi/file_roll
a2.sinks.sk1.sink.rollInterval = 0
# 配置Channel通道,主要负责数据缓冲
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a2.sources.s1.channels = c1
a2.sinks.sk1.channel = c1
先启动a2部分
[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent --conf conf/ --conf-file conf/emple10.properties --name a2 -Dflume.root.logger=INFO,console
再启动a1部分
[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent --conf conf/ --conf-file conf/emple10.properties --name a1 -Dflume.root.logger=INFO,console
kafka的生产者
[root@CentOS kafka_2.11-2.2.0]# ./bin/kafka-console-producer.sh --broker-list Centos:9092 --topic topic01