flume笔记

Flume是什么?

Flume是⼀种分布式，可靠且可⽤的服务，⽤于有效地收集，聚合和移动⼤量⽇志数据。Flume构建在⽇志流之上⼀个简单灵活的架构。它具有可靠的可靠性机制和许多故障转移和恢复机制，具有强⼤的容错性。使⽤Flume这套架构实现对⽇志流数据的实时在线分析。Flume⽀持在⽇志系统中定制各类数据发送⽅，⽤于收集数据；同时，Flume提供对数据进⾏简单处理，并写到各种数据接受⽅（可定制）的能⼒。当前Flume有两个版本Flume 0.9X版本的统称Flume-og，Flume1.X版本的统称Flume-ng。由于Flumeng经过重⼤重构，与Flume-og有很⼤不同，使⽤时请注意区分。本次课程使⽤的是apache-flume-1.9.0-bin.tar.gz

Flume安装

安装JDK 1.8+ 配置JAVA_HOME环境变量-略

[root@CentOS ~]# tar -zxf apache-flume-1.9.0-bin.tar.gz -C /usr/
[root@CentOS ~]# cd /usr/apache-flume-1.9.0-bin/
[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng version
Flume 1.9.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: d4fcab4f501d41597bc616921329a4339f73585e
Compiled by fszabo on Mon Dec 17 20:45:25 CET 2018
From source with checksum 35db629a3bda49d23e9b3690c80737f9

Agent配置模板

声明组件信息
.sources = .sinks =
.channels =
组件配置
.sources.. =
.channels.. =
.sinks.. =

链接组件
.sources..channels = …
.sinks..channel =

测试部分：

[root@CentOS ~]# tar -zxf apache-flume-1.9.0-bin.tar.gz -C /usr/ 
[root@CentOS ~]# cd /usr/apache-flume-1.9.0-bin/ 
[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng version 
Flume 1.9.0 
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git Revision: d4fcab4f501d41597bc616921329a4339f73585e 
Compiled by fszabo on Mon Dec 17 20:45:25 CET 2018 
From source with checksum 35db629a3bda49d23e9b3690c80737f9

① example1.properties 单个Agent的配置,将该配置⽂件放置在flume安装⽬录下的conf⽬录下。

#声明基本组件 Source Channel Sink
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
#配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = netcat
a1.sources.s1.bind = CentOS
a1.sources.s1.port = 44444
配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = logger
#配置Channel通道，主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1

1、安装⼀下 yum -y install nmap-ncat ，这样⽅便后续的测试。 2、需要安装 yum -y install telnet ，⽅便做测试。

②启动a1 采集组件 [root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent
–conf conf/ --name a1 – conf-file conf/example1.properties -Dflume.root.logger=INFO,console

组件概述

Source-输⼊源

Avro Source:内部启动⼀个Avro 服务器，⽤于接收来⾃Avro Client的请求，并且将接收数据存储到Chanel中。

#声明基本组件 Source Channel Sink example2.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = avro
a1.sources.s1.bind = CentOS
a1.sources.s1.port = 44444
#配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = logger
#配置Channel通道，主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
#进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1

下面命令一定要写在一行上

[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/example2.properties -Dflume.root.logger=INFO,console
[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng avro-client --host CentOS --port 44444 --filename /root/t_emp

Exec Source:可以将指令在控制台输出采集过来。

# 声明基本组件 Source Channel Sink example3.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = exec
a1.sources.s1.command = tail -F /root/t_user
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = logger
# 配置Channel通道，主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1

下面命令必须在一行执行

root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent --conf conf/ --name a1 – conf-file conf/example3.properties -Dflume.root.logger=INFO,console

实时观察 t_user的动态变化

tail -f t_user

Spooling Directory Source：采集静态⽬录下，新增⽂本⽂件，采集完成后会修改⽂件后缀，但是不会删除采集的源⽂件，如果⽤户只想采集⼀次，可以修改该source默认⾏为。

# 声明基本组件 Source Channel Sink example4.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = spooldir
a1.sources.s1.spoolDir = /root/baizhi/spooldir
a1.sources.s1.fileHeader = true
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = logger
# 配置Channel通道，主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1

下面的命令必须写在一行上

[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent --conf conf/ --name a1 – conf-file conf/example4.properties -Dflume.root.logger=INFO,console

必须事先创建目录 否则报错

[root@Centos baizhi]# mkdir spooldir

Taildir Source：实时监测动态⽂本⾏的追加，并且记录采集的⽂件读取的位置了偏移量，即使下⼀
次再次采集，可以实现增量采集。

# 声明基本组件 Source Channel Sink example5.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = TAILDIR
a1.sources.s1.filegroups = g1 g2
a1.sources.s1.filegroups.g1 = /root/baizhi/.*\.sql$
a1.sources.s1.filegroups.g2 = /root/baizhi/.*\.txt$
a1.sources.s1.headers.g1.type = log
a1.sources.s1.headers.g2.type = java
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = logger
# 配置Channel通道，主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1

执行前确保/root/baizhi下面存在..sql$ 和 ..txt$
下面代码必须在一行执行

[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent --conf conf/ --name a1 – conf-file conf/example5.properties
-Dflume.root.logger=INFO,console

Kafka Source

数据从kafka中的生产者输入的 kafka source是kafka的消费者 消费到channel

# 声明基本组件 Source Channel Sink example9.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.s1.batchSize = 100
a1.sources.s1.batchDurationMillis = 2000
Sink-输出
Logger Sink ：通常⽤于测试/调试⽬的。
File Roll Sink: 可以将采集的数据写⼊到本地⽂件
HDFS Sink: 可以将数据写⼊到HDFS⽂件系统。
a1.sources.s1.kafka.bootstrap.servers = Centos:9092
a1.sources.s1.kafka.topics = topic01
a1.sources.s1.kafka.consumer.group.id = g1
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = logger
# 配置Channel通道，主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1

注意事项
启动zookeeper

./bin/zkServer.sh start zoo.cfg

启动kafka

./bin/kafka-server-start.sh -daemon config/server.properties 

生产

./bin/kafka-console-producer.sh 
                  --broker-list CentOSA:9092,CentOSB:9092,CentOSC:9092 
                  --topic topic01

a1.sources.s1.batchSize = 100 (批量写入通道的最大消息数)小于等于a1.channels.c1.transactionCapacity = 100
小于等于a1.channels.c1.capacity = 1000

Sink-输出

Logger Sink ：通常⽤于测试/调试⽬的。
File Roll Sink: 可以将采集的数据写⼊到本地⽂件

创建文件example6.properties

# 声明基本组件 Source Channel Sink example6.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = netcat
a1.sources.s1.bind = Centos
a1.sources.s1.port = 44444
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = file_roll
a1.sinks.sk1.sink.directory = /root/baizhi/file_roll
a1.sinks.sk1.sink.rollInterval = 0
# 配置Channel通道，主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1

启动flume执行命令必须在一行

> [root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent --conf conf/ --name a1 --conf-file conf/example6.properties

HDFS Sink

可以将数据写⼊到HDFS⽂件系统

# 声明基本组件 Source Channel Sink example7.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = netcat
a1.sources.s1.bind = Centos
a1.sources.s1.port = 44444
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = hdfs
a1.sinks.sk1.hdfs.path = /flume-hdfs/%y-%m-%d
a1.sinks.sk1.hdfs.rollInterval = 0
a1.sinks.sk1.hdfs.rollSize = 0
a1.sinks.sk1.hdfs.rollCount = 0
a1.sinks.sk1.hdfs.useLocalTimeStamp = true
a1.sinks.sk1.hdfs.fileType = DataStream
Kafka Sink
将数据写⼊Kafka的Topic中
Avro Sink: 将数据写出给Avro Source
# 配置Channel通道，主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1

Kafka Sink

将数据写⼊Kafka的Topic中

# 声明基本组件 Source Channel Sink example8.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = netcat
a1.sources.s1.bind = Centos
a1.sources.s1.port = 44444
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.sk1.kafka.bootstrap.servers = Centos:9092
a1.sinks.sk1.kafka.topic = topic01
a1.sinks.sk1.kafka.flumeBatchSize = 20
a1.sinks.sk1.kafka.producer.acks = 1
a1.sinks.sk1.kafka.producer.linger.ms = 1
a1.sinks.sk1.kafka.producer.compression.type = snappy
# 配置Channel通道，主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1

Avro Sink: 将数据写出给Avro Source

# 声明基本组件 Source Channel Sink example10.properties
a1.sources = s1
a1.sinks = sk1
a1.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a1.sources.s1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.s1.batchSize = 100
a1.sources.s1.batchDurationMillis = 2000
a1.sources.s1.kafka.bootstrap.servers = Centos:9092
a1.sources.s1.kafka.topics = topic01
a1.sources.s1.kafka.consumer.group.id = g1
# 配置Sink组件,将接收数据打印在⽇志控制台
a1.sinks.sk1.type = avro
a1.sinks.sk1.hostname = Centos
a1.sinks.sk1.port = 44444
# 配置Channel通道，主要负责数据缓冲
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1
# 声明基本组件 Source Channel Sink example10.properties
a2.sources = s1
a2.sinks = sk1
a2.channels = c1
# 配置Source组件,从Socket中接收⽂本数据
a2.sources.s1.type = avro
a2.sources.s1.bind = Centos
a2.sources.s1.port = 44444
# 配置Sink组件,将接收数据打印在⽇志控制台
a2.sinks.sk1.type = file_roll
a2.sinks.sk1.sink.directory = /root/baizhi/file_roll
a2.sinks.sk1.sink.rollInterval = 0
# 配置Channel通道，主要负责数据缓冲
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100
# 进⾏组件间的绑定
a2.sources.s1.channels = c1
a2.sinks.sk1.channel = c1

先启动a2部分

[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent --conf conf/ --conf-file conf/emple10.properties --name a2 -Dflume.root.logger=INFO,console

再启动a1部分

[root@CentOS apache-flume-1.9.0-bin]# ./bin/flume-ng agent --conf conf/ --conf-file conf/emple10.properties --name a1 -Dflume.root.logger=INFO,console

kafka的生产者

[root@CentOS kafka_2.11-2.2.0]# ./bin/kafka-console-producer.sh --broker-list Centos:9092 --topic topic01