Flume 环境部署与配置详解

最新推荐文章于 2024-05-19 17:32:56 发布

weixin_34255793

最新推荐文章于 2024-05-19 17:32:56 发布

阅读量151

点赞数

文章标签：大数据开发工具 python

原文链接：https://my.oschina.net/u/1757002/blog/868519

版权

2019独角兽企业重金招聘Python工程师标准>>>

flume是一个分布式、可靠、和高可用的海量日志采集、聚合和传输的系统。支持在日志系统中定制各类数据发送方，用于收集数
据;同时，Flume提供对数据进行简单处理，并写到各种数据接受方(比如文本、HDFS、Hbase等)的能力 。

什么是Flume？

flume 作为 cloudera 开发的实时日志收集系统，受到了业界的认可与广泛应用。Flume 初始的发行版本目前被统称
为 Flume OG（original generation），属于 cloudera。但随着 FLume 功能的扩展，Flume OG 代码工程臃肿、核心组件
设计不合理、核心配置不标准等缺点暴露出来，尤其是在 Flume OG 的最后一个发行版本 0.94.0 中，日志传输不稳定的现象
尤为严重，为了解决这些问题，2011 年 10 月 22 号，cloudera 完成了 Flume-728，对 Flume 进行了里程碑式的
改动：重构核心组件、核心配置以及代码架构，重构后的版本统称为 Flume NG（next generation）；改动的另一原因是
将 Flume 纳入 apache 旗下，cloudera Flume 改名为 Apache Flume。

flume的特点：

　　flume是一个分布式、可靠、和高可用的海量日志采集、聚合和传输的系统。支持在日志系统中定制各类数据发送方，用于
收集数据;同时，Flume提供对数据进行简单处理，并写到各种数据接受方(比如文本、HDFS、Hbase等)的能力 。
　　flume的数据流由事件(Event)贯穿始终。事件是Flume的基本数据单位，它携带日志数据(字节数组形式)并且携带有头信息，
这些Event由Agent外部的Source生成，当Source捕获事件后会进行特定的格式化，然后Source会把事件推入(单个或多个)
Channel中。你可以把Channel看作是一个缓冲区，它将保存事件直到Sink处理完该事件。Sink负责持久化日志或者把事件推向
另一个Source。

flume的可靠性

　　当节点出现故障时，日志能够被传送到其他节点上而不会丢失。Flume提供了三种级别的可靠性保障，从强到弱依次分别
为：end-to-end（收到数据agent首先将event写到磁盘上，当数据传送成功后，再删除；如果数据发送失败，可以重新发送。），
Store on failure（这也是scribe采用的策略，当数据接收方crash时，将数据写到本地，待恢复后，继续发送），
Besteffort（数据发送到接收方后，不会进行确认）。

flume的可恢复性

　　还是靠Channel。推荐使用FileChannel，事件持久化在本地文件系统里(性能较差)。

flume的一些核心概念

Agent使用JVM 运行Flume。每台机器运行一个agent，但是可以在一个agent中包含多个sources和sinks。
     Client生产数据，运行在一个独立的线程。
     Source从Client收集数据，传递给Channel。
     Sink从Channel收集数据，运行在一个独立线程。
     Channel连接 sources 和 sinks ，这个有点像一个队列。
     Events可以是日志记录、 avro 对象等。

Flume以agent为最小的独立运行单位。一个agent就是一个JVM。单agent由Source、Sink和Channel三大组件构成，如下图：

值得注意的是，Flume提供了大量内置的Source、Channel和Sink类型。不同类型的Source,Channel和Sink可以自由组合。
组合方式基于用户设置的配置文件，非常灵活。比如：Channel可以把事件暂存在内存里，也可以持久化到本地硬盘上。
Sink可以把日志写入HDFS, HBase，甚至是另外一个Source等等。Flume支持用户建立多级流，也就是说，多个agent可以协同
工作，并且支持Fan-in、Fan-out、Contextual Routing、Backup Routes，这也正是NB之处。如下图所示:

flume的官方网站在哪里？

官方网站：http://flume.apache.org/

如何安装

wget http://mirrors.hust.edu.cn/apache/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz
   tar -zxvf apache-flume-1.6.0-bin.tar.gz
   mkdir /usr/local/flume
   mv apache-flume-1.6.0 /usr/local/flume 
   cd /usr/local/flume 
   cp conf/flume-conf.properties.template conf/flume-conf.properties #配置文件
   mkdir -p /tmp/log/flume  建立输出目录

编辑配置 conf/flume-conf.properties

vim conf/flume-conf.properties

agent.sources = r1
   agent.channels = c1
   agent.sinks = s1

   agent.sources.r1.type = netcat
   agent.sources.r1.bind = localhost
   agent.sources.r1.port = 8888
   agent.sources.r1.channels = c1

   agent.sinks.s1.type = file_roll
   agent.sinks.s1.sink.directory = /tmp/log/flume
   agent.sinks.s1.channel = c1

   agent.channels.c1.type = memory
   agent.channels.c1.capacity = 100

启动 flume服务

bin/flume-ng agent --conf conf -f conf/flume-conf.properties -n agent& #启动服务
注：运行日志位于logs目录，或者启动时添加-Dflume.root.logger=INFO,console 选项前台启动，输出打印日志，
查看具体运行日志，服务异常时查原因。

安装验证

bin/flume-ng version  #查看flume版本
telnet localhost 8888 #发送数据
输入数据
hello world!
hello Flume!
查看 /tmp/log/flume文件中文件内容

flume配置文件说明

Avro模式

Avro可以发送一个给定的文件给Flume，Avro 源使用AVRO RPC机制。
创建agent配置文件

vim /usr/local/flume/cong/avro.conf
 -------------------------------------------avro.conf------------------------------------------------
 a1.sources = r1
 a1.sinks = k1
 a1.channels = c1
  
 # Describe/configure the source
 a1.sources.r1.type = avro
 a1.sources.r1.channels = c1
 a1.sources.r1.bind = 0.0.0.0
 a1.sources.r1.port = 4141
  
 # Describe the sink
 a1.sinks.k1.type = logger
  
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
  
 # Bind the source and sink to the channel
 a1.sources.r1.channels = c1
 a1.sinks.k1.channel = c1
 ------------------------------------------- end ------------------------------------------------
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/avro.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a1
 echo "hello world" > /usr/local/flume/log/log.00                                                                    #创建指定文件
 /usr/local/flume/bin/flume-ng avro-client -c . -H m1 -p 4141 -F /usr/local/flume/log/log.00                         #使用avro-client发送文件
 控制台信息输出

Spool模式

Spool监测配置的目录下新增的文件，并将文件中的数据读取出来。需要注意两点：
     1) 拷贝到spool目录下的文件不可以再打开编辑。
     2) spool目录下不可包含相应的子目录
创建agent配置文件

vim /usr/local/flume/cong/spool.conf
 -------------------------------------------spool.conf------------------------------------------------
 a1.sources = r1
 a1.sinks = k1
 a1.channels = c1
 # Describe/configure the source
 a1.sources.r1.type = spooldir
 a1.sources.r1.channels = c1
 a1.sources.r1.spoolDir = /usr/local/flume/log
 a1.sources.r1.fileHeader = true
 # Describe the sink
 a1.sinks.k1.type = logger
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
 # Bind the source and sink to the channel
 a1.sources.r1.channels = c1
 a1.sinks.k1.channel = c1
 ------------------------------------------------------------------------------------------------------
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/spool.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a2
 echo "spool test1" > /usr/local/flume/log/spool_text.log                                                             #内容追加到指定目录
 控制台信息输出

Exec模式

EXEC执行一个给定的命令获得输出的源,如果要使用tail命令，必选使得file足够大才能看到输出内容
创建agent配置文件

vim /usr/local/flume/cong/exec_tail.conf
 -------------------------------------------exec_tail.conf------------------------------------------------
 a1.sources = r1
 a1.sinks = k1
 a1.channels = c1
 # Describe/configure the source
 a1.sources.r1.type = exec
 a1.sources.r1.channels = c1
 a1.sources.r1.command = tail -F /usr/local/flume/log/log_exec_tail
 # Describe the sink
 a1.sinks.k1.type = logger
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
 # Bind the source and sink to the channel
 a1.sources.r1.channels = c1
 a1.sinks.k1.channel = c1
 ----------------------------------------------------------------------------------------------------------
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/exec_tail.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a3
 for i in {1..100};do echo "exec tail$i" >> /usr/local/flume/log/log_exec_tail;echo $i;sleep 0.1;done             #内容追加到指定目录
 控制台信息输出

Syslogtcp模式

1.Syslogtcp监听TCP的端口做为数据源 
2.Syslogudp监听UDP的端口做为数据源
创建agent配置文件

vim /usr/local/flume/cong/syslog_tcp.conf
 -------------------------------------------syslog_tcp.conf------------------------------------------------
 a1.sources = r1
 a1.sinks = k1
 a1.channels = c1
 # Describe/configure the source
 a1.sources.r1.type = syslogtcp
 a1.sources.r1.port = 5140
 a1.sources.r1.host = localhost
 a1.sources.r1.channels = c1
 # Describe the sink
 a1.sinks.k1.type = logger
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
 # Bind the source and sink to the channel
 a1.sources.r1.channels = c1
 a1.sinks.k1.channel = c1
 -----------------------------------------------------------------------------------------------------------
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/syslog_tcp.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a3
 echo "hello idoall.org syslog" | nc localhost 5140                                                                        #测试产生syslog
 控制台信息输出

JSONHandler模式

创建agent配置文件

vim /usr/local/flume/cong/post_json.conf
 -------------------------------------------post_json.conf------------------------------------------------
 a1.sources = r1
 a1.sinks = k1
 a1.channels = c1
 # Describe/configure the source
 a1.sources.r1.type = org.apache.flume.source.http.HTTPSource
 a1.sources.r1.port = 8888
 a1.sources.r1.channels = c1
 # Describe the sink
 a1.sinks.k1.type = logger
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
 # Bind the source and sink to the channel
 a1.sources.r1.channels = c1
 a1.sinks.k1.channel = c1
 ---------------------------------------------------------------------------------------------------------
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/post_json.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a3
 curl -X POST -d '[{ "headers" :{"a" : "a1","b" : "b1"},"body" : "idoall.org_body"}]' http://localhost:8888               #测试产生syslog
 控制台信息输出

Hadoop sink模式

其中关于hadoop2.2.0部分的安装部署，请参考其他文档
创建agent配置文件

vim /usr/local/flume/cong/hdfs_sink.conf
 -------------------------------------------hdfs_sink.conf------------------------------------------------
 a1.sources = r1
 a1.sinks = k1
 a1.channels = c1
 # Describe/configure the source
 a1.sources.r1.type = syslogtcp
 a1.sources.r1.port = 5140
 a1.sources.r1.host = localhost
 a1.sources.r1.channels = c1
 # Describe the sink
 a1.sinks.k1.type = hdfs
 a1.sinks.k1.channel = c1
 a1.sinks.k1.hdfs.path = hdfs://m1:9000/user/flume/syslogtcp
 a1.sinks.k1.hdfs.filePrefix = Syslog
 a1.sinks.k1.hdfs.round = true
 a1.sinks.k1.hdfs.roundValue = 10
 a1.sinks.k1.hdfs.roundUnit = minute
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
 # Bind the source and sink to the channel
 a1.sources.r1.channels = c1
 a1.sinks.k1.channel = c1
 ---------------------------------------------------------------------------------------------------------
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/hdfs_sink.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a3
 echo "hello idoall flume -> hadoop testing one" | nc localhost 5140                                                      #测试产生syslog
 控制台信息输出

File Roll Sink模式

创建agent配置文件

vim /usr/local/flume/cong/file_roll.conf
 -------------------------------------------file_roll.conf------------------------------------------------
 a1.sources = r1
 a1.sinks = k1
 a1.channels = c1
 # Describe/configure the source
 a1.sources.r1.type = syslogtcp
 a1.sources.r1.port = 5555
 a1.sources.r1.host = localhost
 a1.sources.r1.channels = c1
 # Describe the sink
 a1.sinks.k1.type = file_roll
 a1.sinks.k1.sink.directory = /usr/local/flume/log
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
 # Bind the source and sink to the channel
 a1.sources.r1.channels = c1
 a1.sinks.k1.channel = c1
 --------------------------------------------------------------------------------------------------------
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/file_roll.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a3
 echo "hello idoall.org syslog" | nc localhost 5555
 echo "hello idoall.org syslog 2" | nc localhost 5555
 查看/usr/local/flume/log下是否生成文件,默认每30秒生成一个新文件

Replicating Channel Selector模式

Flume支持Fan out流从一个源到多个通道。有两种模式的Fan out，分别是复制和复用。在复制的情况下，流的事件被发送到
所有的配置通道。在复用的情况下，事件被发送到可用的渠道中的一个子集。Fan out流需要指定源和Fan out通道的规则。
这次需要用到m1,m2两台机器
在m1创建replicating_Channel_Selector配置文件

vim /usr/local/flume/cong/replicating_Channel_Selector.conf
 ------------------------------------------- replicating_Channel_Selector.conf------------------------------------------------
 a1.sources = r1
 a1.sinks = k1 k2
 a1.channels = c1 c2
 # Describe/configure the source
 a1.sources.r1.type = syslogtcp
 a1.sources.r1.port = 5140
 a1.sources.r1.host = localhost
 a1.sources.r1.channels = c1 c2
 a1.sources.r1.selector.type = replicating
 # Describe the sink
 a1.sinks.k1.type = avro
 a1.sinks.k1.channel = c1
 a1.sinks.k1.hostname = m1
 a1.sinks.k1.port = 5555
 a1.sinks.k2.type = avro
 a1.sinks.k2.channel = c2
 a1.sinks.k2.hostname = m2
 a1.sinks.k2.port = 5555
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
 a1.channels.c2.type = memory
 a1.channels.c2.capacity = 1000
 a1.channels.c2.transactionCapacity = 100
 ---------------------------------------------------------------------------------------------------------------------------------
 vim /usr/local/flume/cong/replicating_Channel_Selector_avro.conf
 ------------------------------------------- replicating_Channel_Selector_avro.conf-----------------------------------------------
 a1.sources = r1
 a1.sinks = k1
 a1.channels = c1
 # Describe/configure the source
 a1.sources.r1.type = avro
 a1.sources.r1.channels = c1
 a1.sources.r1.bind = 0.0.0.0
 a1.sources.r1.port = 5555
 # Describe the sink
 a1.sinks.k1.type = logger
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
 # Bind the source and sink to the channel
 a1.sources.r1.channels = c1
 a1.sinks.k1.channel = c1
 ---------------------------------------------------------------------------------------------------------------------------------
 在m1上将2个配置文件复制到m2上一份
 打开4个窗口，在m1和m2上同时启动两个flume agent
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/replicating_Channel_Selector_avro.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a1
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/replicating_Channel_Selector.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a1
 然后在m1或m2的任意一台机器上，测试产生syslog
 echo "hello idoall.org syslog" | nc localhost 5140
 在m1和m2的sink窗口，分别可以看到以下信息,这说明信息得到了同步：
 /12/28 14:08:18 INFO ipc.NettyServer: Connection to /192.168.1.51:46844 disconnected.
 /12/28 14:08:52 INFO ipc.NettyServer: [id: 0x90f8fe1f, /192.168.1.50:35873 => /192.168.1.50:5555] OPEN
 /12/28 14:08:52 INFO ipc.NettyServer: [id: 0x90f8fe1f, /192.168.1.50:35873 => /192.168.1.50:5555] BOUND: /192.168.1.50:5555
 /12/28 14:08:52 INFO ipc.NettyServer: [id: 0x90f8fe1f, /192.168.1.50:35873 => /192.168.1.50:5555] CONNECTED: /192.168.1.50:35873
 /12/28 14:08:59 INFO ipc.NettyServer: [id: 0xd6318635, /192.168.1.51:46858 => /192.168.1.50:5555] OPEN
 /12/28 14:08:59 INFO ipc.NettyServer: [id: 0xd6318635, /192.168.1.51:46858 => /192.168.1.50:5555] BOUND: /192.168.1.50:5555
 /12/28 14:08:59 INFO ipc.NettyServer: [id: 0xd6318635, /192.168.1.51:46858 => /192.168.1.50:5555] CONNECTED: /192.168.1.51:46858
 /12/28 14:09:20 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 68 65 6C 6C 6F 20 69 64 6F 61 6C 6C 2E 6F 72 67 hello idoall.org }

Multiplexing Channel Selector模式

在m1创建Multiplexing_Channel_Selector配置文件

vim /usr/local/flume/cong/Multiplexing_Channel_Selector.conf
 ------------------------------------------- Multiplexing_Channel_Selector.conf------------------------------------------------
 a1.sources = r1
 a1.sinks = k1 k2
 a1.channels = c1 c2
 # Describe/configure the source
 a1.sources.r1.type = org.apache.flume.source.http.HTTPSource
 a1.sources.r1.port = 5140
 a1.sources.r1.channels = c1 c2
 a1.sources.r1.selector.type = multiplexing
 a1.sources.r1.selector.header = type
 #映射允许每个值通道可以重叠。默认值可以包含任意数量的通道。
 a1.sources.r1.selector.mapping.baidu = c1
 a1.sources.r1.selector.mapping.ali = c2
 a1.sources.r1.selector.default = c1
 # Describe the sink
 a1.sinks.k1.type = avro
 a1.sinks.k1.channel = c1
 a1.sinks.k1.hostname = m1
 a1.sinks.k1.port = 5555
 a1.sinks.k2.type = avro
 a1.sinks.k2.channel = c2
 a1.sinks.k2.hostname = m2
 a1.sinks.k2.port = 5555
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
 a1.channels.c2.type = memory
 a1.channels.c2.capacity = 1000
 a1.channels.c2.transactionCapacity = 100
 ------------------------------------------------------------------------------------------------------------------------------
 vim /usr/local/flume/cong/Multiplexing_Channel_Selector_avro.conf
 ------------------------------------------- Multiplexing_Channel_Selector_avro.conf------------------------------------------------
 a1.sources = r1
 a1.sinks = k1
 a1.channels = c1
 # Describe/configure the source
 a1.sources.r1.type = avro
 a1.sources.r1.channels = c1
 a1.sources.r1.bind = 0.0.0.0
 a1.sources.r1.port = 5555
 # Describe the sink
 a1.sinks.k1.type = logger
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
 # Bind the source and sink to the channel
 a1.sources.r1.channels = c1
 a1.sinks.k1.channel = c1
 -----------------------------------------------------------------------------------------------------------------------------------
 在m1上将2个配置文件复制到m2上一份
 打开4个窗口，在m1和m2上同时启动两个flume agent
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/Multiplexing_Channel_Selector.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a1
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/Multiplexing_Channel_Selector_avro.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a1
 然后在m1或m2的任意一台机器上，测试产生syslog
 curl -X POST -d '[{ "headers" :{"type" : "baidu"},"body" : "idoall_TEST1"}]' http://localhost:5140 && curl -X POST -d '[{ "headers" :{"type" : "ali"},"body" : "idoall_TEST2"}]' http://localhost:5140 && curl -X POST -d '[{ "headers" :{"type" :   "qq"},"body" : "idoall_TEST3"}]' http://localhost:5140

 ----在m1的sink窗口，可以看到以下信息----->
 16/12/28 14:32:21 INFO node.Application: Starting Sink k1
 16/12/28 14:32:21 INFO node.Application: Starting Source r1
 16/12/28 14:32:21 INFO source.AvroSource: Starting Avro source r1: { bindAddress: 0.0.0.0, port: 5555 }...
 16/12/28 14:32:21 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
 16/12/28 14:32:21 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: r1 started
 16/12/28 14:32:21 INFO source.AvroSource: Avro source r1 started.
 16/12/28 14:32:36 INFO ipc.NettyServer: [id: 0xcf00eea6, /192.168.1.50:35916 => /192.168.1.50:5555] OPEN
 16/12/28 14:32:36 INFO ipc.NettyServer: [id: 0xcf00eea6, /192.168.1.50:35916 => /192.168.1.50:5555] BOUND: /192.168.1.50:5555
 16/12/28 14:32:36 INFO ipc.NettyServer: [id: 0xcf00eea6, /192.168.1.50:35916 => /192.168.1.50:5555] CONNECTED: /192.168.1.50:35916
 16/12/28 14:32:44 INFO ipc.NettyServer: [id: 0x432f5468, /192.168.1.51:46945 => /192.168.1.50:5555] OPEN
 16/12/28 14:32:44 INFO ipc.NettyServer: [id: 0x432f5468, /192.168.1.51:46945 => /192.168.1.50:5555] BOUND: /192.168.1.50:5555
 16/12/28 14:32:44 INFO ipc.NettyServer: [id: 0x432f5468, /192.168.1.51:46945 => /192.168.1.50:5555] CONNECTED: /192.168.1.51:46945 
 16/12/28 14:34:11 INFO sink.LoggerSink: Event: { headers:{type=baidu} body: 69 64 6F 61 6C 6C 5F 54 45 53 54 31       idoall_TEST1 }
 16/12/28 14:34:57 INFO sink.LoggerSink: Event: { headers:{type=qq} body: 69 64 6F 61 6C 6C 5F 54 45 53 54 33       idoall_TEST3 }
 ----在m2的sink窗口，可以看到以下信息----->
 16/12/28 14:32:27 INFO node.Application: Starting Sink k1
 16/12/28 14:32:27 INFO node.Application: Starting Source r1
 16/12/28 14:32:27 INFO source.AvroSource: Starting Avro source r1: { bindAddress: 0.0.0.0, port: 5555 }...
 16/12/28 14:32:27 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
 16/12/28 14:32:27 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: r1 started
 16/12/28 14:32:27 INFO source.AvroSource: Avro source r1 started.
 16/12/28 14:32:36 INFO ipc.NettyServer: [id: 0x7c2f0aec, /192.168.1.50:38104 => /192.168.1.51:5555] OPEN
 16/12/28 14:32:36 INFO ipc.NettyServer: [id: 0x7c2f0aec, /192.168.1.50:38104 => /192.168.1.51:5555] BOUND: /192.168.1.51:5555
 16/12/28 14:32:36 INFO ipc.NettyServer: [id: 0x7c2f0aec, /192.168.1.50:38104 => /192.168.1.51:5555] CONNECTED: /192.168.1.50:38104
 16/12/28 14:32:44 INFO ipc.NettyServer: [id: 0x3d36f553, /192.168.1.51:48599 => /192.168.1.51:5555] OPEN
 16/12/28 14:32:44 INFO ipc.NettyServer: [id: 0x3d36f553, /192.168.1.51:48599 => /192.168.1.51:5555] BOUND: /192.168.1.51:5555
 16/12/28 14:32:44 INFO ipc.NettyServer: [id: 0x3d36f553, /192.168.1.51:48599 => /192.168.1.51:5555] CONNECTED: /192.168.1.51:48599
 16/12/28 14:34:33 INFO sink.LoggerSink: Event: { headers:{type=ali} body: 69 64 6F 61 6C 6C 5F 54 45 53 54 32       idoall_TEST2 }

可以看到，根据header中不同的条件分布到不同的channel上

Flume Sink Processors 模式

failover的机器是一直发送给其中一个sink，当这个sink不可用的时候，自动发送到下一个sink
在m1创建Flume_Sink_Processors配置文件

vim /usr/local/flume/cong/Flume_Sink_Processors.conf
 ------------------------------------------- Flume_Sink_Processors.conf------------------------------------------------
 a1.sources = r1
 a1.sinks = k1 k2
 a1.channels = c1 c2
  
 #这个是配置failover的关键，需要有一个sink group
 a1.sinkgroups = g1
 a1.sinkgroups.g1.sinks = k1 k2
 #处理的类型是failover
 a1.sinkgroups.g1.processor.type = failover
 #优先级，数字越大优先级越高，每个sink的优先级必须不相同
 a1.sinkgroups.g1.processor.priority.k1 = 5
 a1.sinkgroups.g1.processor.priority.k2 = 10
 #设置为10秒，当然可以根据你的实际状况更改成更快或者很慢
 a1.sinkgroups.g1.processor.maxpenalty = 10000
  
 # Describe/configure the source
 a1.sources.r1.type = syslogtcp
 a1.sources.r1.port = 5140
 a1.sources.r1.channels = c1 c2
 a1.sources.r1.selector.type = replicating
   
 # Describe the sink
 a1.sinks.k1.type = avro
 a1.sinks.k1.channel = c1
 a1.sinks.k1.hostname = m1
 a1.sinks.k1.port = 5555
   
 a1.sinks.k2.type = avro
 a1.sinks.k2.channel = c2
 a1.sinks.k2.hostname = m2
 a1.sinks.k2.port = 5555
   
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
   
 a1.channels.c2.type = memory
 a1.channels.c2.capacity = 1000
 a1.channels.c2.transactionCapacity = 100
 -----------------------------------------------------------------------------------------------------------------------------------
 vim /usr/local/flume/cong/Flume_Sink_Processors_avro.conf
 ------------------------------------------- Flume_Sink_Processors_avro.conf--------------------------------------------------------
 a1.sources = r1
 a1.sinks = k1
 a1.channels = c1
  
 # Describe/configure the source
 a1.sources.r1.type = avro
 a1.sources.r1.channels = c1
 a1.sources.r1.bind = 0.0.0.0
 a1.sources.r1.port = 5555
  
 # Describe the sink
 a1.sinks.k1.type = logger
  
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
  
 # Bind the source and sink to the channel
 a1.sources.r1.channels = c1
 a1.sinks.k1.channel = c1
 ------------------------------------------------------------------------------------------------------------------------------------
 在m1上将2个配置文件复制到m2上一份
 打开4个窗口，在m1和m2上同时启动两个flume agent
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/Flume_Sink_Processors_avro.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a1
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/Flume_Sink_Processors.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a1
 然后在m1或m2的任意一台机器上，测试产生log
 echo "idoall.org test1 failover" | nc localhost 5140
 因为m2的优先级高，所以在m2的sink窗口，可以看到以下信息，而m1没有：
 16/12/28 15:02:46 INFO ipc.NettyServer: Connection to /192.168.1.51:48692 disconnected.
 16/12/28 15:03:12 INFO ipc.NettyServer: [id: 0x09a14036, /192.168.1.51:48704 => /192.168.1.51:5555] OPEN
 16/12/28 15:03:12 INFO ipc.NettyServer: [id: 0x09a14036, /192.168.1.51:48704 => /192.168.1.51:5555] BOUND: /192.168.1.51:5555
 16/12/28 15:03:12 INFO ipc.NettyServer: [id: 0x09a14036, /192.168.1.51:48704 => /192.168.1.51:5555] CONNECTED: /192.168.1.51:48704
 16/12/28 15:03:26 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 31 idoall.org test1 }
 这时我们停止掉m2机器上的sink(ctrl+c)，再次输出测试数据：
 echo "idoall.org test2 failover" | nc localhost 5140
 可以在m1的sink窗口，看到读取到了刚才发送的两条测试数据：
 16/12/28 15:02:46 INFO ipc.NettyServer: Connection to /192.168.1.51:47036 disconnected.
 16/12/28 15:03:12 INFO ipc.NettyServer: [id: 0xbcf79851, /192.168.1.51:47048 => /192.168.1.50:5555] OPEN
 16/12/28 15:03:12 INFO ipc.NettyServer: [id: 0xbcf79851, /192.168.1.51:47048 => /192.168.1.50:5555] BOUND: /192.168.1.50:5555
 16/12/28 15:03:12 INFO ipc.NettyServer: [id: 0xbcf79851, /192.168.1.51:47048 => /192.168.1.50:5555] CONNECTED: /192.168.1.51:47048
 16/12/28 15:07:56 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 31 idoall.org test1 }
 16/12/28 15:07:56 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 32 idoall.org test2 }
 我们再在m2的sink窗口中，启动sink：
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/Flume_Sink_Processors_avro.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a1
 输入两批测试数据
 echo "idoall.org test3 failover" | nc localhost 5140 && echo "idoall.org test4 failover" | nc localhost 5140
 在m2的sink窗口，我们可以看到以下信息，因为优先级的关系，log消息会再次落到m2上
 16/12/28 15:09:47 INFO node.Application: Starting Sink k1
 16/12/28 15:09:47 INFO node.Application: Starting Source r1
 16/12/28 15:09:47 INFO source.AvroSource: Starting Avro source r1: { bindAddress: 0.0.0.0, port: 5555 }...
 16/12/28 15:09:47 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
 16/12/28 15:09:47 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: r1 started
 16/12/28 15:09:47 INFO source.AvroSource: Avro source r1 started.
 16/12/28 15:09:54 INFO ipc.NettyServer: [id: 0x96615732, /192.168.1.51:48741 => /192.168.1.51:5555] OPEN
 16/12/28 15:09:54 INFO ipc.NettyServer: [id: 0x96615732, /192.168.1.51:48741 => /192.168.1.51:5555] BOUND: /192.168.1.51:5555
 16/12/28 15:09:54 INFO ipc.NettyServer: [id: 0x96615732, /192.168.1.51:48741 => /192.168.1.51:5555] CONNECTED: /192.168.1.51:48741
 16/12/28 15:09:57 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 32 idoall.org test2 }
 16/12/28 15:10:43 INFO ipc.NettyServer: [id: 0x12621f9a, /192.168.1.50:38166 => /192.168.1.51:5555] OPEN
 16/12/28 15:10:43 INFO ipc.NettyServer: [id: 0x12621f9a, /192.168.1.50:38166 => /192.168.1.51:5555] BOUND: /192.168.1.51:5555
 16/12/28 15:10:43 INFO ipc.NettyServer: [id: 0x12621f9a, /192.168.1.50:38166 => /192.168.1.51:5555] CONNECTED: /192.168.1.50:38166
 16/12/28 15:10:43 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 33 idoall.org test3 }
 16/12/28 15:10:43 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 34 idoall.org test4 }

Load balancing Sink Processor模式

load balance type和failover不同的地方是，load balance有两个配置，一个是轮询，一个是随机。两种情况下如果被选择的sink不可用，就会自动尝试发送到下一个可用的sink上面。
在m1创建Load_balancing_Sink_Processors配置文件

vim /usr/local/flume/cong/Load_balancing_Sink_Processors.conf
 ------------------------------------------- Load_balancing_Sink_Processors.conf------------------------------------------------
 a1.sources = r1
 a1.sinks = k1 k2
 a1.channels = c1
  
 #这个是配置Load balancing的关键，需要有一个sink group
 a1.sinkgroups = g1
 a1.sinkgroups.g1.sinks = k1 k2
 a1.sinkgroups.g1.processor.type = load_balance
 a1.sinkgroups.g1.processor.backoff = true
 a1.sinkgroups.g1.processor.selector = round_robin
  
 # Describe/configure the source
 a1.sources.r1.type = syslogtcp
 a1.sources.r1.port = 5140
 a1.sources.r1.channels = c1
   
 # Describe the sink
 a1.sinks.k1.type = avro
 a1.sinks.k1.channel = c1
 a1.sinks.k1.hostname = m1
 a1.sinks.k1.port = 5555
  
 a1.sinks.k2.type = avro
 a1.sinks.k2.channel = c1
 a1.sinks.k2.hostname = m2
 a1.sinks.k2.port = 5555
   
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
 ---------------------------------------------------------------------------------------------------------------------------------
 在m1创建Load_balancing_Sink_Processors_avro配置文件
 vim /usr/local/flume/cong/Load_balancing_Sink_Processors_avro.conf
 ------------------------------------------- Load_balancing_Sink_Processors_avro.conf----------------------------------------------
 a1.sources = r1
 a1.sinks = k1
 a1.channels = c1
  
 # Describe/configure the source
 a1.sources.r1.type = avro
 a1.sources.r1.channels = c1
 a1.sources.r1.bind = 0.0.0.0
 a1.sources.r1.port = 5555
  
 # Describe the sink
 a1.sinks.k1.type = logger
  
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
  
 # Bind the source and sink to the channel
 a1.sources.r1.channels = c1
 a1.sinks.k1.channel = c1
 ----------------------------------------------------------------------------------------------------------------------------------
 在m1上将2个配置文件复制到m2上一份
 打开4个窗口，在m1和m2上同时启动两个flume agent
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/Load_balancing_Sink_Processors_avro.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a1
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/Load_balancing_Sink_Processors.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a1
 然后在m1或m2的任意一台机器上，测试产生log，一行一行输入，输入太快，容易落到一台机器上
 echo "idoall.org test1" | nc localhost 5140
 echo "idoall.org test2" | nc localhost 5140
 echo "idoall.org test3" | nc localhost 5140
 echo "idoall.org test4" | nc localhost 5140
 在m1的sink窗口，可以看到以下信息：
 16/12/28 15:35:29 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 32 idoall.org test2 }
 16/12/28 15:35:33 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 34 idoall.org test4 }
 在m2的sink窗口，可以看到以下信息：
 16/12/28 15:35:27 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 31 idoall.org test1 }
 16/12/28 15:35:29 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 33 idoall.org test3 }
 说明轮询模式起到了作用

Hbase sink模式

在测试之前，请先参考《ubuntu12.04+hadoop2.2.0+zookeeper3.4.5+hbase0.96.2+hive0.13.1分布式环境部署》将hbase启动
然后将以下文件复制到flume中：

cp /home/hadoop/hbase-0.96.2-hadoop2/lib/protobuf-java-2.5.0.jar /usr/local/flume/lib
 cp /home/hadoop/hbase-0.96.2-hadoop2/lib/hbase-client-0.96.2-hadoop2.jar /usr/local/flume/lib
 cp /home/hadoop/hbase-0.96.2-hadoop2/lib/hbase-common-0.96.2-hadoop2.jar /usr/local/flume/lib
 cp /home/hadoop/hbase-0.96.2-hadoop2/lib/hbase-protocol-0.96.2-hadoop2.jar /usr/local/flume/lib
 cp /home/hadoop/hbase-0.96.2-hadoop2/lib/hbase-server-0.96.2-hadoop2.jar /usr/local/flume/lib
 cp /home/hadoop/hbase-0.96.2-hadoop2/lib/hbase-hadoop2-compat-0.96.2-hadoop2.jar /usr/local/flume/lib
 cp /home/hadoop/hbase-0.96.2-hadoop2/lib/hbase-hadoop-compat-0.96.2-hadoop2.jar /usr/local/flume/lib@@@
 cp /home/hadoop/hbase-0.96.2-hadoop2/lib/htrace-core-2.04.jar /usr/local/flume/lib

 确保test_idoall_org表在hbase中已经存在，test_idoall_org表的格式以及字段请参考《ubuntu12.04+hadoop2.2.0+zookeeper3.4.5+hbase0.96.2+hive0.13.1分布式环境部署》中关于hbase部分的建表代码。
 在m1创建hbase_simple配置文件
 vim /usr/local/flume/cong/hbase_simple.conf
 ------------------------------------------- hbase_simple.conf------------------------------------------------------
 a1.sources = r1
 a1.sinks = k1
 a1.channels = c1
  
 # Describe/configure the source
 a1.sources.r1.type = syslogtcp
 a1.sources.r1.port = 5140
 a1.sources.r1.host = localhost
 a1.sources.r1.channels = c1
  
 # Describe the sink
 a1.sinks.k1.type = logger
 a1.sinks.k1.type = hbase
 a1.sinks.k1.table = test_idoall_org
 a1.sinks.k1.columnFamily = name
 a1.sinks.k1.column = idoall
 a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
 a1.sinks.k1.channel = memoryChannel
  
 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100
   
 # Bind the source and sink to the channel
 a1.sources.r1.channels = c1
 a1.sinks.k1.channel = c1 
 -------------------------------------------------------------------------------------------------------------------
 启动flume agent
 /usr/local/flume/bin/flume-ng agent -c . -f /usr/local/flume/conf/hbase_simple.conf -n a1 -Dflume.root.logger=INFO,console  #启动flume agent a1
 测试产生syslog
 echo "hello idoall.org from flume" | nc localhost 5140
 这时登录到hbase中，可以发现新数据已经插入

Kafka模式

1创建kafka.conf配置文件
vim /usr/local/flume/cong/kafka.conf
------------------------------------------- kafka.conf------------------------------------------------------
agent.sources = r1
agent.channels = c1
agent.sinks = s1
# For each one of the sources, the type is defined
agent.sources.r1.type = syslogudp
agent.sources.r1.bind = 192.168.201.73
agent.sources.r1.port = 7520
#消息平均分布在多个分区上  官方自带插件
agent.sources.r1.interceptors = i2
agent.sources.r1.interceptors.i2.type=org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder
agent.sources.r1.interceptors.i2.headerName=key
agent.sources.r1.interceptors.i2.preserveExisting=false

# The channel can be defined as follows.
agent.sources.r1.channels = c1
# Each sink's type must be defined
agent.sinks.s1.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.s1.topic=php      #kafka消息池
agent.sinks.s1.brokerList=192.168.201.73:9092 #kafka主机和端口
agent.sinks.s1.requiredAcks = 1
agent.sinks.s1.batchSize = 20
agent.sinks.s1.channel = c1 

#agent.sinks.s1.requireAcks 0不需要确认消息是否成功 -1 确认消息成功 还需备份 
#agent.sinks.s1.batchSize  配置每次下沉多少条消息 每次下沉的数量越多延迟越高
# Each channel's type is defined.
agent.channels.c1.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.c1.capacity = 10000
-----------------------------------------------------------------------------------------------------------------
启动flume /usr/local/flume/bin/flume-ng agent --conf conf -f conf/kafka.conf -n agent&

转载于:https://my.oschina.net/u/1757002/blog/868519