flume简易高可用（组件高可用）

一、使用范围

       适用于单机环境，或者flume非跨服务器高可用架构

        场景：flume更新需要重启（一般重启阶段可能丢失数据，如果做个简易集群，可以避免该问题）

二、配置

       flume.conf配置与flume01.conf配置差别（其余配置一致）：

agent.sources.s1.bind = 172.16.1.25
agent.sources.s1.bind =  0.0.0.0

      以下是完整的配置：

      flume.conf配置

#flume源、channel、sink名称
agent.sources = s1
agent.sinks = k1
agent.channels = c1

#flume源类型
agent.sources.s1.type = syslogudp
#flume绑定网络地址
agent.sources.s1.bind = 172.16.1.25
#flume绑定端口
agent.sources.s1.port = 514
#flume源绑定的channel名称
agent.sources.s1.channels = c1
#flume拦截器配置
agent.sources.s1.interceptors=i1 i2 i3 i4
agent.sources.s1.interceptors.i1.type = static
agent.sources.s1.interceptors.i1.key=key
agent.sources.s1.interceptors.i1.value={"agent_name":"read"}
agent.sources.s1.keepFields = true
agent.sources.s1.interceptors.i2.type = com.fs.flume.source.interceptor.EZEncodingInterceptor$Builder
agent.sources.s1.interceptors.i3.type = com.fusionskye.interceptor.MorphlineHeadToBodyInterceptor$Builder
# 通过比较主机 IP 来插入配置文件给定的键值对
agent.sources.s1.interceptors.i3.hostIp2EntriesPath=/opt/ops-ng/flume/flume/conf/hostIp2Entries.json

agent.sources.s1.interceptors.i4.type = org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder
agent.sources.s1.interceptors.i4.headerName=key
agent.sources.s1.interceptors.i4.preserveExisting=false

#flume channel配置
agent.channels.c1.type = SPILLABLEMEMORY
agent.channels.c1.capacity = 1000000
agent.channels.c1.overflowCapacity = 2000000
agent.channels.c1.byteCapacity = 800000
agent.channels.c1.checkpointDir = /ezdata/flume/flume/read/checkpoint1
agent.channels.c1.dataDirs = /ezdata/flume/flume/read/data1
agent.channels.c1.transactionCapacity = 5000

#flume sink类型
agent.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
#kafka topic名称
agent.sinks.k1.topic = read
agent.sinks.k1.custom.encoding=UTF-8
#kafka地址
agent.sinks.k1.brokerList = 172.16.1.25:9092
#sink绑定channel名称
agent.sinks.k1.channel=c1

    flume01.conf配置

#flume源、channel、sink名称
agent.sources = s1
agent.sinks = k1
agent.channels = c1

#flume源类型
agent.sources.s1.type = syslogudp
#flume绑定网络地址
agent.sources.s1.bind = 0.0.0.0
#flume绑定端口
agent.sources.s1.port = 514
#flume源绑定的channel名称
agent.sources.s1.channels = c1
#flume拦截器配置
agent.sources.s1.interceptors=i1 i2 i3 i4
agent.sources.s1.interceptors.i1.type = static
agent.sources.s1.interceptors.i1.key=key
agent.sources.s1.interceptors.i1.value={"agent_name":"read"}
agent.sources.s1.keepFields = true
agent.sources.s1.interceptors.i2.type = com.fs.flume.source.interceptor.EZEncodingInterceptor$Builder
agent.sources.s1.interceptors.i3.type = com.fusionskye.interceptor.MorphlineHeadToBodyInterceptor$Builder
# 通过比较主机 IP 来插入配置文件给定的键值对
agent.sources.s1.interceptors.i3.hostIp2EntriesPath=/opt/ops-ng/flume/flume/conf/hostIp2Entries.json
agent.sources.s1.interceptors.i4.type = org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder
agent.sources.s1.interceptors.i4.headerName=key
agent.sources.s1.interceptors.i4.preserveExisting=false

#flume channel配置
agent.channels.c1.type = SPILLABLEMEMORY
agent.channels.c1.capacity = 1000000
agent.channels.c1.overflowCapacity = 2000000
agent.channels.c1.byteCapacity = 800000
agent.channels.c1.checkpointDir = /ezdata/flume/flume/read/checkpoint1
agent.channels.c1.dataDirs = /ezdata/flume/flume/read/data1
agent.channels.c1.transactionCapacity = 5000

#flume sink类型
agent.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
#kafka topic名称
agent.sinks.k1.topic = read
agent.sinks.k1.custom.encoding=UTF-8
#kafka地址
agent.sinks.k1.brokerList = 172.16.1.25:9092
#sink绑定channel名称
agent.sinks.k1.channel=c1

三、验证

       两个配置文件启动后，开启测试。#注意修改启动脚本的端口

      在172.16.1.12上面打数据：echo "123456" > /dev/udp/172.16.1.25/514

      并且172.16.1.25上消费kafka数据：./kafka-console-consumer.sh --bootstrap-server 172.16.1.25:9092 --topic read

      测试1：

      直接打一条数据，kafka只消费到1条数据，说明数据未重复消费

      测试2：

      关闭任意一个flume进程，再打一条数据，kafka只消费到1条数据，重复多次，都只消费到1条数据，说明实现数据高可用

      测试3：

      随意打数据，均未丢失或者重复消费数据