Flume Interceptors:拦截器,与Spring中拦截器是类似
功能:通过拦截器对每条数据进行过滤护或者包装
Timestamp Interceptor:时间拦截器
在每一个event的头部添加一个KeyValue
key: timestamp
value:当前封装event的时间
Host Interceptor:主机名拦截器
在每一个event的头部添加一个KeyValue
key: host
value:当前封装event所在机器的主机名
Static Interceptor:自定义拦截器
Regex Filtering Interceptor通过自定义正则表达式,实现对数据过滤
符合该正则,该条数据才会被留下
实现过滤配置文件
# in this case called 'a1'
a1.sources = s1
a1.channels = c1
a1.sinks = k1
# For each one of the sources, the type is defined
a1.sources.s1.type = exec
a1.sources.s1.command = tail -F /opt/datas/20190716.txt
a1.sources.s1.shell = /bin/sh -c
a1.sources.s1.interceptors = i1 i2 i3
#设置时间过滤
a1.sources.s1.interceptors.i1.type = timestamp
#设置主机名过滤
a1.sources.s1.interceptors.i2.type = host
a1.sources.s1.interceptors.i2.useIP = false
a1.sources.s1.interceptors.i2.hostHeader = hostname
#设置正则过滤
a1.sources.s1.interceptors.i3.type = regex_filter
#a1.sources.s1.ignorePattern=([^ ]*\.tmp)
a1.sources.s1.interceptors.i3.excludeEvents = false
a1.sources.s1.interceptors.i3.regex= \\{*\\}
# Each channel's type is defined.
a1.channels.c1.type = file
a1.channels.c1.checkpointDir=/opt/datas/flume/channel/checkpoint
a1.channels.c1.dataDirs=/opt/datas/flume/channel/data
# Each sink's type must be defined
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path=/flume/time_filter/
a1.sinks.k1.hdfs.path=/flume/time_filter/yearst=%Y/monthstr=%m/daystr=%d/minutestr=%M
#设置文件类型和写的格式,解决中文乱码
a1.sinks.k1.hdfs.fileType=DataStream
a1.sinks.k1.hdfs.writeFormat=Text
#设置多久进行一次保存
a1.sinks.k1.hdfs.rollInterval=0
#设置HDFS文件大小
a1.sinks.k1.hdfs.rollSize=10240
#在滚动之前写入文件的事件数
a1.sinks.k1.hdfs.rollCount=0
#Specify the channel the sink should use
a1.sinks.k1.channel = c1
a1.sources.s1.channels = c1
Flume Channel Selectors选择器
selector.type根据该值确定功能
Replicating Channel Selector(默认)
source将每条数据发给每一个channel
source将数据发了多份
Multiplexing Channel Selector
Flume Sink Processors(sink的处理器)
Failover Sink Processor(故障转移)
启动了多个,但是工作的只有一个,只有active状态进程死掉,其他才可能接替工作。
那么多个有多个sink到底谁先工作,根据权重来,谁的权重高,谁先干活
一般故障转移的话,2个sink的类型不一样(HDFS sink ,file sink)
比如往HDFS写数据,HDFS宕机了,数据不丢失,往文件里写
Load balancing Sink Processor(负载均衡)
processor.type=load_balance
processor.selector = round_robin(轮询)|random(随机)
负载均衡与故障转移,只能实现一个,不能同时实现,往往选择负载均衡