Flume拦截器自定义开发

一.流程图

二.拦截器开发

添加依赖：

<dependency>
	<groupId>org.apache.flume</groupId>
	<artifactId>flume-ng-core</artifactId>
	<version>1.6.0-cdh5.15.1</version>
</dependency>

开始网不好，下载的依赖总是爆红。
也可以直接去 https://repository.cloudera.com/artifactory/cloudera-repos/ 下载jar包。
下载 flume-ng-core-1.6.0-cdh5.15.1.jar 和 flume-ng-sdk-1.6.0-cdh5.15.1.jar 两个jar包。然后加入到module中。

Interceptor：

public class DomainIntercepter implements Interceptor {

    List<Event> events;
    @Override
    public void initialize() {
        events = new ArrayList<>();
    }

    @Override
    public Event intercept(Event event) {
        Map<String, String> headers = event.getHeaders();
        String body = new String(event.getBody());

        if (body.contains("imooc")){
            headers.put("type","imooc");
        }else
            headers.put("type","others");
        return event;
    }

    @Override
    public List<Event> intercept(List<Event> list) {
        events.clear();
        for (Event event : list){
            events.add(intercept(event));
        }
        return events;
    }

    @Override
    public void close() {
        events = null;
    }

    public static class Builder implements Interceptor.Builder{

        @Override
        public Interceptor build() {
            return new DomainIntercepter();
        }

        @Override
        public void configure(Context context) {
        }
    }
}

通过判断body中的信息在header中加入key值，selector根据key值进行分类到指定channel。
写好后打成jar包，加入到 $FLUME_HOME/lib 目录下。

三.配置文件

flume01.conf

a1.sources = r1
a1.channels = c1 c2           //两个channel
a1.sinks = k1 k2              //两个sink

a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 44444

a1.sources.r1.interceptors = i1        //自定义拦截器
a1.sources.r1.interceptors.i1.type = com.chengyanban.DomainIntercepter$Builder

a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = type
a1.sources.r1.selector.mapping.imooc = c1      //event的key值
a1.sources.r1.selector.mapping.others = c2      //event的key值

a1.channels.c1.type = memory
a1.channels.c2.type = memory

a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop000
a1.sinks.k1.port = 44445

a1.sinks.k2.type = avro
a1.sinks.k2.hostname = hadoop000
a1.sinks.k2.port = 44446

a1.sources.r1.channels = c1 c2
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2

flume02.conf

a2.sources = r1
a2.sinks = k1
a2.channels = c1

a2.sources.r1.type = avro
a2.sources.r1.bind = hadoop000
a2.sources.r1.port = 44445

a2.channels.c1.type = memory

a2.sinks.k1.type = logger

a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1

flume03.conf

a3.sources = r1
a3.sinks = k1
a3.channels = c1

a3.sources.r1.type = avro
a3.sources.r1.bind = hadoop000
a3.sources.r1.port = 44446

a3.channels.c1.type = memory

a3.sinks.k1.type = logger

a3.sources.r1.channels = c1
a3.sinks.k1.channel = c1

四.运行结果

[hadoop@hadoop000 apache-flume-1.6.0-cdh5.15.1-bin]$ telnet localhost 44444
Trying 192.168.198.128...
Connected to localhost.
Escape character is '^]'.
hi
OK
immoc^H^H^H
OK
imooc.com
OK
www.imooc.com
OK

//agent:a2
21/01/25 16:01:15 INFO sink.LoggerSink: Event: { headers:{type=imooc} body: 69 6D 6F 6F 63 2E 63 6F 6D 0D                   imooc.com. }
21/01/25 16:01:17 INFO sink.LoggerSink: Event: { headers:{type=imooc} body: 77 77 77 2E 69 6D 6F 6F 63 2E 63 6F 6D 0D       www.imooc.com. }


//agent:a3
21/01/25 15:14:34 INFO sink.LoggerSink: Event: { headers:{type=others} body: 68 69 0D                                        hi. }
21/01/25 16:01:13 INFO sink.LoggerSink: Event: { headers:{type=others} body: 69 6D 6D 6F 63 08 08 08 0D                      immoc.... }