Flume-自定义拦截器

自定义拦截器Interceptor

拦截器一般与选择器共同使用，来对数据进行拦截处理，并可以根据数据的不同来进行区分处理。

需求：使用Flume采集服务器本地日志，需要根据日志类型的不同，将不同种类的日志发往不同的分析系统（HDFS）。

此时会用到Flume拓扑结构中的Mutiplexing结构，其原理是根据event中的header的某个key的值，将不同的event发送到不同的channel中，因此我们需要自定义一个interceptor，为不同类型的event的header中的key赋予不同的值。

在本案例中，我们可以以端口数据模拟日志，以是否包含test字符串来模拟不同类型的日志，通过自定义的interceptor区分否包含test字符串，将其分别发送到不同的channel中，并打印出来。

我们在hadoop113上监听端口44444，并通过flume打印到hadoop114或者hadoop115的控制台。

新建一个maven工程，并在pom文件中导入如下包

<dependencies>
    <dependency>
        <groupId>org.apache.flume</groupId>
        <artifactId>flume-ng-core</artifactId>
        <version>1.9.0</version>
    </dependency>
</dependencies>

然后新建一个Interceptor类，如下：

public class JudgeTestStringInterceptor implements Interceptor {

    // 声明一个存放事件的List
    private List<Event> allEvents;

    public void initialize () {

        // 初始化
        allEvents = new ArrayList<Event>();
    }

    /**
     * 单个事件拦截
     * @param event
     * @return
     */
    public Event intercept (Event event) {

        // 1、获取事件中的头信息
        Map<String, String> headers = event.getHeaders();

        // 2、获取事件中的body信息
        String body = new String(event.getBody());

        // 3、根据body中是否有“test”来决定添加怎样的头信息
        // 有的话添加<type, test>没有则添加<type, other>
        if (body.contains("test")) {
            headers.put("type", "test");
        } else {
            headers.put("type", "other");
        }

        return event;
        // 如果返回null则认为该事件无用，将会被过滤
    }

    /**
     * 批量事件拦截
     * @param list
     * @return
     */
    public List<Event> intercept (List<Event> list) {

        // 1、清空集合
        allEvents.clear();

        // 2、遍历event
        for (Event event : list) {
            // 3、给每个事件添加头信息
            allEvents.add(intercept(event));
        }

        return allEvents;
    }

    public void close () {

    }

    // 定义一个Builder对象
    public static class Builder implements Interceptor.Builder {

        public Interceptor build () {

            return new JudgeTestStringInterceptor();
        }

        public void configure (Context context) {

        }
    }
}

把将其打包放到flume的lib目录下。接下来进行配置文件的书写。

Flume1配置文件如下：

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1 c2

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# set interceptors
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = com.starnet.interceptor.JudgeTestStringInterceptor$Builder
# set channel selector
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = type
a1.sources.r1.selector.mapping.test = c1
a1.sources.r1.selector.mapping.other = c2

# Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop114
a1.sinks.k1.port = 4444
a1.sinks.k2.type = avro
a1.sinks.k2.hostname = hadoop115
a1.sinks.k2.port = 4444

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.channels.c2.type = memory
a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1 c2
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2 

Flume2配置文件如下：

# Name the components on this agent
a2.sources = r2
a2.sinks = k2
a2.channels = c2

# Describe/configure the source
a2.sources.r2.type = avro
a2.sources.r2.bind = hadoop114
a2.sources.r2.port = 4444

# Describe the sink
a2.sinks.k2.type = logger

# Use a channel which buffers events in memory
a2.channels.c2.type = memory
a2.channels.c2.capacity = 1000
a2.channels.c2.transactionCapacity = 100

# Bind the source and sink to the channel
a2.sources.r2.channels = c2
a2.sinks.k2.channel = c2

Flume3配置文件如下：

# Name the components on this agent
a3.sources = r2
a3.sinks = k2
a3.channels = c2

# Describe/configure the source
a3.sources.r2.type = avro
a3.sources.r2.bind = hadoop115
a3.sources.r2.port = 4444

# Sink
a3.sinks.k2.type = logger

# Use a channel which buffers events in memory
a3.channels.c2.type = memory
a3.channels.c2.capacity = 1000
a3.channels.c2.transactionCapacity = 100

# Bind the source and sink to the channel
a3.sources.r2.channels = c2
a3.sinks.k2.channel = c2

先启动Flume3和Flume2，最后启动Flume；然后使用nc往113的44444端口一次发送test、test1、hello、word、testhelloword查看结果如下：

可以看到包含test的和不包含test的被分别打印了，说明最终拦截成功了。