flume 拦截器简单实现

最新推荐文章于 2024-04-04 16:45:46 发布

m0_51022702

最新推荐文章于 2024-04-04 16:45:46 发布

阅读量1.4k

点赞数

分类专栏：大数据文章标签： flume maven interceptor

本文链接：https://blog.csdn.net/m0_51022702/article/details/121348641

版权

大数据专栏收录该内容

8 篇文章 0 订阅

订阅专栏

本文介绍了如何创建一个Maven工程，并实现一个自定义的Flume拦截器。拦截器根据接收到的数据首字节判断是字母还是数字，从而决定将其发送到不同的目的地。之后将拦截器类打包成jar，放入Flume的lib目录，并配置Flume配置文件，实现了数据从hadoop101的8888端口监听，按首字节类型分别发送到hadoop104和hadoop103。最后，提供了启动Flume agent的命令。

摘要由CSDN通过智能技术生成

1.创建maven工程自定义拦截器类

1.导入依赖

<dependencies>
        <dependency>
            <groupId>org.apache.flume</groupId>
            <artifactId>flume-ng-core</artifactId>
            <version>1.9.0</version>
        </dependency>

    </dependencies>

package myInterceptor;

import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.interceptor.Interceptor;

import java.util.List;

/**
 * @author jsh
 * @Description:
 * @create 2021-11-14-21:40
 */
public class CustomInterceptor implements Interceptor {
    @Override
    public void initialize() {

    }
    // 单个event拦截规则
    @Override
    public Event intercept(Event event) {
        byte[] body = event.getBody();
        if (body[0] < 'z' && body[0] > 'a'){
            event.getHeaders().put("type","letter");
        }else if (body[0] > '0' && body[0] < '9'){
            event.getHeaders().put("type","number");
        }
        return event;
    }
    // 批量处理event 其他两个方法可以不用实现
    @Override
    public List<Event> intercept(List<Event> list) {
        for (Event event : list) {
            intercept(event);
        }
        return list;
    }
    public static class Builder implements Interceptor.Builder{

        @Override
        public Interceptor build() {
            return new CustomInterceptor();
        }

        @Override
        public void configure(Context context) {

        }
    }

    @Override
    public void close() {

    }
}

2.将该类打成jar包

3.将jar包放在flume的lib目录下

4.书写自定义配置文件

H.案例需求：多路
  hadoop101 的flume1 监控 8888端口的数据 并将已字母开头的发给Hadoop104，数字开头的发送给Hadoop103
  flume2，flume3打印到控制台
        flume   sources   channels  sinks
  分析： flume1  netcat    memory    avro
        flume2   avro     memory    logger
        flume3   avro     memory    logger
  flume3.conf        
  #name
  a3.sources = r1
  a3.channels = c1
  a3.sinks = k1

  #sources
  a3.sources.r1.type = avro
  a3.sources.r1.bind = hadoop104
  a3.sources.r1.port = 8888

  #channels
  a3.channels.c1.type =memory
  a3.channels.c1.capacity = 10000
  a3.channels.c1.transactionCapacity = 100

  #sinks
  a3.sinks.k1.type = logger

  #bind
  a3.sources.r1.channels = c1
  a3.sinks.k1.channel = c1

  flume2.conf
  #name
  a2.sources = r1
  a2.channels = c1
  a2.sinks = k1

  #sources
  a2.sources.r1.type = avro
  a2.sources.r1.bind = hadoop103
  a2.sources.r1.port = 7777

  #channels
  a2.channels.c1.type =memory
  a2.channels.c1.capacity = 10000
  a2.channels.c1.transactionCapacity = 100

  #sinks
  a2.sinks.k1.type = logger

  #bind
  a2.sources.r1.channels = c1
  a2.sinks.k1.channel = c1

  flume1.conf
  #name
  a1.sources = r1
  a1.channels = c1 c2
  a1.sinks = k1 k2

  #sources
  a1.sources.r1.type = netcat
  a1.sources.r1.bind = localhost
  a1.sources.r1.port = 6666

  # Interceptor
  a1.sources.r1.interceptors = i1
  # myInterceptor.CustomInterceptor 为类的引用路径 Builder 自定义拦截类的内部类
  a1.sources.r1.interceptors.i1.type = myInterceptor.CustomInterceptor$Builder

  #channel selector
  a1.sources.r1.selector.type = multiplexing
  a1.sources.r1.selector.header = type
  a1.sources.r1.selector.mapping.letter = c1
  a1.sources.r1.selector.mapping.number = c2

  #channels
  a1.channels.c1.type = memory
  a1.channels.c1.capacity = 10000
  a1.channels.c1.transactionCapacity = 100 

  a1.channels.c2.type = memory
  a1.channels.c2.capacity = 10000
  a1.channels.c2.transactionCapacity = 100

  #sink
  a1.sinks.k1.type = avro
  a1.sinks.k1.hostname = hadoop104
  a1.sinks.k1.port = 8888

  a1.sinks.k2.type = avro
  a1.sinks.k2.hostname = hadoop103
  a1.sinks.k2.port = 7777

  #bind
  a1.sources.r1.channels = c1 c2
  a1.sinks.k1.channel = c1
  a1.sinks.k2.channel = c2      

  启动3：
  flume-ng agent -c $FLUME_HOME/conf -f $FLUME_HOME/jobs/multiplexing/flume3.conf -n a3 -Dflume.root.logger=INFO,console
  启动2：
  flume-ng agent -c $FLUME_HOME/conf -f $FLUME_HOME/jobs/multiplexing/flume2.conf -n a2 -Dflume.root.logger=INFO,console
  启动1：
  flume-ng agent -c $FLUME_HOME/conf -f $FLUME_HOME/jobs/multiplexing/flume1.conf -n a1 -Dflume.root.logger=INFO,console