Java 代码实现自定义flume Interceptor

custom.txt文件如下，存放在虚拟机/opt/datas文件夹中

1,张三,男,40
2,李三,女,4
3,张四,男,14
4,李三三,女,44

任务需求：需要把性别一列“男”转换为“1”，“女”转换为“2”，其他值转换为“0”。

创建maven项目，在pom.xml文件中导入flume依赖包

<!-- https://mvnrepository.com/artifact/org.apache.flume/flume-ng-core -->
<dependency>
  <groupId>org.apache.flume</groupId>
  <artifactId>flume-ng-core</artifactId>
  <version>1.6.0</version>
</dependency>

创建拦截器类CustomInterceptor 实现Interceptor接口 (org.apache.flume.interceptor包中)，实现intercept方法，自定义处理操作；在类中创建内部类Builder 实现接口Interceptor.Builder，在builder方法中创建返回CustomInterceptor 对象。

package com.flume.interceptor;
import org.apache.flume.interceptor.Interceptor
public class CustomInterceptor implements Interceptor {
    @Override
    public void initialize() {

    }
    //单个事件拦截
    @Override
    public Event intercept(Event event) {
        byte[] body = event.getBody();//事件body
        String line = new String(body);//1，张三，男，40
        String[] sps = line.split(",");
        switch (sps[2]){
            case "男":sps[2]="1";
                break;
            case "女":sps[2]="2";
                break;
            default:sps[2]="0";
                break;
        }
        //完整事件信息组合
        String newStr=sps[0]+","+sps[1]+","+sps[2]+","+sps[3];
        //事件body
        event.setBody(newStr.getBytes());
        return event;
    }

    //批量事件拦截
    @Override
    public List<Event> intercept(List<Event> list) {
        for (Event event:list){
            intercept(event);
        }
        return list;
    }

    @Override
    public void close() {

    }
    //静态内部类
    public static class Builder implements Interceptor.Builder{

        @Override
        public Interceptor build() {
            return new CustomInterceptor();
        }

        @Override
        public void configure(Context context) {

        }
    }

修改pom.xml文件中为如下所示，打胖包。“clean”->“package”,在target中找到myintc.jar复制到linux的flume安装目录的lib中。

  <build>
    <finalName>myintc</finalName>
    <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
      <plugins>
        <plugin>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>2.3.2</version>
          <configuration>
            <source>1.8</source>
            <target>1.8</target>
          </configuration>
        </plugin>
        <plugin>
          <artifactId>maven-assembly-plugin</artifactId>
          <configuration>
            <descriptorRefs>
              <descriptorRef>jar-with-dependencies</descriptorRef>
            </descriptorRefs>
            <archive>
              <manifest>
                <mainClass>com.flume.interceptor.App</mainClass><!--这里改成自己的主类位置-->
              </manifest>
            </archive>
          </configuration>
          <executions>
            <execution>
              <id>make-assembly</id>
              <phase>package</phase>
              <goals>
                <goal>single</goal>
              </goals>
            </execution>
          </executions>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>

创建配置文件/opt/flumeconf/flume_interceptor.properties

a4.channels=c4
a4.sources=s4
a4.sinks=k4

a4.sources.s4.type=spooldir
a4.sources.s4.spoolDir=/opt/datas
a4.sources.s4.interceptors=myintec		//过滤器名
a4.sources.s4.interceptors.myintec.type=com.flume.interceptor.CustomInterceptor$Builder		//拦截器对象创建 类路径$内部类名

a4.channels.c4.type=memory

a4.sinks.k4.type=logger

a4.sinks.k4.channel=c4
a4.sources.s4.channels=c4

测试拦截器，信息以日志格式输出控制台

flume-ng agent -n a4 -c conf -f /opt/flumeconf/conf_0806_interceptor.properties -Dflume.root.logger=INFO,console
=====================================================================
20/08/06 14:24:45 INFO sink.LoggerSink: Event: { headers:{} body: 31 2C E5 BC A0 E4 B8 89 2C 31 2C 34 30 0D       1,......,1,40. }
20/08/06 14:24:45 INFO sink.LoggerSink: Event: { headers:{} body: 32 2C E6 9D 8E E4 B8 89 2C 32 2C 34 0D          2,......,2,4. }
20/08/06 14:24:45 INFO sink.LoggerSink: Event: { headers:{} body: 33 2C E5 BC A0 E5 9B 9B 2C 31 2C 31 34 0D       3,......,1,14. }
20/08/06 14:24:45 INFO sink.LoggerSink: Event: { headers:{} body: 34 2C E6 9D 8E E4 B8 89 E4 B8 89 2C 32 2C 34 34 4,.........,2,44 }