custom.txt文件如下,存放在虚拟机/opt/datas文件夹中
1,张三,男,40
2,李三,女,4
3,张四,男,14
4,李三三,女,44
任务需求:需要把性别一列“男”转换为“1”,“女”转换为“2”,其他值转换为“0”。
创建maven项目,在pom.xml文件中导入flume依赖包
<!-- https://mvnrepository.com/artifact/org.apache.flume/flume-ng-core -->
<dependency>
<groupId>org.apache.flume</groupId>
<artifactId>flume-ng-core</artifactId>
<version>1.6.0</version>
</dependency>
创建拦截器类CustomInterceptor 实现Interceptor接口 (org.apache.flume.interceptor包中),实现intercept方法,自定义处理操作;在类中创建内部类Builder 实现接口Interceptor.Builder,在builder方法中创建返回CustomInterceptor 对象。
package com.flume.interceptor;
import org.apache.flume.interceptor.Interceptor
public class CustomInterceptor implements Interceptor {
@Override
public void initialize() {
}
//单个事件拦截
@Override
public Event intercept(Event event) {
byte[] body = event.getBody();//事件body
String line = new String(body);//1,张三,男,40
String[] sps = line.split(",");
switch (sps[2]){
case "男":sps[2]="1";
break;
case "女":sps[2]="2";
break;
default:sps[2]="0";
break;
}
//完整事件信息组合
String newStr=sps[0]+","+sps[1]+","+sps[2]+","+sps[3];
//事件body
event.setBody(newStr.getBytes());
return event;
}
//批量事件拦截
@Override
public List<Event> intercept(List<Event> list) {
for (Event event:list){
intercept(event);
}
return list;
}
@Override
public void close() {
}
//静态内部类
public static class Builder implements Interceptor.Builder{
@Override
public Interceptor build() {
return new CustomInterceptor();
}
@Override
public void configure(Context context) {
}
}
修改pom.xml文件中为如下所示,打胖包。“clean”->“package”,在target中找到myintc.jar复制到linux的flume安装目录的lib中。
<build>
<finalName>myintc</finalName>
<pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>com.flume.interceptor.App</mainClass><!--这里改成自己的主类位置-->
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
</build>
创建配置文件/opt/flumeconf/flume_interceptor.properties
a4.channels=c4
a4.sources=s4
a4.sinks=k4
a4.sources.s4.type=spooldir
a4.sources.s4.spoolDir=/opt/datas
a4.sources.s4.interceptors=myintec //过滤器名
a4.sources.s4.interceptors.myintec.type=com.flume.interceptor.CustomInterceptor$Builder //拦截器对象创建 类路径$内部类名
a4.channels.c4.type=memory
a4.sinks.k4.type=logger
a4.sinks.k4.channel=c4
a4.sources.s4.channels=c4
测试拦截器,信息以日志格式输出控制台
flume-ng agent -n a4 -c conf -f /opt/flumeconf/conf_0806_interceptor.properties -Dflume.root.logger=INFO,console
=====================================================================
20/08/06 14:24:45 INFO sink.LoggerSink: Event: { headers:{} body: 31 2C E5 BC A0 E4 B8 89 2C 31 2C 34 30 0D 1,......,1,40. }
20/08/06 14:24:45 INFO sink.LoggerSink: Event: { headers:{} body: 32 2C E6 9D 8E E4 B8 89 2C 32 2C 34 0D 2,......,2,4. }
20/08/06 14:24:45 INFO sink.LoggerSink: Event: { headers:{} body: 33 2C E5 BC A0 E5 9B 9B 2C 31 2C 31 34 0D 3,......,1,14. }
20/08/06 14:24:45 INFO sink.LoggerSink: Event: { headers:{} body: 34 2C E6 9D 8E E4 B8 89 E4 B8 89 2C 32 2C 34 34 4,.........,2,44 }