flume-interceptor
假设我们要拦截flume传输日志中包含“ pink ” 字符串的事件
每个event都是由一个header+body组成的
配置maven依赖
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>flume</groupId>
<artifactId>flume_intercepter</artifactId>
<version>1.0-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>6</source>
<target>6</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<!-- flume核心依赖 -->
<dependencies>
<dependency>
<groupId>org.apache.flume</groupId>
<artifactId>flume-ng-core</artifactId>
<version>1.9.0</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.62</version>
</dependency>
</dependencies>
</project>
public class PinkInterceptor implements Interceptor {
//声明一个存放事件的集合
private List<Event> addEvents;
public void initialize() {
events = new ArrayList<>();
}
//针对一个event做处理
public Event intercept(Event event) {
//获取event内容
byte[] body = event.getBody();
//不论传的是什么,flume都将body转化成byte数组,指定编码格式,防止出现乱码
String log = new String(body, Charset.forName("UTF-8"));
Map<String,String> headers = event.getHeaders();
//解析json格式数据
//JSONObject jsonObject = JSONObject.parseObject(log);
if(body.contains("pink")){
headers.put("type","Pink");
}else{
headers.put("type","noPink");
}
return event;
}
//event集合,批量事件
public List<Event> intercept(List<Event> list) {
//清空集合
addEvents.clear();
//遍历list给每一个事件添加头信息
for (Event event : list) {
addEvents(intercept(event));
}
return addEvents;
}
public void close() {
}
public static class Builder implements Interceptor.Builder{
@Override
public Interceptor build() {
return new PinkInterceptor();
}
@Override
public void configure(Context context) {
}
}
}
打包,将jar包上传到flume的lib目录下。
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = com.xpu.gmall.flume.interceptor.PinkInterceptor$Builder
此处type为全类名