Flume(二十)自定义Sink

要在Flume中实现自定义Sink，你需要继承AbstractSinkRunner类并实现Configurable和Sink接口,自定义Sink需要考虑事务问题

Flume使用两个独立的事务分别是从Source到Channel的Put事务和从Channel到Sink之间的Take事务。
Put事务
当Source接收到数据时，它会启动一个事务，Source进行doPut将数据写到临时的缓冲区PutList中；PutList会推送数据给Channel,如果Channel中有足够的位置，则数据推送成功（doCommit），如果Channel中没有位置，则推送失败，进行回滚（doRoollback），数据不会写入Channel。
Take事务
当Sink准备将数据从Channel中取出时，也会启动一个事务，Channel进行doTake操作将数据写到临时缓冲区TakeList中；将TakeList中的数据通过Sink批量写往目的地；如果写成功了，则执行doCommit操作；如果写失败了，则执行doRollback操作。

1.添加pom依赖

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>flume</artifactId>
    <version>1.0-SNAPSHOT</version>


    <dependencies>
        <dependency>
            <groupId>org.apache.flume</groupId>
            <artifactId>flume-ng-core</artifactId>
            <version>1.11.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flume</groupId>
            <artifactId>flume-ng-sdk</artifactId>
            <version>1.11.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flume</groupId>
            <artifactId>flume-ng-configuration</artifactId>
            <version>1.11.0</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

2.编写自定义Sink的java代码

a1.sources=r1
a1.channels=c1
a1.sinks=k1
a1.sources.r1.type=netcat
a1.sources.r1.bind = m1
a1.sources.r1.port= 6666
#自定义sink类
a1.sinks.k1.type = com.st.sink.CustomSink
#自定义上传参数，类路径
a1.sinks.k1.path = /
a1.sinks.k1.sink.rollInterval = 3600
a1.channels.c1.type=memory
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1 

2.1大包成jar包放到flume的lib目录下

3.编写配置文件，配置生成文件的路径为/

package com.st.sink;

import org.apache.flume.*;
import org.apache.flume.conf.Configurable;
import org.apache.flume.sink.AbstractSink;

import java.io.FileNotFoundException;
import java.io.PrintStream;
import java.util.Map;

//模拟 file roll sink将数据写到磁盘上
public class CustomSink extends AbstractSink implements Sink, Configurable {
    private String path;
    private     PrintStream ps;
    @Override
    public void configure(Context context) {
        path = context.getString("path");
        if(path==null) try {
            throw new IllegalAccessException("必须输入");
        } catch (IllegalAccessException e) {
            e.printStackTrace();
        }
    }
    @Override
    public Status process()  {
        //获取sink对应的channel
        Channel c = this.getChannel();
        Transaction t = c.getTransaction();
        t.begin();
       //获取数据
        Event e;
        try{
            while ((e=c.take())!=null){
                    Map<String,String> map =    e.getHeaders();
                    ps.println("headers");
                    for (Map.Entry<String,String>  h:map.entrySet()){
                        ps.println("\t"+h.getKey()+":"+h.getValue());
                    }
                   byte[] body =  e.getBody();
                ps.println("body");
                ps.println(new String(body));
            }
            t.commit();
            return Status.READY;
        }catch(Exception ex){
            t.rollback();
            return Status.BACKOFF;
        }finally {
            t.close();
        }

    }

    @Override
    public synchronized void start() {

        try {
             ps = new PrintStream(path+"/"+System.currentTimeMillis());
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        }

    }

    @Override
    public synchronized void stop() {
        if(ps!=null){
            ps.close();
        }

    }
}

4.启动flume

flume-ng agent -c $FLUME_HOME/conf -n a1  -f $FLUME_HOME/selfsink.conf  -Dflume.root.logger=INFO,console

5.使用nc m1 6666 发送数据

6.去根目录下查看文件