Flume编程-自定义Agent

最新推荐文章于 2022-01-24 23:43:05 发布
最新推荐文章于 2022-01-24 23:43:05 发布
阅读量239 收藏
点赞数
分类专栏： # Flume
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_42455459/article/details/111569455
版权

1.自定义Interceptor

可以自定义Interceptor，对agent接收的event header中添加KV值。在后面的Selector中根据添加的KV值来决定将event分发到channel的分发规则。

例如下面代码定义了一个interceptor，当接收event的body中含有Hello字符串时，向header添加type:hello键值对；否则添加type:nonhello键值对。若selector接收的event header type是hello，则将其分发到channel c1；否则分发到channel c2。自定义interceptor使用方法Event intercept(Event event)对source进行处理，返回处理后的Event。

package com.fyk.flume.Interceptor;

import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.interceptor.Interceptor;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;

public class CustomInterceptor implements Interceptor {
    private List<Event> addHeaderEvents;

    public void initialize() {
        addHeaderEvents = new ArrayList<Event>();
    }

    public Event intercept(Event event) {
        Map<String, String> headers = event.getHeaders();
        String body = new String(event.getBody());

        if(body.contains("Hello")){
            headers.put("type", "hello");
        }
        else{
            headers.put("type", "nonhello");
        }

        return event;
    }

    public List<Event> intercept(List<Event> events) {

        addHeaderEvents.clear();

        for(Event event : events){
            addHeaderEvents.add(intercept(event));
        }

        return addHeaderEvents;
    }

    public void close() {

    }

    public static class Builder implements Interceptor.Builder {


        public Interceptor build() {
            return new CustomInterceptor();
        }

        public void configure(Context context) {

        }
    }
}

测试结果如下：

###启动
[root@master flume-1.6.0]# ./bin/flume-ng agent --conf conf --conf-file ./conf/flume-custom/flume-custominteceptor-master.conf --name a1 -Dflume.root.logger=INFO,console
[root@slave1 flume-1.6.0]# ./bin/flume-ng agent --conf conf --conf-file ./conf/flume-custom/flume-custominterceptor-slave1.conf --name a1 -Dflume.root.logger=INFO,console
[root@slave2 flume-1.6.0]# ./bin/flume-ng agent --conf conf --conf-file ./conf/flume-custom/flume-custominteceptor-slave2.conf --name a1 -Dflume.root.logger=INFO,console


###测试结果
[root@slave1 ~]# telnet master 50000
Trying 192.168.56.100...
Connected to master.
Escape character is '^]'.
abcdef
OK
hello ghi
OK
Hello world
OK
Hello python
OK
Hello AI
OK
hello Bigdata
OK

#slave1
2020-12-22 07:21:37,117 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{type=hello} body: 48 65 6C 6C 6F 20 77 6F 72 6C 64 0D             Hello world. }
2020-12-22 07:22:07,125 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{type=hello} body: 48 65 6C 6C 6F 20 70 79 74 68 6F 6E 0D          Hello python. }
2020-12-22 07:22:07,125 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{type=hello} body: 48 65 6C 6C 6F 20 41 49 0D                      Hello AI. }

#slave2
2020-12-22 07:20:01,966 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{type=nonhello} body: 61 62 63 64 65 66 0D                            abcdef. }
2020-12-22 07:20:26,254 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{type=nonhello} body: 68 65 6C 6C 6F 20 67 68 69 0D                   hello ghi. }
2020-12-22 07:22:11,298 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{type=nonhello} body: 68 65 6C 6C 6F 20 42 69 67 64 61 74 61 0D       hello Bigdata. }




###使用的配置文件
#master
[root@master flume-1.6.0]# cat ./conf/flume-custom/flume-custominteceptor-master.conf
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1 c2

a1.sources.r1.type = netcat
a1.sources.r1.bind = master
a1.sources.r1.port = 50000

#Interceptor
a1.sources.r1.interceptors = i1
#自定义Interceptor类
a1.sources.r1.interceptors.i1.type = com.fyk.flume.Interceptor.CustomInterceptor$Builder

#channel selector， 根据自定义Interceptor值来分配channel
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = type
a1.sources.r1.selector.mapping.hello = c1
a1.sources.r1.selector.mapping.nonhello = c2

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

a1.channels.c2.type = memory
a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 100

a1.sinks.k1.type = avro
a1.sinks.k1.hostname = slave1
a1.sinks.k1.port = 50000

a1.sinks.k2.type = avro
a1.sinks.k2.hostname = slave2
a1.sinks.k2.port = 50000

a1.sources.r1.channels = c1 c2
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2

#slave1
[root@slave1 flume-1.6.0]# cat ./conf/flume-custom/flume-custominterceptor-slave1.conf
a1.sources = r1
a1.sinks = k1
a1.channels = c1

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

a1.sources.r1.type = avro
a1.sources.r1.bind = slave1
a1.sources.r1.port = 50000

a1.sinks.k1.type = logger

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1


#slave2
[root@slave2 flume-1.6.0]# cat ./conf/flume-custom/flume-custominteceptor-slave2.conf
a1.sources = r1
a1.sinks = k1
a1.channels = c1

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

a1.sources.r1.type = avro
a1.sources.r1.bind = slave2
a1.sources.r1.port = 50000

a1.sinks.k1.type = logger

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

1.2 自定义Source

以下代码自定义一个source，封装数据为event并由getChannelProcessor().processEvent(event);来处理。processEvent方法中使用事务处理机制来提交event给channel。

package com.fyk.flume.Source;

import org.apache.flume.Context;
import org.apache.flume.EventDeliveryException;
import org.apache.flume.PollableSource;
import org.apache.flume.Source;
import org.apache.flume.channel.ChannelProcessor;
import org.apache.flume.conf.Configurable;
import org.apache.flume.event.SimpleEvent;
import org.apache.flume.lifecycle.LifecycleState;
import org.apache.flume.source.AbstractSource;

public class CustomSource extends AbstractSource implements Configurable, PollableSource {

    //global variable
    private String prefix;
    private String suffix;

    public void configure(Context context) {
        prefix = context.getString("prefix");
        suffix = context.getString("suffix", "Flume");
    }

    /*
    *1. receive out data
    * 2. make event
    * 3. send event to channel
     */
    public Status process() throws EventDeliveryException {
        Status status = null;
        try{
            for(int i = 0; i < 5; i++)
            {
                SimpleEvent event = new SimpleEvent();
                event.setBody((prefix + "--"+ i + suffix).getBytes());
                getChannelProcessor().processEvent(event);
                status = Status.READY;
            }
        }
        catch (Exception e)
        {
            e.printStackTrace();
            status = Status.BACKOFF;
        }

        try{
            Thread.sleep(2000);
        }catch(InterruptedException e)
        {
            e.printStackTrace();
        }
        return status;
    }

    public long getBackOffSleepIncrement() {
        return 0;
    }

    public long getMaxBackOffSleepInterval() {
        return 0;
    }


}

##启动
[root@master flume-1.6.0]# ./bin/flume-ng agent --conf conf --conf-file ./conf/flume-custom/flume-customsource-master.conf --name a1 -Dflume.root.logger=INFO,console


##配置文件
[root@master flume-1.6.0]# cat ./conf/flume-custom/flume-customsource-master.conf
a1.sources = r1
a1.sinks = k1
a1.channels = c1

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

a1.sources.r1.type = com.fyk.flume.Source.CustomSource
a1.sources.r1.prefix = Bigdata
a1.sources.r1.suffix = spark

a1.sinks.k1.type = logger

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1



##测试结果
2020-12-22 23:54:01,951 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 42 69 67 64 61 74 61 2D 2D 30 73 70 61 72 6B    Bigdata--0spark }
2020-12-22 23:54:01,951 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 42 69 67 64 61 74 61 2D 2D 31 73 70 61 72 6B    Bigdata--1spark }
2020-12-22 23:54:01,952 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 42 69 67 64 61 74 61 2D 2D 32 73 70 61 72 6B    Bigdata--2spark }
2020-12-22 23:54:01,952 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 42 69 67 64 61 74 61 2D 2D 33 73 70 61 72 6B    Bigdata--3spark }
2020-12-22 23:54:01,952 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 42 69 67 64 61 74 61 2D 2D 34 73 70 61 72 6B    Bigdata--4spark }
2020-12-22 23:54:03,950 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 42 69 67 64 61 74 61 2D 2D 30 73 70 61 72 6B    Bigdata--0spark }
2020-12-22 23:54:03,951 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 42 69 67 64 61 74 61 2D 2D 31 73 70 61 72 6B    Bigdata--1spark }
2020-12-22 23:54:03,951 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 42 69 67 64 61 74 61 2D 2D 32 73 70 61 72 6B    Bigdata--2spark }
2020-12-22 23:54:03,952 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 42 69 67 64 61 74 61 2D 2D 33 73 70 61 72 6B    Bigdata--3spark }
2020-12-22 23:54:03,952 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 42 69 67 64 61 74 61 2D 2D 34 73 70 61 72 6B    Bigdata--4spark }

1.3 自定义sink

以下定义一个sink，使用事务机制去channel pull event。

package com.fyk.flume.Sink;

import org.apache.flume.*;
import org.apache.flume.conf.Configurable;
import org.apache.flume.sink.AbstractSink;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class CustomSink extends AbstractSink implements Configurable {
    private Logger logger = LoggerFactory.getLogger(CustomSink.class);
    private String prefix;
    private String suffix;

    public Status process() throws EventDeliveryException {
        Status status = null;

        Channel channel = getChannel();

        Transaction txn = channel.getTransaction();

        txn.begin();
        try{
            Event event = channel.take();
            if(event != null) {
                String body = new String(event.getBody());
                logger.info(prefix + body + suffix);
            }

            txn.commit();
            status = Status.READY;
        }
        catch(Throwable e)
        {
            txn.rollback();
            status = Status.BACKOFF;
            if(e instanceof Exception){
                throw (Error)e;
            }
        }
        finally {
            txn.close();
        }

        return status;
    }

    public void configure(Context context) {
        prefix = context.getString("prefix");
        suffix = context.getString("suffix", "BigData");
    }
}

测试结果如下：

#启动
[root@master flume-1.6.0]# ./bin/flume-ng agent --conf conf --conf-file ./conf/flume-custom/flume-customsink-master.conf --name a1 -Dflume.root.logger=INFO,console

##配置文件
a1.sources = r1
a1.sinks = k1
a1.channels = c1

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

a1.sources.r1.type = netcat
a1.sources.r1.bind = master
a1.sources.r1.port = 50000

a1.sinks.k1.type = com.fyk.flume.Sink.CustomSink
a1.sinks.k1.prefix = sleep--
a1.sinks.k1.suffix = --boy

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

##测试结果
#slave1
[root@slave1 ~]# telnet master 50000
Trying 192.168.56.100...
Connected to master.
Escape character is '^]'.
abc def ghi
OK
Hello Bigdata
OK



#master
--boy12-23 00:03:51,530 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - com.fyk.flume.Sink.CustomSink.process(CustomSink.java:26)] sleep--abc def ghi
--boy12-23 00:05:12,854 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - com.fyk.flume.Sink.CustomSink.process(CustomSink.java:26)] sleep--Hello Bigdata

 

fyk_lovelife
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
使用flume问题总结2——flume event的简单理解
anjing900812的博客
10-09 6536
flume event的简单理解
flume学习（七）、（八）：如何使用event header中的key值以及自定义source
wang_zhenwei的博客
09-30 8025
1.如何使用event header中的key值？ 2.如何部署扩展自定义的spooling directory source？ 前面我们已经说到我们在header中添加了一个key为：flume.client.log4j.logger.source  ，然后有两个应用程序，一个设置为app1,一个设置为app2。 现在有这么一个需求，要将app1的日志输出到
Flume 架构
MaggieCN0622的博客
09-03 294
Flume 架构 Flume 架构图 上图是Flume官网最基础的数据流图。后面还会介绍多路复用、拦截等不同配置。 Flume Event 定义 一行文本内容被反序列化成一个event。 event的最大定义为2048字节，超过会被切割放到下一个event中。 event 包含两个 header、body header是一个map，body是一个字节数组 body是真正传输的数据，header传输的数据不会被sink出去。 源码 Event.java (Interface) package or
Flume基础概述
G_A34的博客
10-29 342
Flume基础概述 1、什么是flume flume是由cloudera软件公司产出的可分布式日志收集系统，后与2009年被捐赠了apache软件基金会，为hadoop相关组件之一。尤其近几年随着flume的不断被完善以及升级版本的逐一推出，特别是flume-ng;同时flume内部的各种组件不断丰富，用户在开发的过程中使用的便利性得到很大的改善，现已成为apache top项目之一。 Flume 是一个具有高可用，分布式，配置工具，其设计的原理也是基于将数据流，如日志数据从各种网站服务器上汇集起来存
Flume的基础架构
gddemon的博客
10-11 251
每日积累学习 Flume的基础架构 Source 负责接收数据到flume Agent的组件；可以处理各种类型、各种格式的日志数据， Channel Sink
Flume进阶-自定义拦截器jar包
最新发布
04-28
Flume进阶-自定义拦截器jar包
flume-ng-sql-source-1.5.2
10-12
flume-ng-sql-source-1.5.2源码
Flume-ng-1.6.0-cdh.zip
04-08
Flume-ng-1.6.0-cdh.zip 内压缩了 3 个项目，分别为：flume-ng-1.6.0-cdh5.5.0.tar.gz、flume-ng-1.6.0-cdh5.7.0.tar.gz 和 flume-ng-1.6.0-cdh5.10.1.tar.gz，选择你需要的版本。
apache-flume-1.5.0-cdh5.3.6-bin.zip
03-15
flume-1.5.0-cdh5.3.6。 大数据日志收集工具 flume-1.5.0-cdh5.3.6。 大数据日志收集工具flume-1.5.0-cdh5.3.6。 大数据日志收集工具flume-1.5.0-cdh5.3.6。 大数据日志收集工具flume-1.5.0-cdh5.3.6。 大数据日志...
flumeng-kafka-plugin:flumeng-kafka-plugin
05-12
flumeng-kafka-plugin 技术指标水槽1.4 Kafka 0.8.0 Beta
Flume--Event定义及源码解析
weixin_42946624的博客
10-18 3457
Flume基本构成与Event结构 1. Event定义 一行文本内容被反序列化成一个event。event的最大定义为2048字节，超过会被切割放到下一个event中。 public interface Event { /** * Returns a map of name-value pairs describing the data stored in the body. ...
Flume
あずにゃん梓喵的博客
06-18 420
日萌社 人工智能AI：Keras PyTorch MXNet TensorFlow PaddlePaddle 深度学习实战（不定时更新） 4.2 Flume 学习目标 目标 了解Flume结构 掌握Flume的安装 知道Flume的参数配置、插件以及作用 了解Flume的负载均衡 应用 应用Flume监听端口数据并输出到控制台 实现Flume采集文件/目录到HDFS 4.2.1 Flume概述 Flume是由cloudera软件公司产出的可分布式日志收集系统，后.
Flume Agent配置
Maqiuqiu520的博客
11-08 2090
Flume Agent配置 Flume介绍 Flume的Agent主要是由三个重要的组件组成：分别为Source、Channel、Sink。 （1）Source：完成对日志数据的收集，分成transtion和event导入到Channel之中。 （2）Channel：主要提供一个队列的功能，对Source提供的数据进行简单的缓存。 （3）Sink：取出Channel中的数据，相应的存储到文件系统，...
flume —— NullPointerException: Expected timestamp in the Flume event headers, but it was null
idream
03-25 872
如题错误： 可能大多人送所这个问题都会告诉你： 修改hdfs-sink： a1.sinks.k1.hdfs.useLocalTimeStamp = true 的确，这么做的确也可以解决上述问题，但你确定这是你想要的结果么？ 解释： 首先看错，很明显，event的header中获取timestamp属性为空，就是你sink的时候需要使用timestamp的属性。 一般使用这个属性的场景应该都是 a1.sinks.k1.hdfs.path = hdfs://nameservice/*/
Flume基础介绍
yingzi的技术博客
01-24 574
文章目录1.Flume定义2.Flume基础架构2.1Agent2.2Source2.3Channel2.4Sink2.5Event 1.Flume定义 Flume 是 Cloudera 提供的一个高可用的，高可靠的，分布式的海量日志采集、聚合和传输的系统。Flume 基于流式架构，灵活简单。 2.Flume基础架构 2.1Agent Agent是一个JVM进程，它以事件的形式将数据从源头送至目的 Agent主要由三部分组成：Source,Channel,Sinke 2.2Source ​ Sourc
Flume笔记
编程笔记
11-02 151
一.环境搭建 二.概念
flume数据丢失与重复_Flume基础学习
weixin_39702335的博客
12-26 772
Flume是一款非常优秀的日志采集工具。支持多种形式的日志采集，作为apache的顶级开源项目，Flume再大数据方面具有广泛的应用首先需要在Flume的解压目录中conf文件夹中将flume-env.sh.templete更改未flume.env.sh并修改jdk的位置Source我们可以从Avro，NetCat。Http,TailDir。我们在Java开发中通常都是使用的log4j等日志工具进...
Expected timestamp in the Flume event headers, but it was null：搭建flume多路复用写数据到hdfs时所遇问题与解决
破晓的成长之路
09-19 1874
在进行本机实践时遇到的错误具体见这篇博客：flume复制和多路复用简单示例（监控日志文件对接hdfs、kafka、本地文件系统） 错误如下： 19/09/17 03:59:34 ERROR hdfs.HDFSEventSink: process failed java.lang.NullPointerException: Expected timestamp in the Flume event ...
第十章：Flume系统高级编程实战
10-24
1.Flume核心概念 2.Flume的分布式安装与配置 3.Flume的Agent应用开发 4.flume 拦截器：自定义拦截器，Regex拦截器等 5.flume自定义Source、Sink和Interceptor
[ys@hadoop102 flume]$ flume-ng agent --conf-file /path/to/file-flume-kafka.conf --name kafka-flume-agent -Dflume.root.logger=INFO,console bash: flume-ng: 未找到命令...
06-10
/home/ys/flume/bin/flume-ng agent --conf-file /path/to/file-flume-kafka.conf --name kafka-flume-agent -Dflume.root.logger=INFO,console ``` 请将路径/home/ys/flume/bin/替换为您自己的Flume安装路径。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值