flume自定义分流日志interceptor、source、sink，采用ganglia进行flume日志数据流监控

最新推荐文章于 2022-05-25 12:24:14 发布
最新推荐文章于 2022-05-25 12:24:14 发布
阅读量579 收藏 2
点赞数
分类专栏： hadoop hive flume 文章标签： hadoop flume 大数据 java sdl
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_37056888/article/details/106753148
版权
hadoop 同时被 3 个专栏收录
40 篇文章 0 订阅
订阅专栏
hive
15 篇文章 0 订阅
订阅专栏
flume
3 篇文章 0 订阅
订阅专栏

1.Interceptor分流日志

1.1idea开发interceptor分流日志

1
2
3

1.2 创建自定义interceptor.jar

package com.cevent.interceptor;/**
 * Created by Cevent on 2020/6/13.
 */

import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.interceptor.Interceptor;

import java.util.List;
import java.util.Map;

/**
 * @author cevent
 * @description 拦截器：文件分流
 * @date 2020/6/13 21:40
 */
public class FlumeInterceptor implements Interceptor{
    //1.初始化方法
    public void initialize() {

    }

    //2.单一事件:添加自定义header所需<key,value>对
    public Event intercept(Event event) {
        //2.1获取header和body
        Map<String,String> headers=event.getHeaders();
        byte[] body=event.getBody();

        //2.2根据body内容，添加自定义KEY,VALUE进入header，字符码判断是否是数字<='9',0-9之间
        if(body[0]<='9' && body[0]>='0'){
            headers.put("type","number");
        }else{
            headers.put("type","not_number");
        }

        return event;
    }

    //3.事件集合List
    public List<Event> intercept(List<Event> eventlist) {
        //遍历集合
        for(Event event:eventlist){
            intercept(event);
        }

        return eventlist;
    }

    //4.初始化开启资源，最后关闭
    public void close() {

    }

    //5.创建内部类，实现builder
    public static class InterceptorBuilderMe implements Interceptor.Builder {

        public Interceptor build() {
            return new FlumeInterceptor();
        }

        public void configure(Context context) {

        }
    }
}

打包

1.3导入jar

导入

2.自定义Source

2.1source源码

链接：http://flume.apache.org/releases/content/1.7.0/FlumeUserGuide.html#thrift-legacy-source

package com.cevent.source;/**
 * Created by Cevent on 2020/6/14.
 */

import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.EventDeliveryException;
import org.apache.flume.PollableSource;
import org.apache.flume.channel.ChannelProcessor;
import org.apache.flume.conf.Configurable;
import org.apache.flume.event.SimpleEvent;
import org.apache.flume.source.AbstractSource;

import java.util.HashMap;

/**
 * @author cevent
 * @description
 * @date 2020/6/14 10:32
 */
public class CeventSource extends AbstractSource implements Configurable,PollableSource{

    //使用configure配置自定义source元
    private String prefix;
    private long interval;


    //1.source：获取上游数据，包装成event，发送给channel
    public Status process() throws EventDeliveryException {
        //1.1模拟上游：每隔一段时间向channel发送5条虚拟数据
        //投递状态
        Status status=null;
        //获取channel
        ChannelProcessor channelProcessor=this.getChannelProcessor();

        //1.2数据处理的方法
        try {
            for(int i=1;i<=5;i++){
                //1.3将数据包装为event
                Event event=new SimpleEvent();
                event.setHeaders(new HashMap<String, String>());
                //event.setBody(("LOG"+i).getBytes());

                event.setBody((prefix+i).getBytes());

                //1.4获取channel进程
                channelProcessor.processEvent(event);

                //Thread.sleep(1000);
                Thread.sleep(interval);
            }

            //1.5输出处理成功
            status=Status.READY;
        }
        catch (InterruptedException e){
            //1.6数据处理失败
            status=Status.BACKOFF;
        }

        return status;
    }

    //2.source投送到channel，channel反馈已满载（投递失败），需要等待挂起，设置2s
    public long getBackOffSleepIncrement() {
        return 2000;
    }

    //3.投递失败，最大等待时间
    public long getMaxBackOffSleepInterval() {
        return 10000;
    }

    //4.根据配置文件，配置自定义source
    public void configure(Context context) {
        //4.1自定义source前缀，不用process的"LOG"，如果没有获取到key，则使用默认value"LOG"
        prefix=context.getString("defined_prefix","LOG");
        interval=context.getLong("defined_interval",1000L);
    }
}

导入

2.2配置、启动source.conf




 
  
  [cevent@hadoop207 apache-flume-1.7.0]$ ll
  总用量 176
  drwxr-xr-x.  2 cevent cevent  4096 6月  11 13:35 bin
  -rw-r--r--.  1 cevent cevent 77387 10月 11 2016 CHANGELOG
  drwxr-xr-x.  2 cevent cevent  4096 6月  12 12:11 conf
  -rw-r--r--.  1 cevent cevent  6172 9月  26 2016 DEVNOTES
  -rw-r--r--.  1 cevent cevent  2873 9月  26 2016 doap_Flume.rdf
  drwxr-xr-x. 10 cevent cevent  4096 10月 13 2016 docs
  drwxrwxr-x.  2 cevent cevent  4096 6月  12 16:43 files
  drwxrwxr-x.  6 cevent cevent  4096 6月  13 22:21 job
  drwxrwxr-x.  2 cevent cevent  4096 6月  14 13:26 lib
  -rw-r--r--.  1 cevent cevent 27625 10月 13 2016 LICENSE
  drwxrwxr-x.  2 cevent cevent  4096 6月  12 11:48 loggers
  drwxrwxr-x.  2 cevent cevent  4096 6月  11 17:05 logs
  -rw-r--r--.  1 cevent cevent   249 9月  26 2016 NOTICE
  -rw-r--r--.  1 cevent cevent  2520 9月  26 2016 README.md
  -rw-r--r--.  1 cevent cevent  1585 10月 11 2016 RELEASE-NOTES
  -rw-rw-r--.  1 cevent cevent   177 6月  12 16:57 tail_dir.json
  drwxrwxr-x.  2 cevent cevent  4096 6月  11 13:35 tools
  -rw-rw-r--.  1 cevent cevent    16 6月  12 16:45 tutu.txt
  drwxrwxr-x.  3 cevent cevent  4096 6月  12 14:23 upload
  [cevent@hadoop207 apache-flume-1.7.0]$ cd
  job/
  [cevent@hadoop207 job]$ ll
  总用量 32
  -rw-rw-r--. 1 cevent cevent 1542 6月  12 14:22 flume-dir-hdfs.conf
  -rw-rw-r--. 1 cevent cevent 1641 6月  12 13:36 flume-file-hdfs.conf
  -rw-rw-r--. 1 cevent cevent  495 6月  11 17:02 flume-netcat-logger.conf
  -rw-rw-r--. 1 cevent cevent 1522 6月  12 16:40 flume-taildir-hdfs.conf
  drwxrwxr-x. 2 cevent cevent 4096 6月  13 13:28 group1
  drwxrwxr-x. 2 cevent cevent 4096 6月  13 14:56 group2
  drwxrwxr-x. 2 cevent cevent 4096 6月  13 20:59 group3
  drwxrwxr-x. 2 cevent cevent 4096 6月  13 22:38 interceptor
  [cevent@hadoop207 job]$ mkdir source
  [cevent@hadoop207 job]$ vim source/flume-source-1.conf
  ### 自定义源数据source
  # Name the components on this agent
  a1.sources = r1
  a1.sinks = k1
  a1.channels = c1
   
  # Describe/configure the source
  a1.sources.r1.type = com.cevent.source.CeventSource
  ##jar自定义属性
  a1.sources.r1.defined_interval = 1000
  a1.sources.r1.defined_prefix = ceventLOG
   
  # Describe the sink
  a1.sinks.k1.type = logger
   
  # Use a channel which buffers events in memory
  a1.channels.c1.type = memory
  a1.channels.c1.capacity = 1000
  a1.channels.c1.transactionCapacity = 100
   
  # Bind the source and sink to the channel
  a1.sources.r1.channels = c1
  a1.sinks.k1.channel = c1
   
  ~
  "source/flume-source-1.conf" [新] 23L, 589C 已写入                       
  [cevent@hadoop207 job]$ cd ../..
  [cevent@hadoop207 module]$ cd
  apache-flume-1.7.0/
  [cevent@hadoop207 apache-flume-1.7.0]$ ll
  总用量 176
  drwxr-xr-x.  2 cevent cevent  4096 6月  11 13:35 bin
  -rw-r--r--.  1 cevent cevent 77387 10月 11 2016 CHANGELOG
  drwxr-xr-x.  2 cevent cevent  4096 6月  12 12:11 conf
  -rw-r--r--.  1 cevent cevent  6172 9月  26 2016 DEVNOTES
  -rw-r--r--.  1 cevent cevent  2873 9月  26 2016 doap_Flume.rdf
  drwxr-xr-x. 10 cevent cevent  4096 10月 13 2016 docs
  drwxrwxr-x.  2 cevent cevent  4096 6月  12 16:43 files
  drwxrwxr-x.  7 cevent cevent  4096 6月  14 13:30 job
  drwxrwxr-x.  2 cevent cevent  4096 6月  14 13:26 lib
  -rw-r--r--.  1 cevent cevent 27625 10月 13 2016 LICENSE
  drwxrwxr-x.  2 cevent cevent  4096 6月  12 11:48 loggers
  drwxrwxr-x.  2 cevent cevent  4096 6月  11 17:05 logs
  -rw-r--r--.  1 cevent cevent   249 9月  26 2016 NOTICE
  -rw-r--r--.  1 cevent cevent  2520 9月  26 2016 README.md
  -rw-r--r--.  1 cevent cevent  1585 10月 11 2016 RELEASE-NOTES
  -rw-rw-r--.  1 cevent cevent   177 6月  12 16:57 tail_dir.json
  drwxrwxr-x.  2 cevent cevent  4096 6月  11 13:35 tools
  -rw-rw-r--.  1 cevent cevent    16 6月  12 16:45 tutu.txt
  drwxrwxr-x.  3 cevent cevent  4096 6月  12 14:23 upload
  [cevent@hadoop207 apache-flume-1.7.0]$ 启动并控制台打印数据
  bin/flume-ng agent
  -n a1 -c conf/ -f job/source/flume-source-1.conf
  -Dflume.root.logger=INFO,console
  Info: Sourcing environment configuration script
  /opt/module/apache-flume-1.7.0/conf/flume-env.sh
  Info: Including Hadoop libraries found via (/opt/module/hadoop-2.7.2/bin/hadoop)
  for HDFS access
  Info: Including Hive libraries found via (/opt/module/hive-1.2.1) for
  Hive access
  + exec /opt/module/jdk1.7.0_79/bin/java -Xmx20m
  -Dflume.root.logger=INFO,console -cp
  '/opt/module/apache-flume-1.7.0/conf:/opt/module/apache-flume-1.7.0/lib/*:/opt/module/hadoop-2.7.2/etc/hadoop:/opt/module/hadoop-2.7.2/share/hadoop/common/lib/*:/opt/module/hadoop-2.7.2/share/hadoop/common/*:/opt/module/hadoop-2.7.2/share/hadoop/hdfs:/opt/module/hadoop-2.7.2/share/hadoop/hdfs/lib/*:/opt/module/hadoop-2.7.2/share/hadoop/hdfs/*:/opt/module/hadoop-2.7.2/share/hadoop/yarn/lib/*:/opt/module/hadoop-2.7.2/share/hadoop/yarn/*:/opt/module/hadoop-2.7.2/share/hadoop/mapreduce/lib/*:/opt/module/hadoop-2.7.2/share/hadoop/mapreduce/*:/opt/module/hadoop-2.7.2/contrib/capacity-scheduler/*.jar:/opt/module/hive-1.2.1/lib/*'
  -Djava.library.path=:/opt/module/hadoop-2.7.2/lib/native
  org.apache.flume.node.Application -n a1 -f job/source/flume-source-1.conf
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in
  [jar:file:/opt/module/apache-flume-1.7.0/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in
  [jar:file:/opt/module/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
  explanation.
  2020-06-14 13:32:12,811 (lifecycleSupervisor-1-0) [INFO -
  org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)]
  Configuration provider starting
  2020-06-14 13:32:12,816 (conf-file-poller-0) [INFO -
  org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:134)]
  Reloading configuration file:job/source/flume-source-1.conf
  2020-06-14 13:32:12,824 (conf-file-poller-0) [INFO -
  org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)]
  Added sinks: k1 Agent: a1
  2020-06-14 13:32:12,824 (conf-file-poller-0) [INFO -
  org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)]
  Processing:k1
  2020-06-14 13:32:12,824 (conf-file-poller-0) [INFO -
  org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)]
  Processing:k1
  2020-06-14 13:32:12,841 (conf-file-poller-0) [INFO -
  org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)]
  Post-validation flume configuration contains configuration for agents: [a1]
  2020-06-14 13:32:12,841 (conf-file-poller-0) [INFO -
  org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:147)]
  Creating channels
  2020-06-14 13:32:12,856 (conf-file-poller-0) [INFO -
  org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)]
  Creating instance of channel c1 type memory
  2020-06-14 13:32:12,860 (conf-file-poller-0) [INFO -
  org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:201)]
  Created channel c1
  2020-06-14 13:32:12,861 (conf-file-poller-0) [INFO -
  org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)]
  Creating instance of source r1, type com.cevent.source.CeventSource
  2020-06-14 13:32:12,870 (conf-file-poller-0) [INFO -
  org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)]
  Creating instance of sink: k1, type: logger
  2020-06-14 13:32:12,873 (conf-file-poller-0) [INFO -
  org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:116)]
  Channel c1 connected to [r1, k1]
  2020-06-14 13:32:12,881 (conf-file-poller-0) [INFO -
  org.apache.flume.node.Application.startAllComponents(Application.java:137)]
  Starting new configuration:{ sourceRunners:{r1=PollableSourceRunner: {
  source:com.cevent.source.CeventSource{name:r1,state:IDLE} counterGroup:{
  name:null counters:{} } }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@584c4b3e
  counterGroup:{ name:null counters:{} } }}
  channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
  2020-06-14 13:32:12,896 (conf-file-poller-0) [INFO -
  org.apache.flume.node.Application.startAllComponents(Application.java:144)]
  Starting Channel c1
  2020-06-14 13:32:12,934 (lifecycleSupervisor-1-0) [INFO -
  org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)]
  Monitored counter group for type: CHANNEL, name: c1: Successfully registered
  new MBean.
  2020-06-14 13:32:12,935 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)]
  Component type: CHANNEL, name: c1 started
  2020-06-14 13:32:12,936 (conf-file-poller-0) [INFO -
  org.apache.flume.node.Application.startAllComponents(Application.java:171)]
  Starting Sink k1
  2020-06-14 13:32:12,937 (conf-file-poller-0) [INFO -
  org.apache.flume.node.Application.startAllComponents(Application.java:182)]
  Starting Source r1
  2020-06-14 13:32:12,948
  (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)]
  Event: { headers:{} body: 63 65 76 65 6E 74 4C 4F 47 31                   ceventLOG1 }
  2020-06-14 13:32:13,947
  (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
  org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: {
  headers:{} body: 63 65 76 65 6E 74 4C 4F 47 32                   ceventLOG2 }
  2020-06-14 13:32:14,947
  (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
  org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: {
  headers:{} body: 63 65 76 65 6E 74 4C 4F 47 33                   ceventLOG3 }

2.3source配置




 
  
  ### 自定义源数据source
  # Name the components on this agent
  a1.sources = r1
  a1.sinks = k1
  a1.channels = c1
   
  # Describe/configure the source
  a1.sources.r1.type =
  com.cevent.source.CeventSource
  ##jar自定义属性
  a1.sources.r1.defined_interval = 1000
  a1.sources.r1.defined_prefix = ceventLOG
   
  # Describe the sink
  a1.sinks.k1.type = logger
   
  # Use a channel which buffers events in
  memory
  a1.channels.c1.type = memory
  a1.channels.c1.capacity = 1000
  a1.channels.c1.transactionCapacity = 100
   
  # Bind the source and sink to the channel
  a1.sources.r1.channels = c1
  a1.sinks.k1.channel = c1

2.4自定义源文件配置



（1）自定义源文件配置

package com.cevent.source;/**

 * Created by Cevent on 2020/6/14.

 */



import org.apache.flume.Context;

import org.apache.flume.Event;

import org.apache.flume.EventDeliveryException;

import org.apache.flume.PollableSource;

import org.apache.flume.channel.ChannelProcessor;

import org.apache.flume.conf.Configurable;

import org.apache.flume.event.SimpleEvent;

import org.apache.flume.source.AbstractSource;



import java.io.*;

import java.util.HashMap;



/**

 * @author cevent

 * @description

 * @date 2020/6/14 10:32

 */

public class CeventSourceFileData extends AbstractSource implements Configurable,PollableSource{



    //使用configure配置自定义source元

    private String file;

    private long interval;





    //1.source：获取上游数据，包装成event，发送给channel

    public Status process() throws EventDeliveryException {

        //1.1模拟上游：每隔一段时间向channel发送5条虚拟数据

        //投递状态

        Status status=null;

        //获取channel

        ChannelProcessor channelProcessor=this.getChannelProcessor();

        BufferedReader bufferedReader=null;



        //1.2数据处理的方法

        try {

            bufferedReader= new BufferedReader(new InputStreamReader(new FileInputStream(file)));

            String strBuffered;

            while ((strBuffered=bufferedReader.readLine())!=null){



                Event event=new SimpleEvent();

                event.setBody(strBuffered.getBytes());

                event.setHeaders(new HashMap<String, String>());

                //打包event

                channelProcessor.processEvent(event);



                Thread.sleep(interval);



            }

            //1.5输出处理成功

            status=Status.READY;

        }

        catch (IOException  e){

            //1.6数据处理失败

            status=Status.BACKOFF;

            e.printStackTrace();

        } catch (InterruptedException e) {

            e.printStackTrace();

        } finally {

            if(bufferedReader!=null){

                try {

                    bufferedReader.close();

                } catch (IOException e) {

                    e.printStackTrace();

                }

            }

        }



        return status;

    }



    //2.source投送到channel，channel反馈已满载（投递失败），需要等待挂起，设置2s

    public long getBackOffSleepIncrement() {

        return 2000;

    }



    //3.投递失败，最大等待时间

    public long getMaxBackOffSleepInterval() {

        return 10000;

    }



    //4.根据配置文件，配置自定义source

    public void configure(Context context) {

        //4.1自定义source前缀，不用process的"LOG"，如果没有获取到key，则使用默认value"LOG"



        interval=context.getLong("defined_interval",1000L);

        file=context.getString("defined_file");

    }

}

daoru

2.5配置source.conf




 
  
  ### 自定义源数据文件source-file
  # Name the components on this agent
  a1.sources = r1
  a1.sinks = k1
  a1.channels = c1
   
  # Describe/configure the source
  a1.sources.r1.type =
  com.cevent.source.CeventSourceFileData
  ##jar自定义属性
  a1.sources.r1.defined_interval = 1000
  a1.sources.r1.defined_file =
  /opt/module/datas/dept.txt
   
  # Describe the sink
  a1.sinks.k1.type = logger
   
  # Use a channel which buffers events in
  memory
  a1.channels.c1.type = memory
  a1.channels.c1.capacity = 1000
  a1.channels.c1.transactionCapacity = 100
   
  # Bind the source and sink to the channel
  a1.sources.r1.channels = c1
  a1.sinks.k1.channel = c1

2.6启动source-file




 
  
  [cevent@hadoop207 module]$ cd apache-flume-1.7.0/
  [cevent@hadoop207 apache-flume-1.7.0]$ ll
  总用量 176
  drwxr-xr-x.  2 cevent cevent  4096 6月  11 13:35 bin
  -rw-r--r--.  1 cevent cevent 77387 10月 11 2016 CHANGELOG
  drwxr-xr-x.  2 cevent cevent  4096 6月  12 12:11 conf
  -rw-r--r--.  1 cevent cevent  6172 9月  26 2016 DEVNOTES
  -rw-r--r--.  1 cevent cevent  2873 9月  26 2016 doap_Flume.rdf
  drwxr-xr-x. 10 cevent cevent  4096 10月 13 2016 docs
  drwxrwxr-x.  2 cevent cevent  4096 6月  12 16:43 files
  drwxrwxr-x.  7 cevent cevent  4096 6月  14 13:30 job
  drwxrwxr-x.  2 cevent cevent  4096 6月  14 13:26 lib
  -rw-r--r--.  1 cevent cevent 27625 10月 13 2016 LICENSE
  drwxrwxr-x.  2 cevent cevent  4096 6月  12 11:48 loggers
  drwxrwxr-x.  2 cevent cevent  4096 6月  11 17:05 logs
  -rw-r--r--.  1 cevent cevent   249 9月  26 2016 NOTICE
  -rw-r--r--.  1 cevent cevent  2520 9月  26 2016 README.md
  -rw-r--r--.  1 cevent cevent  1585 10月 11 2016 RELEASE-NOTES
  -rw-rw-r--.  1 cevent cevent   177 6月  12 16:57 tail_dir.json
  drwxrwxr-x.  2 cevent cevent  4096 6月  11 13:35 tools
  -rw-rw-r--.  1 cevent cevent    16 6月  12 16:45 tutu.txt
  drwxrwxr-x.  3 cevent cevent  4096 6月  12 14:23 upload
  [cevent@hadoop207 apache-flume-1.7.0]$ 启动conf
  bin/flume-ng agent -n a1 -c conf/ -f job/source/flume-source-1.conf
  -Dflume.root.logger=INFO,console
  Info: Sourcing environment configuration
  script /opt/module/apache-flume-1.7.0/conf/flume-env.sh
  Info: Including Hadoop libraries found
  via (/opt/module/hadoop-2.7.2/bin/hadoop) for HDFS access
  Info: Including Hive libraries found via
  (/opt/module/hive-1.2.1) for Hive access
  + exec /opt/module/jdk1.7.0_79/bin/java -Xmx20m
  -Dflume.root.logger=INFO,console -cp '/opt/module/apache-flume-1.7.0/conf:/opt/module/apache-flume-1.7.0/lib/*:/opt/module/hadoop-2.7.2/etc/hadoop:/opt/module/hadoop-2.7.2/share/hadoop/common/lib/*:/opt/module/hadoop-2.7.2/share/hadoop/common/*:/opt/module/hadoop-2.7.2/share/hadoop/hdfs:/opt/module/hadoop-2.7.2/share/hadoop/hdfs/lib/*:/opt/module/hadoop-2.7.2/share/hadoop/hdfs/*:/opt/module/hadoop-2.7.2/share/hadoop/yarn/lib/*:/opt/module/hadoop-2.7.2/share/hadoop/yarn/*:/opt/module/hadoop-2.7.2/share/hadoop/mapreduce/lib/*:/opt/module/hadoop-2.7.2/share/hadoop/mapreduce/*:/opt/module/hadoop-2.7.2/contrib/capacity-scheduler/*.jar:/opt/module/hive-1.2.1/lib/*'
  -Djava.library.path=:/opt/module/hadoop-2.7.2/lib/native
  org.apache.flume.node.Application -n a1 -f job/source/flume-source-1.conf
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in
  [jar:file:/opt/module/apache-flume-1.7.0/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in
  [jar:file:/opt/module/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
  explanation.
  2020-06-14 13:32:12,811 (lifecycleSupervisor-1-0) [INFO -
  org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)]
  Configuration provider starting
  2020-06-14 13:32:12,816 (conf-file-poller-0) [INFO -
  org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:134)]
  Reloading configuration file:job/source/flume-source-1.conf
  2020-06-14 13:32:12,824 (conf-file-poller-0) [INFO -
  org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)]
  Added sinks: k1 Agent: a1
  2020-06-14 13:32:12,824 (conf-file-poller-0) [INFO -
  org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)]
  Processing:k1
  2020-06-14 13:32:12,824 (conf-file-poller-0) [INFO -
  org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)]
  Processing:k1
  2020-06-14 13:32:12,841 (conf-file-poller-0) [INFO -
  org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)]
  Post-validation flume configuration contains configuration for agents: [a1]
  2020-06-14 13:32:12,841 (conf-file-poller-0) [INFO -
  org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:147)]
  Creating channels
  2020-06-14 13:32:12,856 (conf-file-poller-0) [INFO -
  org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)]
  Creating instance of channel c1 type memory
  2020-06-14 13:32:12,860 (conf-file-poller-0) [INFO -
  org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:201)]
  Created channel c1
  2020-06-14 13:32:12,861 (conf-file-poller-0) [INFO -
  org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)]
  Creating instance of source r1, type com.cevent.source.CeventSource
  2020-06-14 13:32:12,870 (conf-file-poller-0) [INFO -
  org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)]
  Creating instance of sink: k1, type: logger
  2020-06-14 13:32:12,873 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:116)]
  Channel c1 connected to [r1, k1]
  2020-06-14 13:32:12,881 (conf-file-poller-0) [INFO -
  org.apache.flume.node.Application.startAllComponents(Application.java:137)]
  Starting new configuration:{ sourceRunners:{r1=PollableSourceRunner: {
  source:com.cevent.source.CeventSource{name:r1,state:IDLE} counterGroup:{
  name:null counters:{} } }} sinkRunners:{k1=SinkRunner: {
  policy:org.apache.flume.sink.DefaultSinkProcessor@584c4b3e counterGroup:{
  name:null counters:{} } }}
  channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
  2020-06-14 13:32:12,896 (conf-file-poller-0) [INFO -
  org.apache.flume.node.Application.startAllComponents(Application.java:144)] Starting
  Channel c1
  2020-06-14 13:32:12,934 (lifecycleSupervisor-1-0) [INFO -
  org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)]
  Monitored counter group for type: CHANNEL, name: c1: Successfully registered
  new MBean.
  2020-06-14 13:32:12,935 (lifecycleSupervisor-1-0) [INFO -
  org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)]
  Component type: CHANNEL, name: c1 started
  2020-06-14 13:32:12,936 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:171)]
  Starting Sink k1
  2020-06-14 13:32:12,937 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:182)]
  Starting Source r1
  2020-06-14 13:32:12,948
  (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
  org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: {
  headers:{} body: 63 65 76 65 6E 74 4C 4F 47 31                   ceventLOG1 }
  2020-06-14 13:32:13,947
  (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
  org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: {
  headers:{} body: 63 65 76 65 6E 74 4C 4F 47 32                   ceventLOG2 }
  2020-06-14 13:32:14,947
  (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
  org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: {
  headers:{} body: 63 65 76 65 6E 74 4C 4F 47 33                   ceventLOG3 }
  2020-06-14 13:32:15,947
  (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
  org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: {
  headers:{} body: 63 65 76 65 6E 74 4C 4F 47 34                   ceventLOG4 }
  2020-06-14 13:32:16,948
  (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
  org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: {
  headers:{} body: 63 65 76 65 6E 74 4C 4F 47 35                   ceventLOG5 }

3.自定义Sink

3.1jar包配置



1.     jar包配置

package com.cevent.sink;/**

 * Created by Cevent on 2020/6/14.

 */



import org.apache.flume.*;

import org.apache.flume.conf.Configurable;

import org.apache.flume.sink.AbstractSink;



/**

 * @author cevent

 * @description sink事务控制channel的传输

 * @date 2020/6/14 14:17

 */

public class CeventSink extends AbstractSink implements Configurable {



    private String prefix;

    private String suffix;



    /**

     * 轮循数据方法，执行channel数据传输

     * @return

     * @throws EventDeliveryException

     */

    public Status process() throws EventDeliveryException {

        Status status=null;

        //1.获取channel

        Channel channel=this.getChannel();

        //2.开启事务

        Transaction transaction=channel.getTransaction();

        transaction.begin();



        try {



            //3.处理数据

            Event event;

            //如果获取的数据=null，则等待

            while((event=channel.take())==null){

                Thread.sleep(1000);

            }

            //获取event成功

            byte [] body=event.getBody();

            String line=new String(body,"UTF-8");

            System.out.println(prefix+line+suffix);





            status=Status.READY;

            //成功提交事务

            transaction.commit();

        }catch (Exception e){

            //失败回滚

            transaction.rollback();

            status=Status.BACKOFF;

        }finally {

            //关闭事务

            transaction.close();

        }



        return status;

    }



    /**

     * 配置

     * @param context

     */

    public void configure(Context context) {

        prefix=context.getString("prefix","cevent_prefix");

        suffix=context.getString("suffix","cevent_suffix");

    }

}

import

3.2Sink-conf配置




 
  
  [cevent@hadoop207 job]$ mkdir sinks
  [cevent@hadoop207 job]$ ll
  总用量 40
  -rw-rw-r--. 1 cevent cevent 1542 6月  12 14:22 flume-dir-hdfs.conf
  -rw-rw-r--. 1 cevent cevent 1641 6月  12 13:36 flume-file-hdfs.conf
  -rw-rw-r--. 1 cevent cevent  495 6月  11 17:02
  flume-netcat-logger.conf
  -rw-rw-r--. 1 cevent cevent 1522 6月  12 16:40 flume-taildir-hdfs.conf
  drwxrwxr-x. 2 cevent cevent 4096 6月  13 13:28 group1
  drwxrwxr-x. 2 cevent cevent 4096 6月  13 14:56 group2
  drwxrwxr-x. 2 cevent cevent 4096 6月  13 20:59 group3
  drwxrwxr-x. 2 cevent cevent 4096 6月  13 22:38 interceptor
  drwxrwxr-x. 2 cevent cevent 4096 6月  14 16:51 sinks
  drwxrwxr-x. 2 cevent cevent 4096 6月  14 14:07 source
  [cevent@hadoop207 job]$ vim sinks/flume-sinks-1.conf
  ### 自定义sinks控制channel输出
  # Name the components on this agent
  a1.sources = r1
  a1.sinks = k1
  a1.channels = c1
   
  # Describe/configure the source
  a1.sources.r1.type = netcat
  a1.sources.r1.bind = localhost
  a1.sources.r1.port = 44444
   
  # Describe the sink
  a1.sinks.k1.type = com.cevent.sink.CeventSink
  #a1.sinks.k1.prefix = cevent619:
  a1.sinks.k1.suffix = :cevent619
   
  # Use a channel which buffers events in memory
  a1.channels.c1.type = memory
  a1.channels.c1.capacity = 1000
  a1.channels.c1.transactionCapacity = 100
   
  # Bind the source and sink to the channel
  a1.sources.r1.channels = c1
  a1.sinks.k1.channel = c1
  ~

3.3 启动nc




 
  
  [cevent@hadoop207 apache-flume-1.7.0]$ nc
  localhost 44444
  cev123
  OK
   
  rce.start(NetcatSource.java:169)] Created
  serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
  cevent_prefixcev123:cevent619

3.4onf配置解析




 
  
  ### 自定义sinks控制channel输出
  # Name the components on this agent
  a1.sources = r1
  a1.sinks = k1
  a1.channels = c1
   
  # Describe/configure the source
  a1.sources.r1.type = netcat
  a1.sources.r1.bind = localhost
  a1.sources.r1.port = 44444
   
  # Describe the sink
  a1.sinks.k1.type = com.cevent.sink.CeventSink
  #a1.sinks.k1.prefix = cevent619:
  a1.sinks.k1.suffix = :cevent619
   
  # Use a channel which buffers events in
  memory
  a1.channels.c1.type = memory
  a1.channels.c1.capacity = 1000
  a1.channels.c1.transactionCapacity = 100
   
  # Bind the source and sink to the channel
  a1.sources.r1.channels = c1
  a1.sinks.k1.channel = c1

4.flume数据流监控ganglia

4.1安装yum-gangliam依赖




 
  
  ### ganglia神经网络==>数据监控
  #1.安装httpd服务与php
  sudo yum -y install httpd php
  #2.安装 rrd tool 和 apr devel依赖
  sudo yum -y install rrdtool
  perl-rrdtool-rrdtool-devel
  sudo yum -y install apr-devel
  #3.安装ganglia，及3组件（gmetad、web、gmond）
  sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
  sudo yum -y install ganglia-gmetad
  sudo yum -y install ganglia-web
  sudo yum -y install ganglia-gmond

4.2配置ganglia前端




 
  
  [cevent@hadoop207 apache-flume-1.7.0]$
  sudo vim /etc/httpd/conf.d/ganglia.conf 
  #
  # Ganglia monitoring system php web
  frontend
  #
   
  Alias /ganglia /usr/share/ganglia
   
  <Location /ganglia>
   
  Order deny,allow
   
  Deny from all
   
  Allow from 127.0.0.1
   
  Allow from ::1
    #
  Allow from .example.com
  </Location>

解析



[cevent@hadoop207 apache-flume-1.7.0]$ sudo
vim /etc/httpd/conf.d/ganglia.conf 

#

# Ganglia monitoring system php web
frontend

#

 

Alias /ganglia /usr/share/ganglia

 

<Location /ganglia>

 
Order deny,allow

 
Deny from all

 
Allow from 127.0.0.1

 
Allow from ::1

  #
Allow from .example.com

</Location>

<Location /ganglia>

 
Order deny,allow

  ## 拒绝所有连接

 
Deny from all

  ## 允许指定IP连接

 
Allow from 127.0.0.1

  ## 允许所有IP连接

Allow from all

  ## 允许主机连接

Allow from 192.168.1.1

 

  Allow from ::1



  #
Allow from .example.com

</Location>

优化配置




 
  
  [cevent@hadoop207 apache-flume-1.7.0]$ sudo vim /etc/httpd/conf.d/ganglia.conf 
   
  Allow from 127.0.0.1
  #
  # Ganglia monitoring system php web
  frontend
  #
   
  Alias /ganglia /usr/share/ganglia
   
  <Location /ganglia>
   
  Order deny,allow
   
  Deny from all
   
  Allow from 127.0.0.1
   
  Allow from 192.168.1.1
   
  Allow from ::1
    #
  Allow from .example.com
  </Location>
  ~
  ~

4.3配置数据库gmetad.conf




 
  
  [cevent@hadoop207 apache-flume-1.7.0]$ sudo vim /etc/ganglia/gmetad.conf 
  [sudo] password for cevent: 
  # This is an example of a Ganglia Meta
  Daemon configuration file
  #               
  http://ganglia.sourceforge.net/
  #
  #
  #-------------------------------------------------------------------------------
  # Setting the debug_level to 1 will keep
  daemon in the forground and
  # show only error messages. Setting this
  value higher than 1 will make
  # gmetad output debugging information and
  stay in the foreground.
  # default: 0
  # debug_level 10
  #
  #-------------------------------------------------------------------------------
  # What to monitor. The most important
  section of this file.
  #
  # The data_source tag specifies either a cluster
  or a grid to
  # monitor. If we detect the source is a
  cluster, we will maintain a complete
  # set of RRD databases for it, which can
  be used to create historical
  # graphs of the metrics. If the source is
  a grid (it comes from another gmetad),
  # we will only maintain summary RRDs for
  it.
  #
  # Format:
  # data_source "my cluster"
  [polling interval] address1:port addreses2:port ...
  #
  # The keyword 'data_source' must
  immediately be followed by a unique
  # string which identifies the source,
  then an optional polling interval in
  # seconds. The source will be polled at
  this interval on average.
  # If the polling interval is omitted,
  15sec is asssumed.
  #
  # If you choose to set the polling
  interval to something other than the default,
  # note that the web frontend determines a
  host as down if its TN value is less
  # than 4 * TMAX (20sec by default).  Therefore, if you set the polling interval
  # to something around or greater than
  80sec, this will cause the frontend to
  # incorrectly display hosts as down even
  though they are not.
  #
  # A list of machines which service the
  data source follows, in the
  # format ip:port, or name:port. If a port
  is not specified then 8649
  # (the default gmond port) is assumed.
  # default: There is no default value
  #
  # data_source "my cluster" 10
  localhost  my.machine.edu:8649  1.2.3.5:8655
  # data_source "my grid" 50
  1.3.4.7:8655 grid.org:8651 grid-backup.org:8651
  # data_source "another source"
  1.3.4.7:8655  1.3.4.8
   
  data_source "cevent ganglia cluster" hadoop.cevent.com
   
  #
  # Round-Robin Archives
  # You can specify custom Round-Robin
  archives here (defaults are listed below)
  #
  # Old Default RRA: Keep 1 hour of metrics
  at 15 second resolution. 1 day at 6 minute
  # RRAs "RRA:AVERAGE:0.5:1:244"
  "RRA:AVERAGE:0.5:24:244" "RRA:AVERAGE:0.5:168:244"
  "RRA:AVERAGE:0.5:672:244" \
  #     
  "RRA:AVERAGE:0.5:5760:374"
  # New Default RRA
  # Keep 5856 data points at 15 second
  resolution assuming 15 second (default) polling. That's 1 day
  # Two weeks of data points at 1 minute
  resolution (average)
  #RRAs "RRA:AVERAGE:0.5:1:5856"
  "RRA:AVERAGE:0.5:4:20160" "RRA:AVERAGE:0.5:40:52704"
   
  #
  #-------------------------------------------------------------------------------
  # Scalability mode. If on, we summarize
  over downstream grids, and respect
  # authority tags. If off, we take on
  2.5.0-era behavior: we do not wrap our output
  # in <GRID></GRID> tags, we
  ignore all <GRID> tags we see, and always assume
  # we are the "authority" on
  data source feeds. This approach does not scale to
  # large groups of clusters, but is
  provided for backwards compatibility.

4.4配置监控后端gmond.conf




 
  
  [cevent@hadoop207 apache-flume-1.7.0]$
  sudo vim /etc/ganglia/gmond.conf 
  /* This configuration is as close to
  2.5.x default behavior as possible
    
  The values closely match ./gmond/metric.h definitions in 2.5.x */
  globals {
   
  daemonize = yes
   
  setuid = yes
   
  user = ganglia
   
  debug_level = 0
   
  max_udp_msg_len = 1472
   
  mute = no
   
  deaf = no
   
  allow_extra_data = yes
   
  host_dmax = 86400 /*secs. Expires (removes from web interface) hosts
  in 1 day */
   
  host_tmax = 20 /*secs */
   
  cleanup_threshold = 300 /*secs */
   
  gexec = no
    #
  By default gmond will use reverse DNS resolution when displaying your
  hostname
    #
  Uncommeting following value will override that value.
    #
  override_hostname = "mywebserver.domain.com"
    #
  If you are not using multicast this value should be set to something other
  than 0.
    #
  Otherwise if you restart aggregator gmond you will get empty graphs. 60
  seconds is reasonable
   
  send_metadata_interval = 0 /*secs */
   
  }
   
  /*
   *
  The cluster attributes specified will be used as part of the <CLUSTER>
   *
  tag that will wrap all hosts collected by this instance.
   */
  cluster {
    name = "cevent ganglia cluster"
   
  owner = "unspecified"
   
  latlong = "unspecified"
   
  url = "unspecified"
  }
   
  /* The host section describes attributes
  of the host, like the location */
  host {
   
  location = "unspecified"
  }
   
  /* Feel free to specify as many
  udp_send_channels as you like.  Gmond
    
  used to only support having a single channel */
  udp_send_channel {
   
  #bind_hostname = yes # Highly recommended, soon to be default.
                         # This option tells gmond to use a source
  address
                         # that resolves to the
  machine's hostname.  Without
                         # this, the metrics
  may appear to come from any
                         # interface and the
  DNS names associated with
                         # those IPs will be
  used to create the RRDs.
    # mcast_join = 239.2.11.71
    host=hadoop207.cevent.com
   
  port = 8649
   
  ttl = 1
  }
   
  /* You can specify as many udp_recv_channels
  as you like as well. */
  udp_recv_channel {
    # mcast_join = 239.2.11.71
   
  port = 8649
    bind = 0.0.0.0
   
  retry_bind = true
    #
  Size of the UDP buffer. If you are handling lots of metrics you really
    #
  should bump it up to e.g. 10MB or even higher.
    #
  buffer = 10485760
  }
   
  /* You can specify as many
  tcp_accept_channels as you like to share
    
  an xml description of the state of the cluster */
  tcp_accept_channel {
   
  port = 8649
    #
  If you want to gzip XML output
   
  gzip_output = no
  }
   
  /* Channel to receive sFlow datagrams */
  #udp_recv_channel {
  # 
  port = 6343
  #}
   
  /* Optional sFlow settings */
  #sflow {
  # udp_port = 6343
  # accept_vm_metrics = yes
  # accept_jvm_metrics = yes
  # multiple_jvm_instances = no
  # accept_http_metrics = yes
  # multiple_http_instances = no
  # accept_memcache_metrics = yes
  # multiple_memcache_instances = no
  #}
   
  /* Each metrics module that is referenced
  by gmond must be specified and
    
  loaded. If the module has been statically linked with gmond, it does
    
  not require a load path. However all dynamically loadable modules must
    
  include a load path. */
  modules {
   
  module {
     
  name = "core_metrics"
    }
   
  module {
     
  name = "cpu_module"
     
  path = "modcpu.so"
    }
   
  module {
  /* This configuration is as close to
  2.5.x default behavior as possible
    
  The values closely match ./gmond/metric.h definitions in 2.5.x */

4.5配置setenforce




 
  
  [atguigu@hadoop102 flume]$ sudo vim /etc/selinux/config
  修改为：
  # This file controls the state of SELinux on the system.
  # SELINUX= can take one of these three values:
  #     enforcing - SELinux
  security policy is enforced.
  #     permissive - SELinux
  prints warnings instead of enforcing.
  #     disabled - No SELinux
  policy is loaded.
  SELINUX=disabled
  # SELINUXTYPE= can take one of these two values:
  #     targeted - Targeted
  processes are protected,
  #     mls - Multi Level
  Security protection.
  SELINUXTYPE=targeted
   
  本操作重启生效，如果不想重启，可以临时设置setenforce=0，临时生效
   
  [cevent@hadoop207 apache-flume-1.7.0]$ sudo
  setenforce 0

4.6启动ganglia




 
  
  [cevent@hadoop207 apache-flume-1.7.0]$ sudo service httpd start
  正在启动 httpd：                                           [确定]
  [cevent@hadoop207 apache-flume-1.7.0]$ sudo service gmetad start
  Starting GANGLIA gmetad: Warning: we
  failed to resolve data source name hadoop.cevent.com
                                                            
  [确定]
  [cevent@hadoop207 apache-flume-1.7.0]$ sudo service gmond start
  Starting GANGLIA gmond:                                    [确定]

生效连接：http://hadoop207.cevent.com/ganglia/

ganglia

4.7配置flume-env.sh，测试监控




 
  
  [cevent@hadoop207 apache-flume-1.7.0]$
  vim conf/flume-env.sh 
  # Licensed to the Apache Software
  Foundation (ASF) under one
  # or more contributor license
  agreements.  See the NOTICE file
  # distributed with this work for
  additional information
  # regarding copyright ownership.  The ASF licenses this file
  # to you under the Apache License,
  Version 2.0 (the
  # "License"); you may not use
  this file except in compliance
  # with the License.  You may obtain a copy of the License at
  #
  #    
  http://www.apache.org/licenses/LICENSE-2.0
  #
  # Unless required by applicable law or
  agreed to in writing, software
  # distributed under the License is
  distributed on an "AS IS" BASIS,
  # WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.
  # See the License for the specific
  language governing permissions and
  # limitations under the License.
   
  # If this file is placed at
  FLUME_CONF_DIR/flume-env.sh, it will be sourced
  # during Flume startup.
   
  # Enviroment variables can be set here.
   
  #JAVA_HOME
  export JAVA_HOME=/opt/module/jdk1.7.0_79
  export PATH=$PATH:$JAVA_HOME/bin
   
  #HADOOP_HOME
  export
  HADOOP_HOME=/opt/module/hadoop-2.7.2
  export PATH=$PATH:$HADOOP_HOME/bin
  export PATH=$PATH:$HADOOP_HOME/sbin
   
  #HIVE_HOME
  export HIVE_HOME=/opt/module/hive-1.2.1
  export PATH=$PATH:$HIVE_HOME/bin
   
   
  # Give Flume more memory and
  pre-allocate, enable remote monitoring via JMX
  # export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"
   
  export
  JAVA_OPTS="-Dflume.monitoring.type=ganglia
  -Dflume.monitoring.hosts=hadoop207.cevent.com:8649
  -Xms100m
  -Xmx200m"
   
  # Let Flume write raw event data and
  configuration information to its log files for debugging
  # purposes. Enabling these flags is not
  recommended in production,
  # as it may result in logging sensitive
  user information or encryption secrets.
  # export JAVA_OPTS="$JAVA_OPTS
  -Dorg.apache.flume.log.rawdata=true -Dorg.apache.flume.log.printconfig=true
  "
   
  # Note that the Flume conf directory is
  always included in the classpath.
  #FLUME_CLASSPATH=""

4.8启动




 
  
  [cevent@hadoop207 apache-flume-1.7.0]$ bin/flume-ng agent \
  > --conf conf/ \
  > --name a1 \
  > --conf-file
  job/flume-netcat-logger.conf \
  >
  -Dflume.root.logger==INFO,console \
  >
  -Dflume.monitoring.type=ganglia \
  >
  -Dflume.monitoring.hosts=hadoop207.cevent.com:8649
  Info: Sourcing environment configuration
  script /opt/module/apache-flume-1.7.0/conf/flume-env.sh
  Info: Including Hadoop libraries found
  via (/opt/module/hadoop-2.7.2/bin/hadoop) for HDFS access
   
  
 





 
  
  [cevent@hadoop207 job]$ nc localhost
  44444
  cevent619
  OK
  echo1226
  OK
cevent
关注
  • 0
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Flume -自定义Interceptor（拦截器）
weixin_42289266的博客
08-06 316
1、引入 pom依赖 <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4
Flume开发 -- 自定义Interceptor
a1786742005的博客
03-04 344
一、需求 使用 Flume 采集服务器本地日志，需要按照日志类型的不同，将不同种类的日志发往不同的分析系统。 二、需求分析 在实际的开发中，一台服务器产生的日志类型可能有很多种，不同类型的日志可能需要发送到不同的分析系统。此时会用到 Flume 拓扑结构中的 Multiplexing 结构，Multiplexing的原理是，根据 event 中 Header 的某个 key 的值，将不同的 eve...
Flume自定义Source、Sink和Interceptor(简单功能实现)
weixin_33842328的博客
01-20 155
　　1.Event　　　　event是flume传输的最小对象，从source获取数据后会先封装成event，然后将event发送到channel，sink从channel拿event消费。　　　　event由头headers和身体(body)两部分组成：Headers部分是一个map，body部分可以是String或者byte[]等。其中body部分是真正存放数据的地方，headers部分用于本...
Flume自定义Source、Sink、数据流监控
扛麻袋的少年的博客
08-05 201
参考：  3.6 自定义Source  3.7 自定义Sink  3.8 Flume数据流监控  地址：   阿里云盘：https://www.aliyundrive.com/s/qZqsEN47GZQ   百度网盘：https://pan.baidu.com/s/1j4t8w0rtZ5gVNdy8Fqwo2g  提取码：cobq
flume 自定义Interceptor 按照日志类型的不同，将日志分类打印
xiaonuoyo的博客
05-25 524
flume 自定义拦截器 Interceptor 运行在不同主机上
flume自定义source,sink,intercepor的demo，已经配置好maven插件
08-23
nohup flume-ng agent -n custom_interceptor --conf conf -f conf/custom.conf --plugins-path plugins.d -Dflume.root.logger=INFO,LOGFILE -Dorg.apache.flume.log.printconfig=true -Dorg.apache.flume.log.raw...
Flume监控之Ganglia
01-07
Ganglia简介 Ganglia是UC Berkeley发起的一个开源集群监视可视化工具，设计用于测量数以千计的节点。Ganglia的核心包含gmond、gmetad以及一个Web前端。主要是用来监控系统性能，如：cpu 、mem、硬盘利用率， I/O负载...
Docker容器中运行flume及启动不输出运行日志问题
09-30
在app1和app2的配置中，sink被设置为`file_roll`类型，这意味着它会将数据写入文件，并根据设定的`rollInterval`（86400秒，即24小时）进行文件滚动。`batchSize`定义了每次写入文件的数据量（这里是100个事件），`...
已上线的日志采集系统，使用flume收集日志.zip
最新发布
01-04
Flume是一款由Apache开发的高可用、高可靠的数据收集系统，常用于实时日志流处理。它具有分布式、容错性强的特点，能够从多个源节点收集数据，将数据聚合后传输到一个或多个目标位置。在本系统中，Flume负责从服务器...
Flume自定义source
03-06
Flume自定义Source,数据不丢失,一致性,可以根据自己开发情况选择
Flume自定义sink
HR的博客
01-05 362
Flume自定义sink 1.介绍 Sink不断地轮询Channel 中的事件且批量地移除它们，并将这些事件批量写入到存储或索引系统、或者被发送到另一个Flume Agent。 Sink是完全事务性的。在从Channel 批量删除数据之前，每个Sink用Channel启动一个事务。批量事件一旦成功写出到存储系统或下一个Flume Agent，Sink就利用Channel提交事务。事务一旦被提交，该Channel从自己的内部缓冲区删除事件。 Sink组件目的地包括hdfs、logger、avro、thrift
【Flink】【第五章 流处理API】Source 和 Sink
weixin_43589563的博客
01-19 884
Source 输入Controlling Latency(控制延迟) Source传输控制策略_29集 默认情况下，流中的元素不会一个一个的在网络中传输，而是缓存起来，这样可以减少连接的次数，提高效率； 缓存的大小可以在Flink的配置文件、ExecutionEnvironment、或者某个算子上进行配置； 输入控制延迟默认为100ms 输入延迟的好处：提高吞吐量 输入延迟的缺点：增加了延迟 如何把握平衡？ 如果是为了最大吞吐量，则可以设置setBufferTimeout(-1)，移除timeout
Ganglia
JackLi31742的博客
04-28 1569
Ganglia集群监视项目，包括gmond、gmetad以及一个Web前端 Gmond :是一个守护进程，他运行在每一个需要监测的节点上，收集监测统计，发送和接受在同一个组播或单播通道上的统计信息 如果他是一个发送者(mute=no)他会收集基本指标，比如系统负载（load_one）,CPU利用率。他同时也会发送用户通过添加C/Python模块来自定义的指标。 如果他是一个接收者（deaf=no）
《大数据： Flume 数据流监控》
yexiangCSDN的专栏
10-23 389
一、Flume 数据流监控 1. 在使用flume期间，我们需要监控什么？ 如何实现？ channel当前的容量是多少 channel当前已经使用了多少容量 source向channel中put成功了多少个event sink从channel中take成功了多少个event 使用JMX 2. J2EE定义了14种技术规范 JDBC： java连接数据库的技术规范 Servlet: 所有javaweb写的程序，都最终使用Servlet完成请求的接受和响应 JMX(java mon...
Flume自定义拦截器--idea为例
little_sloth的博客
08-17 412
Flume自定义拦截器–idea为例 文章目录Flume自定义拦截器--idea为例一、新建maven项目二、拦截器逻辑编写三、配置 jar包四、 配置 flume 文件五、 运行程序 一、新建maven项目 maven建项目参考：https://blog.csdn.net/little_sloth/article/details/107026934 导入依赖包 <dependency> <groupId>org.apache.flume</groupId>
关于本地IDEA调试远程的flume
qq_32667911的博客
04-27 311
step1： 在flume/bin/flume-ng文件中对如下配置进行修改： 修改前：JAVA_OPTS="-Xmx20m" 修改后： JAVA_OPTS="-Xmx20m -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y" step2： 在idea的configuration添加remote的配置，修...
Flume内部原理——Interceptor & Selector和IDEA编写自定义拦截器案例
攻城狮Kevin
02-25 712
目录 一、Flume内部原理 二、Interceptor和Selector配置文件案例 三、自定义拦截器案例（IDEA） 一、Flume内部原理 Flume内部从Source -&gt; Channel -&gt; Sink，中间会先后经历拦截器和选择器； （1）Event：一个数据单元，也是Flume数据传输的基本单元，事务类型；          包括一个可选的消息头head...
Flink流处理-DataStream常用Source及Sink
孙大圣的博客
11-26 3405
环境 flink-1.9.0 一、需要的依赖 <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.11</artifactId> <version>1.9.0</versio...
2020-09-06
Nextfugitivedust的博客
09-06 570
第四章、squid代理服务器   一、缓存代理服务： 作为应用层的代理服务软件，squid主要提供缓存加速，应用层过滤控制的功能。   1、代理的工作机制 当客户机通过代理来请求web页面时，指定的代理服务器会先检查自己的缓存，如果缓存中已经有客户机需要的页面，则直接将缓存中的页面内容反馈给客户机，如缓存中没有客户机要访问的页面，则有代理服务器向·Internet发送访问请求，当返回的web页面以后，将网页数据保存到缓存中并发送给客户机。 HTTP代理IDE缓存加速对象主要是文字，图像等
Flume学习笔记：从源到sink的高可用日志采集系统详解
Flume的灵活性体现在其丰富的插件系统，用户可以根据具体场景自定义数据发送方（Source）、数据通道（Channel）和数据接收方（Sink），以适应不同类型的日志源和目的地，如HDFS、Kafka、HBase等。 总结来说，Flume...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值