Flume日志系统简介与应用

一:Flume简介

Flume是一个分布式、可靠、和高可用的海量日志采集、汇聚和传输的系统。Flume针对特殊场景也具备良好的自定义扩展能力，因此，flume可以适用于大部分的日常数据采集场景

二:Flume工作流程

三:Flume架构

1.Client

客户端,数据产生的地方,如web服务器

2.Event

事件,指通过Agent传输的单个数据包,如日志数据通常对应一行数据

3.Agent

代理,一个独立 的JVM进程

Flume以一个或多个Agent部署运行

Agent包含三个组件:Source,Channel,Sink

  

四:安装Flume

1.解压并更名

tar -zxvf /opt/install/apache-flume-1.9.0-bin.tar.gz -C /opt/soft/ --解压
mv apache-flume-1.9.0-bin/ flume190 --更名

 2.进入/opt/soft/flume190/conf目录,复制flume-env.sh-template为flume-env.sh,并编辑

22  export JAVA_HOME=/opt/soft/jdk180/
25  export JAVA_OPTS="-Xms2000m -Xmx2000m -Dcom.sun.managem    ent.jmxremote"

Xms:最小内存大小

Xmx:最大内存大小

3.进入flume190/lib目录下,删除guava11版本,将hadoop的guava27复制过来

rm -rf ./guava-11.0-jre.jar

cp /opt/soft/hadoop313/share/hadoop/hdfs/lib/guava-27.0-jre.jar ./

4. 下载软件

yum install -y net-tools
yum install -y nc
yum install -y telnet-server
yum install -y telnet.*

五:采集日志

(一)指定端口输入

1.flume/conf目录下新建myconf文件夹,进入并新建编辑netcat-logger.conf文件

a1.sources=r1
a1.channels=c1
a1.sinks=k1

a1.sources.r1.type=netcat
a1.sources.r1.bind=localhost
a1.sources.r1.port=7777

a1.channels.c1.type=memory

a1.sinks.k1.type=logger

a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1

 2.另开一个窗口,在flume190的文件夹下输入如下命令进入flume日志系统

./bin/flume-ng agent --name a1 --conf ./conf/ --conf-file ./conf/myconf/filelogger.conf  -Dflume.root.logger=INFO,console

 3.此时在原来的窗口通过telnet输入信息,并可通过flume输出日志

telnet localhost 7777

(二)指定文件输入

1.opt目录下新建tmp文件夹,新建flumelog.conf文件

2./flume190/conf/myconf目录下,新建并编辑filelogger.log文件

a2.sources=r1
a2.channels=c1
a2.sinks=k1

a2.sources.r1.type=exec
a2.sources.r1.command=tail -f /opt/tmp/flumelog.log --指定监控文件的目录

a2.channels.c1.type=memory
a2.channels.c1.capacity=1000
a2.channels.c1.transactionCapacity=100

a2.sinks.k1.type=logger

a2.sources.r1.channels=c1
a2.sinks.k1.channel=c1

3.flume190文件夹下进入flume监控系统,此时能监控到/opt/tmp/flumelog.log

./bin/flume-ng agent --name a2 --conf ./conf/ --conf-file ./conf/myconf/filelogger.conf  -Dflume.root.logger=INFO,console

(三)指定文件并上传到hdfs中

 1./flume190/conf/myconf目录下,新建并编辑file-flume-hdfs.conf文件

a3.sources=r1
a3.channels=c1
a3.sinks=k1

a3.sources.r1.type=exec
a3.sources.r1.command=tail -f /opt/tmp/flumelog.log

a3.channels.c1.type=memory
a3.channels.c1.capacity=1000
a3.channels.c1.transactionCapacity=100

a3.sinks.k1.type=hdfs
a3.sinks.k1.hdfs.fileType=DataStream
a3.sinks.k1.hdfs.filePrefix=flumetohdfs
a3.sinks.k1.hdfs.fileSuffix=.txt
a3.sinks.k1.hdfs.path=hdfs://192.168.10.129:9000/kb23flume/

a3.sources.r1.channels=c1
a3.sinks.k1.channel=c1

2.flume190文件夹下进入flume监控系统,此时hdfs中会出现kb23flume文件夹

./bin/flume-ng agent --name a3 --conf ./conf/ --conf-file ./conf/myconf/file-flume-hdfs.conf  -Dflume.root.logger=INFO,console

(四)多个channel,sinks ,指定端口输入,输出到hdfs中

1./flume190/conf/myconf目录下,新建并编辑file1.conf文件

a5.sources=r1
a5.channels=c1 c2
a5.sinks=k1 k2

a5.sources.r1.type=netcat
a5.sources.r1.bind=localhost
a5.sources.r1.port=7777

a5.channels.c1.type=memory
a5.channels.c2.type=memory
a6.channels.c1.capacity=1000
a6.channels.c1.transactionCapacity=100
a6.channels.c2.capacity=1000
a6.channels.c2.transactionCapacity=100

a5.sinks.k1.type=logger
a5.sinks.k2.type=hdfs
a5.sinks.k2.hdfs.fileType=DataStream
a5.sinks.k2.hdfs.filePrefix=flumetohdfs
a5.sinks.k2.hdfs.fileSuffix=.txt
a5.sinks.k2.hdfs.path=hdfs://192.168.10.129:9000/kb23flume2/

a5.sources.r1.channels=c1 c2
a5.sinks.k1.channel=c1
a5.sinks.k2.channel=c2

2.flume190文件夹下进入flume监控系统,此时hdfs中会出现kb23flume2文件夹

./bin/flume-ng agent --name a5 --conf ./conf/ --conf-file ./conf/myconf/file1.conf  -Dflume.root.logger=INFO,console

 (五)指定文件源,并配置过滤器,并传输到kafka中

1./opt/kb23目录下新建三个文件夹,分别是/checkpoint/events;/data/events;flumeligfile

2.flumeligfile文件夹下导入数据源(ps:由于源数据太大,故不放入,可根据需求自行定义)

3./flume190/conf/myconf目录下,新建并编辑events-flume-kafka-logger.conf

events.sources=er
events.channels=ec
events.sinks=es

events.sources.er.type=spooldir
events.sources.er.spoolDir=/opt/kb23/flumeligfile/events
events.sources.er.deserializer=LINE
events.sources.er.deserializer.maxLineLength=32000
events.sources.er.includePattern=events_[0-9]{4}-[0-9]{2}-[0-9]{2}.csv
events.sources.er.interceptors=head_filter
events.sources.er.interceptors.head_filter.type=regex_filter
events.sources.er.interceptors.head_filter.regex=^event_id*
events.sources.er.interceptors.head_filter.excludeEvents=true

#events.channels.ec.type=memory
events.channels.ec.type=file
events.channels.ec.checkpointDir=/opt/kb23/checkpoint/events
events.channels.ec.dataDirs=/opt/kb23/data/events


events.sinks.es.type=org.apache.flume.sink.kafka.KafkaSink
events.sinks.es.topic=events
events.sinks.es.brokerList=192.168.10.129:9092
events.sinks.es.batchSize=640

events.sources.er.channels=ec
events.sinks.es.channel=ec

4.flume190文件夹下进入flume监控系统,此时kafka中会消费此源数据

./bin/flume-ng agent --name events --conf ./conf/ --conf-file ./conf/myconf/events-flume-kafka-logger.conf  -Dflume.root.logger=INFO,console

(六)编写java程序打包成jar包用于过滤字段,根据字段分别传输到logger/hdfs/kafka

1.idea中编写过滤程序并打包成jar包

package nj.zj.flume;

import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.interceptor.Interceptor;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;

public class interceptorDemo implements Interceptor {
    private ArrayList<Event> opList;
    @Override
    public void initialize() {
        opList = new ArrayList<>();
    }

    @Override
    public Event intercept(Event event) {
        Map<String, String> headers = event.getHeaders();
        String body = new String(event.getBody());
        if(body.startsWith("hello")){
            headers.put("type","hello");
        }else if(body.startsWith("hi")){
            headers.put("type","hi");
        }else {
            headers.put("type","other");
        }
        return event;
    }

    @Override
    public List<Event> intercept(List<Event> list) {
        opList.clear();
        for (Event event:
             list) {
            opList.add(event);
        }
        return opList;
    }

    @Override
    public void close() {
        opList.clear();
        opList=null;
    }

    public static class Builder implements Interceptor.Builder{
        @Override
        public Interceptor build() {
            return new interceptorDemo();
        }

        @Override
        public void configure(Context context) {

        }
    }
}

2.将jar包上传到/opt/soft/flume190/lob目录中

3./flume190/conf/myconf目录下,新建并编辑netcat-myinterceptor.conf文件

my.sources=s1
my.channels=hellochannel hichannel otherchannel
my.sinks=hellosink hisink othersink

my.sources.s1.type=netcat
my.sources.s1.bind=localhost
my.sources.s1.port=7777
my.sources.s1.interceptors=myinterceptors
my.sources.s1.interceptors.myinterceptors.type=nj.zj.flume.interceptorDemo$Builder
my.sources.s1.selector.type=multiplexing
my.sources.s1.selector.mapping.hello=hellochannel
my.sources.s1.selector.mapping.hi=hichannel
my.sources.s1.selector.mapping.other=otherchannel
my.sources.s1.selector.header=type

my.channels.hellochannel.type=memory
my.channels.hichannel.type=memory
my.channels.otherchannel.type=memory

my.sinks.othersink.type=logger

my.sinks.hellosink.type=hdfs
my.sinks.hellosink.hdfs.fileType=DataStream
my.sinks.hellosink.hdfs.filePrefix=hellocontent
my.sinks.hellpsink.hdfs.fileSuffix=.txt
my.sinks.hellosink.hdfs.path=hdfs://192.168.10.129:9000/kb23hello/

my.sinks.hisink.type=org.apache.flume.sink.kafka.KafkaSink
my.sinks.hisink.topic=hitopic
my.sinks.hisink.brokerList=192.168.10.129:9092
my.sinks.hisink.batchSize=100

my.sources.s1.channels=hellochannel hichannel otherchannel
my.sinks.othersink.channel=otherchannel
my.sinks.hisink.channel=hichannel
my.sinks.hellosink.channel=hellochannel

4.flume190文件夹下进入flume监控系统,此时根据过滤可查看对应的上传状态

./bin/flume-ng agent --name my --conf ./conf/ --conf-file ./conf/myconf/netcat-myinterceptor.conf -Dflume.root.logger=INFO,console