Flume日志采集流程(log-＞kafka-＞hdfs)

最新推荐文章于 2024-03-14 16:18:49 发布
最新推荐文章于 2024-03-14 16:18:49 发布
阅读量1.4k 收藏 5
点赞数 1
文章标签： flume log4j kafka java hdfs
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/mogeko1/article/details/129714970
版权

埋点数据：用户访问业务服务器如Nginx，利用log4j的技术，将客户端的埋点数据以日志的形式记录在文件中

服务器日志文件——>HDFS文件

日志文件——>Flume(agent source(interceptor) channel)——>kafka topic ——>
Flume(agent source(interceptor) channel sink) ——> HDFS文件

环境：Hadoop+zookeeper+kafka+flume

准备好日志文件放入本地或hdfs中
日志文件appdemo.log

linux>vi appdemo.log
{"eventid":"appClickEvent","event":{"screen_id":"344","screen_name":"","title":"","element_id":"4"},"user":{"uid":"245498","account":"","email":"","phoneNbr":"18248667380","birthday":"","isRegistered":"",
"isLogin":"","addr":"","gender":"","phone":{"imei":"2881993463620531","mac":"2e-80-50-8e-39-a1-1e","imsi":"8616932323350461","osName":"macos","osVer":"9.0","androidId":"","resolution":"1024*768","deviceType":"360_V","deviceId":"81Kau4","uuid":"L3whyU7BgtLKEkvE"},"app":{"appid":"cn.kgc.mall","appVer":"2.0.1","release_ch":"纽扣助手","promotion_ch":"12"},"loc":{"areacode":210921102,"longtitude":121.56605311428365,"latitude":41.91452099352481,"carrier":"ISP02","netType":"WIFI","cid_sn":"463485993989","ip":"138.117.92.76"},"sessionId":"sid-99fe7648-d8e4-4cbe-86af-17b5b3c3a7fc"},"timestamp":"1575548955000"}{"eventid":"appClickEvent","event":{"screen_id":"344","screen_name":"","title":"","element_id":"6"},"user":{"uid":"245498","account":"","email":"","phoneNbr":"18248667380","birthday":"","isRegistered":"",
"isLogin":"","addr":"","gender":"","phone":{"imei":"2881993463620531","mac":"2e-80-50-8e-39-a1-1e","imsi":"8616932323350461","osName":"macos","osVer":"9.0","androidId":"","resolution":"1024*768","deviceType":"360_V","deviceId":"81Kau4","uuid":"L3whyU7BgtLKEkvE"},"app":{"appid":"cn.kgc.mall","appVer":"2.0.1","release_ch":"纽扣助手","promotion_ch":"12"},"loc":{"areacode":210921102,"longtitude":121.56605311428365,"latitude":41.91452099352481,"carrier":"ISP02","netType":"WIFI","cid_sn":"463485993989","ip":"138.117.92.76"},"sessionId":"sid-99fe7648-d8e4-4cbe-86af-17b5b3c3a7fc"},"timestamp":"1575548957000"}{"eventid":"appClickEvent","event":{"screen_id":"344","screen_name":"","title":"","element_id":"8"},"user":{"uid":"245498","account":"","email":"","phoneNbr":"18248667380","birthday":"","isRegistered":"",
"isLogin":"","addr":"","gender":"","phone":{"imei":"2881993463620531","mac":"2e-80-50-8e-39-a1-1e","imsi":"8616932323350461","osName":"macos","osVer":"9.0","androidId":"","resolution":"1024*768","deviceType":"360_V","deviceId":"81Kau4","uuid":"L3whyU7BgtLKEkvE"},"app":{"appid":"cn.kgc.mall","appVer":"2.0.1","release_ch":"纽扣助手","promotion_ch":"12"},"loc":{"areacode":210921102,"longtitude":121.56605311428365,"latitude":41.91452099352481,"carrier":"ISP02","netType":"WIFI","cid_sn":"463485993989","ip":"138.117.92.76"},"sessionId":"sid-99fe7648-d8e4-4cbe-86af-17b5b3c3a7fc"},"timestamp":"1575548959000"}{"eventid":"appClickEvent","event":{"screen_id":"344","screen_name":"","title":"","element_id":"17"},"user":{"uid":"245498","account":"","email":"","phoneNbr":"18248667380","birthday":"","isRegistered":""
,"isLogin":"","addr":"","gender":"","phone":{"imei":"2881993463620531","mac":"2e-80-50-8e-39-a1-1e","imsi":"8616932323350461","osName":"macos","osVer":"9.0","androidId":"","resolution":"1024*768","deviceType":"360_V","deviceId":"81Kau4","uuid":"L3whyU7BgtLKEkvE"},"app":{"appid":"cn.kgc.mall","appVer":"2.0.1","release_ch":"纽扣助手","promotion_ch":"12"},"loc":{"areacode":210921102,"longtitude":121.56605311428365,"latitude":41.91452099352481,"carrier":"ISP02","netType":"WIFI","cid_sn":"463485993989","ip":"138.117.92.76"},"sessionId":"sid-99fe7648-d8e4-4cbe-86af-17b5b3c3a7fc"},"timestamp":"1575548960000"}{"eventid":"appClickEvent","event":{"screen_id":"344","screen_name":"","title":"","element_id":"4"},"user":{"uid":"245498","account":"","email":"","phoneNbr":"18248667380","birthday":"","isRegistered":"",
"isLogin":"","addr":"","gender":"","phone":{"imei":"2881993463620531","mac":"2e-80-50-8e-39-a1-1e","imsi":"8616932323350461","osName":"macos","osVer":"9.0","androidId":"","resolution":"1024*768","deviceType":"360_V","deviceId":"81Kau4","uuid":"L3whyU7BgtLKEkvE"},"app":{"appid":"cn.kgc.mall","appVer":"2.0.1","release_ch":"纽扣助手","promotion_ch":"12"},"loc":{"areacode":210921102,"longtitude":121.56605311428365,"latitude":41.91452099352481,"carrier":"ISP02","netType":"WIFI","cid_sn":"463485993989","ip":"138.117.92.76"},"sessionId":"sid-99fe7648-d8e4-4cbe-86af-17b5b3c3a7fc"},"timestamp":"1575548961000"}{"eventid":"clickChannelEvent","event":{"belongTab":"6","channelIndex":"6","channelID":"6","channelName":""},"user":{"uid":"245498","account":"","email":"","phoneNbr":"18248667380","birthday":"","isRegist
ered":"","isLogin":"","addr":"","gender":"","phone":{"imei":"2881993463620531","mac":"2e-80-50-8e-39-a1-1e","imsi":"8616932323350461","osName":"macos","osVer":"9.0","androidId":"","resolution":"1024*768","deviceType":"360_V","deviceId":"81Kau4","uuid":"L3whyU7BgtLKEkvE"},"app":{"appid":"cn.kgc.mall","appVer":"2.0.1","release_ch":"纽扣助手","promotion_ch":"12"},"loc":{"areacode":210921102,"longtitude":121.56605311428365,"latitude":41.91452099352481,"carrier":"ISP02","netType":"WIFI","cid_sn":"463485993989","ip":"138.117.92.76"},"sessionId":"sid-99fe7648-d8e4-4cbe-86af-17b5b3c3a7fc"},"timestamp":"1575548963000"}{"eventid":"appviewEvent","event":{"screen_id":"681","screen_name":"","title":""},"user":{"uid":"611429","account":"","email":"","phoneNbr":"15669989777","birthday":"","isRegistered":"","isLogin":"","addr
":"","gender":"","phone":{"imei":"2478250569537801","mac":"a1-27-fd-a6-9a-0a-c9","imsi":"5597152087882061","osName":"windows","osVer":"9.0","androidId":"","resolution":"800*600","deviceType":"XIAOLAJIAO","deviceId":"2Ry864","uuid":"EFVu1NVmld9Dpykq"},"app":{"appid":"cn.kgc.mall","appVer":"1.0.0","release_ch":"应用宝","promotion_ch":"04"},"loc":{"areacode":130403205,"longtitude":114.43980636414214,"latitude":36.690319995072464,"carrier":"ISP07","netType":"N","cid_sn":"021803165366","ip":"42.24.217.87"},"sessionId":"sid-e760ea4b-5016-4eb3-bd6c-a9eaa2f27314"},"timestamp":"1575595490000"}{"eventid":"webStayEvent","event":{"pgid":"681","title":"","url":"http://www.kgcedu.cn/acc/pg681"},"user":{"uid":"611429","account":"","email":"","phoneNbr":"15669989777","birthday":"","isRegistered":"","
isLogin":"","addr":"","gender":"","phone":{"imei":"2478250569537801","mac":"a1-27-fd-a6-9a-0a-c9","imsi":"5597152087882061","osName":"windows","osVer":"9.0","androidId":"","resolution":"800*600","deviceType":"XIAOLAJIAO","deviceId":"2Ry864","uuid":"EFVu1NVmld9Dpykq"},"app":{"appid":"cn.kgc.mall","appVer":"1.0.0","release_ch":"应用宝","promotion_ch":"04"},"loc":{"areacode":130403205,"longtitude":114.43980636414214,"latitude":36.690319995072464,"carrier":"ISP07","netType":"N","cid_sn":"021803165366","ip":"42.24.217.87"},"sessionId":"sid-e760ea4b-5016-4eb3-bd6c-a9eaa2f27314"},"timestamp":"1575595491000"}

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>test.flume</groupId>
    <artifactId>flume-interceptor</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>


    <dependencies>
        <dependency>
            <groupId>org.apache.flume</groupId>
            <artifactId>flume-ng-core</artifactId>
            <version>1.9.0</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.69</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.2</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

idea:编写json连接器过滤出json数据

package flume.intercepter;

import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.interceptor.Interceptor;

import java.nio.charset.StandardCharsets;
import java.util.Iterator;
import java.util.List;

//第一个拦截器
public class JsonInterceptor implements Interceptor {
    @Override
    public void initialize() {
    }

    @Override
    public Event intercept(Event event) {
        /**
         * agent 中传递的数据格式就是一个event：header body
         * 文件：一行就是一个event
         */

        String log = new String(event.getBody(), StandardCharsets.UTF_8);

        /**
         * 每条记录是否是一个完整的jons记录
         */
        if (JsonUtil.isJSONValidate(log)) {
            return event;
        } else {
            return null;
        }
    }

    @Override
    public List<Event> intercept(List<Event> events) {

        Iterator<Event> iterator = events.iterator();
        while (iterator.hasNext()) {
            Event next = iterator.next();
            if (intercept(next) == null) {
                iterator.remove();
            }
        }
        return events;
    }

    @Override
    public void close() {

    }

    public static class Builder implements Interceptor.Builder {

        @Override
        public Interceptor build() {
            return new JsonInterceptor();
        }


        @Override
        public void configure(Context context) {

        }
    }
}

Idea:编写时间戳接口(用于flume采集到hdfs上时将时间设为数据时间而非hdfs的时间)

package flume.intercepter;

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.interceptor.Interceptor;

import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;

//第二个拦截器 提前时间戳放入headers里
public class TimestampIntercepter implements Interceptor {

    List<Event> events=new ArrayList<>();

    @Override
    public void initialize() {
    }

    @Override
    public Event intercept(Event event) {
         Map<String,String> headers=event.getHeaders();      //取头部
         String log=new String(event.getBody(), StandardCharsets.UTF_8); //取主体

         JSONObject jsonObject = JSON.parseObject(log); //解析里面的东西
         String ts = jsonObject.getString("timestamp");

         headers.put("timestamp", ts);    //放到里面

    //取出日志内的时间，否则落盘时可能是系统时间
           return event;
    }

    @Override
    public List<Event> intercept(List<Event> list) {
        events.clear();
        for (Event event : list) {
            events.add(intercept(event));
        }
        return events;
    }

    @Override
    public void close() {
    }


    public static class Builder implements Interceptor.Builder {
        @Override
        public Interceptor build() {
            return new TimestampIntercepter();
        }

        @Override
        public void configure(Context context) {
        }
    }
}

编写flume运行文件
LogToKafka:

linux>vi log_to_kafka.conf
##为各组件命名
logToKafka.sources = logtokafkasource
logToKafka.channels = logkafkachannel

#描述source
logToKafka.sources.logtokafkasource.type = TAILDIR
logToKafka.sources.logtokafkasource.filegroups = f1
logToKafka.sources.logtokafkasource.filegroups.f1 = /root/tmp_data/log/appdemo.*
logToKafka.sources.logtokafkasource.positionFile = /root/tmp_data/log/taildir_position.json
logToKafka.sources.logtokafkasource.interceptors =  i1
logToKafka.sources.logtokafkasource.interceptors.i1.type = flume.intercepter.JsonInterceptor$Builder
#filegroups 可以指定多个文件夹	positionFile 采集完后记录文件位置
#描述channel
logToKafka.channels.logkafkachannel.type = org.apache.flume.channel.kafka.KafkaChannel
logToKafka.channels.logkafkachannel.kafka.bootstrap.servers = 192.168.58.201:9092,192.168.58.202:9092,192.168.58.203:9092
logToKafka.channels.logkafkachannel.kafka.topic = appdemo_log
logToKafka.channels.logkafkachannel.parseAsFlumeEvent = false

#绑定source和channel以及sink和channel的关系
logToKafka.sources.logtokafkasource.channels = logkafkachannel

KafkaToHDFS:

linux>vi kafka_to_hdfs.conf
## 组件
a1.sources=r1
a1.channels=c1
a1.sinks=k1

## source1
a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.batchSize = 5000
a1.sources.r1.batchDurationMillis = 2000
a1.sources.r1.kafka.bootstrap.servers = 192.168.58.201:9092,192.168.58.202:9092,192.168.58.203:9092
a1.sources.r1.kafka.topics=appdemo_log
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = flume.intercepter.TimestampIntercepter$Builder

## channel1
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /root/tmp_data/flume/checkpoint/behavior1
a1.channels.c1.dataDirs = /root/tmp_data/flume/data/behavior1/
a1.channels.c1.maxFileSize = 2146435071
a1.channels.c1.capacity = 1000000
a1.channels.c1.keep-alive = 6
## sink1
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /data/log/appdemo_log/%Y-%m-%d
a1.sinks.k1.hdfs.filePrefix = log-
a1.sinks.k1.hdfs.round = false


a1.sinks.k1.hdfs.rollInterval = 10
a1.sinks.k1.hdfs.rollSize = 134217728
a1.sinks.k1.hdfs.rollCount = 0

## 控制输出文件是原生文件。
#a1.sinks.k1.hdfs.fileType = CompressedStream
#a1.sinks.k1.hdfs.codeC = lzop

## 拼装
a1.sources.r1.channels = c1
a1.sinks.k1.channel= c1

注意:运行kafka_to_hdfs.conf 需要提前在本地创好文件夹

a1.channels.c1.checkpointDir = /root/tmp_data/flume/checkpoint/behavior1
a1.channels.c1.dataDirs = /root/tmp_data/flume/data/behavior1/

这两项需要创建好文件夹

linux>mkdir -p /root/tmp_data/flume/checkpoint/behavior1
linux>mkdir -p /root/tmp_data/flume/data/behavior1/

注意:idea拦截器编辑好后 打包放入flume/lib中

linux>mv flume-interceptor-1.0-SNAPSHOT-jar-with-dependencies.jar /opt/install/flume/lib

运行flume进行logToKafka

linux>bin/flume-ng agent --conf conf --conf-file log_to_kafka.conf --name logToKafka -Dflume.root.logger=INFO,console

运行flume进行KafkaToHdfs

linux>bin/flume-ng agent --conf conf --conf-file kafka_to_hdfs.conf --name a1 -Dflume.root.logger=INFO,console

报错1：运行完后异常 hdfs无文件，需要重置kafka topic 偏移量否则不读或缺数据

旧版kafka重置偏移量:
linux>kafka-streams-application-reset.sh --zookeeper 192.168.58.201:2181,192.168.58.202:2181,192.168.58.203:2181 --bootstrap-servers 192.168.58.201:9092,192.168.58.202:9092,192.168.58.203:9092--application-id flume --input-topics appdemo_log
旧版kafka查看topic的记录的条数
linux>bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 192.168.58.201:9092,192.168.58.202:9092,192.168.58.203:9092- --topic appdemo_log -time -1 --offsets 1

新版kafka重置偏移量:
linux>kafka-consumer-groups.sh --bootstrap-server 192.168.58.201:9092,192.168.58.202:9092,192.168.58.203:9092 --group flume --reset-offsets --topic appdemo_log --to-earliest --execute
#查看组
kafka-consumer-groups.sh --bootstrap-server 192.168.58.201:9092,192.168.58.202:9092,192.168.58.203:9092 --list	
#查看所有Topic
kafka-topics.sh --list --bootstrap-server 192.168.58.201:9092,192.168.58.202:9092,192.168.58.203:9092

#查看Topic
linux>kafka-topics.sh --bootstrap-server  192.168.58.201:9092,192.168.58.202:9092,192.168.58.203:9092 --describe  --topic appdemo_log	

#删除Topic
linux>kafka-topics.sh --bootstrap-server 192.168.58.201:9092,192.168.58.202:9092,192.168.58.203:9092 --delete --topic  appdemo_log	
查看topic的记录的条数
linux>kafka-run-class.sh  kafka.tools.GetOffsetShell --broker-list 192.168.58.201:9092,192.168.58.202:9092,192.168.58.203:9092 --topic appdemo_log	 --time -1

#消费消息（可能端口号与配置文件保持一致，或与发送端口保持一致）
linux>kafka-console-consumer.sh --bootstrap-server 192.168.58.201:9092,192.168.58.202:9092,192.168.58.203:9092 --topic appdemo_log	--from-beginning   #加了--from-beginning 重头消费所有的消息
linux>kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic appdemo_log        #不加--from-beginning 从最新的一条消息开始消费


#查询topic的offset的范围offset的最小值
linux>kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 192.168.58.201:9092,192.168.58.202:9092,192.168.58.203:9092  -topic appdemo_log	 --time -2
#查询offset的最大值：
linux>bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 192.168.58.201:9092,192.168.58.202:9092,192.168.58.203:9092 -topic appdemo_log  --time -1

报错2：java.lang.OutOfMemoryError: Java heap space 内存不足

###解决flume内存不足 OOM  在flume-env.sh
linux>vi flume-env.sh
export JAVA_OPTS="-Xms2048m -Xmx2048m -Xss256k -Xmn1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit"

结果：

 

房石阳明i
关注
  • 1
    点赞
  • 踩
  • 5
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Flume日志采集系统案例-教案.pdf
07-10
Flume日志采集系统案例-教案.pdf 学习资料 复习资料 教学资源
Kafka hdfs flume 数据采集实验
11-17
mysql-connector-java-8.0.23.jar
Flume数据采集（行为日志数据）
最新发布
qq_45111566的博客
03-14 879
用日志文件使用一个脚本来生成存放在集群某一个节点，由于Flume是一个单点操作，故需要在该节点配置日志采集Flume。且采集到的日志进行JSON格式校验，然后将通过校验的JSON日志发送到kafka。Flume怎么校验？Source的类型选择？选择TailDirSource的原因，可以实现断点续传，**多目录多文件实时**监控？Channel的类型选择？选择kafkaChannel，省去了Sink,，提高了效率。kafka可以均衡速率，kafkachannel直接到kafka,不用sink。
flume-ng-kafka-sink
06-13
水槽-ng-kafka-sink 注意flume-ng-kafka-sink 已合并到即将发布的flume 1.6 中。 这个 repo不会在维护中。 该项目用于 与进行通信。 Kafka Sink的配置 agent_log.sinks.kafka.type = com.vipshop.flume.sink.kafka.KafkaSink agent_log.sinks.kafka.channel = all_channel agent_log.sinks.kafka.zk.connect = 127.0.0.1:2181 agent_log.sinks.kafka.topic = all agent_log.sinks.kafka.batchsize = 200 agent_log.sinks.kafka.producer.type = async agent_log.sinks.
flume-kafka-storm-drools-example
06-27
水槽卡夫卡风暴流口水示例 这个项目使用来自kafka的storm spout消费者日志消息并使用drools定义他们的规则，最后将有用的数据保存到redis。 你可以在看到更多信息
Flume-ng-1.6.0-cdh.zip
04-08
Flume-ng-1.6.0-cdh.zip 内压缩了 3 个项目，分别为：flume-ng-1.6.0-cdh5.5.0.tar.gz、flume-ng-1.6.0-cdh5.7.0.tar.gz 和 flume-ng-1.6.0-cdh5.10.1.tar.gz，选择你需要的版本。
nginx+flume 数据采集
需要远程指导仿真实验、代码有问题的，请后台私信或者关注公众号
07-06 3376
本实验任务主要完成基于ubuntu环境使用nginx+flume的工作方式进行数据采集。通过完成本实验任务，要求学生了解并掌握nginx的安装、nginx的基础语法、采集数据方法以及配置格式，为从事大数据平台运维工程师、大数据技术支持工程师等岗位工作奠定夯实的技能基础。掌握flume的应用原理   掌握nginx+flume对日志信息的采集过程本次环境是：Ubuntu16.04+flume-ng-1.5.0-cdh5.3.6Flume是Cloudera提供的日志收集系统，Flume支持在日志系统中定制各类数
大数据之Kafka———Flume日志系统写入kafka
ycz926940的博客
07-06 670
3.启动ag 其中有三个组件Source(监管当前机器的端口)，Channel（进行分流），Sink（进行输出内容）（端口启动后可以在端口中输入内容 可以在上面启动的kafka里查看到）1./opt/events/data文件下的users.csv。2.连接好端口可以在端口输入内容由 ng进行处理。4.通过Sink将数据写到kb22在启动消费者。1.通过Telnet 方式连接当前机器的端口。如果端口启动不了查看一下端口是否被占用。5.将文件拷贝到监控目录下。# 4. 配置sink。3.创建队列 生产者。
如何利用flume进行日志采集
EverythingAtOnce的博客
12-24 2835
Apache Flume 是一个分布式、可靠、高可用的日志收集、聚合和传输系统。它常用于将大量日志数据从不同的源（如Web服务器、应用程序、传感器等）收集到中心化的存储或数据处理系统中。Apache Flume 是一个强大的数据收集和传输工具，可以轻松地处理大量的数据流。使用本文提供的示例和概念，可以开始使用 Flume 并适应不同的数据采集和传输需求。本篇文章就介绍到这里，感兴趣的同学可以自己去深入研究，使用flume完成日志采集、关键字告警等一整套的业务。
日志采集Flume配置
想练武，就得下功夫
07-08 543
日志采集 Flume 配置
Flume采集数据到Kafka操作详解
Helen_1997_1997的博客
04-21 2912
Flume采集数据到Kafka操作详解
Log-File-Processing-Data-Pipeline:使用Lambda架构构建的日志文件处理数据管道| 水槽| Apache Spark | 火花流| Apache Kafka | HDFS | Hbase | 蜂巢| 黑斑羚| Oozie
05-10
日志文件处理数据管道使用Lambda架构构建的日志文件处理数据管道| 水槽| Apache Spark | 火花流| Apache Kafka | HDFS | Hbase | 蜂巢| 黑斑羚| Oozie介绍•从Web服务器日志存储，处理和挖掘数据已成为当今许多公司...
LogAnalyzer:分析大数据组件的客户日志，例如HDFS，Hive，HBase，Yarn，MapReduce，Storm，Spark，Spark 2，Knox，Ambari Metrics，Nifi，Accumulo，Kafka，Flume，Oozie，Falcon，Atlas和Zookeeper
05-12
日志分析器-分析大数据组件的客户日志，例如HDFS，Hive，HBase，Yarn，MapReduce，Storm，Spark，Spark 2，Knox，Ambari Metrics，Nifi，Accumulo，Kafka，Flume，Oozie，Falcon，Atlas和Zookeeper。 内部架构 分析...
Spark-Streaming:Spark Streaming实时解析flume和kafka传来的josn数据写入mysql
05-03
Spark Streaming实时解析flume和kafka传来的josn数据写入mysql 注意，以下文件不提供 配置c3p0-config.xml链接，链接数据库 配置log4j.properties、my.properties 另，还需将您的spark和hadoop安装文件下的core-site...
java大数据案例_7Flume、Kafka、Sqoop、Lucene
01-23
1 获取accesslog 问题 依次完成以下小题的任务 1） 将nginx日志收集到HDFS中
flume 真实案例配置文件
10-18
flume agent收集数据 一个源对于两个sink 同时输出到hdfs和kafka 的配置文件，注意其中的 source绑定channel时候 channel1 channel2 不能分开写，该配置文件已经过集群实验成功收集到数据的
日志采集工具之Flume实操-上
xiaoleilei666的博客
07-11 1503
主要讲述使用flume工具采集日志；
日志系统之Flume日志收集
VinoYang的专栏
06-06 7699
本文介绍在日志系统中如何使用flume agent做日志收集以及为了满足我们的需求对flume agent进行了哪些扩展。
flume采集数据
weixin_42796403的博客
12-10 625
今天写了一个flume的配置文件采集日志文件（json格式），将日志文件采集到kafka，结果在kafkatools工具中发现采集的每一条数据前面都多出了两个字节的乱码，显然后面的操作都是以json格式进行处理，数据被阻塞在kafka中。 后来进行了反复检查，发现问题还是出现在flume配置文件中。。。。。。。。。。。。 在这里配置是否以event格式写入kafka时，多加了kafka，flume读取数据时还是以默认设置以event格式读入数据，结果event中的header信息保存到kafka后变成了乱
帮我写一个数据处理流程从log文件--》Flume--》HDFS--》Hive(ODS)-->DWD-->DWS-->ADS-->上层应用
06-10
1. log文件采集：使用Flume采集服务器的日志文件，Flume可以通过配置文件来实现日志文件的读取和传输。 2. Flume：将采集到的日志数据发送到HDFS中。Flume提供了多种数据发送方式，包括本地文件、网络传输等。 3. ...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值