log4j+flume+kafka实现日志收集

最新推荐文章于 2024-05-04 00:34:06 发布
置顶 最新推荐文章于 2024-05-04 00:34:06 发布
阅读量1.7k 收藏 4
点赞数
分类专栏： • flume 文章标签： log4j flume kafka
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/jssg_tzw/article/details/78206558
版权

1.Zookeeper集群配置

#hostnameipsoftwarenotes
1apollo.dt.com192.168.56.181zookeeperKafka:
broker.id=181
2artemis.dt.com192.168.56.182zookeeperkafka:
borker.id=182
3uranus.dt.com192.168.56.183zookeeperkafka:
broker.id=183
4pandora.dt.com192.168.56.184zookeeperkafka:
broker.id=184

有关zookeeper详细集群搭建请参考：CentOS安装配置Zookeeper集群

2.Kafka集群配置

#hostnameipsoftwarenotes
1apollo.dt.com192.168.56.181kafkaKafka:
broker.id=181
2artemis.dt.com192.168.56.182kafkakafka:
borker.id=182
3uranus.dt.com192.168.56.183kafkakafka:
broker.id=183
4pandora.dt.com192.168.56.184kafkakafka:
broker.id=184

有关Kafka详细集群搭建请参考：CentOS7.0安装配置Kafka集群](http://blog.csdn.net/jssg_tzw/article/details/73106299)

3.启动zookeer和kafka集群

3.1.启动zookeeper

[root@apollo ~]# zkServer.sh start
[root@artemis ~]# zkServer.sh start
[root@uranus ~]# zkServer.sh start
[root@pandora ~]# zkServer.sh start

3.2.启动kafka集群

[root@apollo ~]# kafka-server-start.sh /opt/kafka/config/server.properties
[root@artemis ~]# kafka-server-start.sh /opt/kafka/config/server.properties
[root@uranus ~]# kafka-server-start.sh /opt/kafka/config/server.properties
[root@pandora ~]# kafka-server-start.sh /opt/kafka/config/server.properties

4.flume的安装与配置

4.1.把flume解压到/opt/flume目录下

4.2.在$FLUME_HOME/conf/目录下新建配置文件flume2kafka

# 定义agent
a1.sources=r1
a1.channels=c1
a1.sinks=k1

# 定义source
a1.sources.r1.type=avro
a1.sources.r1.bind=localhost
a1.sources.r1.port=44446

# 定义channel
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=1000
a1.channels.c1.keep-alive=30

# 定义sink (Kafka)
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.topic = dt-receipts
a1.sinks.k1.kafka.bootstrap.servers = 192.168.56.181:9092,192.168.56.182:9092,192.168.56.183:9092,192.168.56.184:9092
a1.sinks.k1.kafka.flumeBatchSize = 20
a1.sinks.k1.kafka.producer.acks = 1
a1.sinks.k1.kafka.producer.linger.ms = 1
a1.sinks.k1.kafka.producer.compression.type = snappy

# 绑定source, sink到channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

4.3.启动flume

[itlocals-MacBook-Pro:flume david.tian]$  bin/flume-ng agent -n a1 -c conf/ --conf-file conf/flume2kafka.conf -Dflume.root.logger=DEBUG,console

5. log4j发日志到flume

源码请从我的git上下载：https://github.com/david-louis-tian/dBD

5.1.这里仅给出pom.xml,模拟日志的代码,和log4j.properties

  • pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.dvtn.www</groupId>
  <artifactId>dBD</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>dBD</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <slf4j.version>1.7.25</slf4j.version>
    <log4j.version>1.2.17</log4j.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>

    <!-- Log Dependency 日志依赖-->

    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
      <version>${slf4j.version}</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-log4j12</artifactId>
      <version>${slf4j.version}</version>
    </dependency>
    <dependency>
      <groupId>log4j</groupId>
      <artifactId>log4j</artifactId>
      <version>${log4j.version}</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.json/json -->
    <dependency>
      <groupId>org.json</groupId>
      <artifactId>json</artifactId>
      <version>20170516</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.avro/avro -->
    <dependency>
      <groupId>org.apache.avro</groupId>
      <artifactId>avro</artifactId>
      <version>1.8.2</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.flume/flume-ng-core -->
    <dependency>
      <groupId>org.apache.flume</groupId>
      <artifactId>flume-ng-core</artifactId>
      <version>1.7.0</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.flume.flume-ng-clients/flume-ng-log4jappender -->
    <dependency>
      <groupId>org.apache.flume.flume-ng-clients</groupId>
      <artifactId>flume-ng-log4jappender</artifactId>
      <version>1.7.0</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.avro/avro-ipc -->
    <dependency>
      <groupId>org.apache.avro</groupId>
      <artifactId>avro-ipc</artifactId>
      <version>1.8.2</version>
    </dependency>

  </dependencies>
</project>
  • log4j.properties
################### set log levels ###############
log4j.rootLogger = INFO,stdout,file,flume

################### flume ########################
log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.layout = org.apache.log4j.PatternLayout
log4j.appender.flume.Hostname = localhost
log4j.appender.flume.Port = 44446

################## stdout #######################
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Threshold = INFO
log4j.appender.stdout.Target = System.out
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] %m%n

################## file ##########################
log4j.appender.file = org.apache.log4j.DailyRollingFileAppender
log4j.appender.file.Threshold = INFO
log4j.appender.file.File = /Users/david.tian/logs/tracker/tracker.log
log4j.appender.file.Append = true
log4j.appender.file.DatePattern = '.'yyyy-MM-dd
log4j.appender.file.layout = org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] %m%n
  • SendReceipts.java
package com.dvtn.www.log4j.jsonlog;

import com.dvtn.www.log4j.logfile.LogProducer;
import com.dvtn.www.model.Area;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericRecord;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.*;
import java.util.*;

/**
 * Created by david.tian on 08/09/2017.
 */
public class SendReceipts {
    private static Logger LOG = LoggerFactory.getLogger(LogProducer.class);
    private static String path = SendReceipts.class.getResource("/").getPath();
    private static String areaJsonString;
    private static String city;
    private static String cityKey;
    private static String province;
    private static String provinceKey;
    private static int separator;

    private static String phonePrefix;
    //private static final Random rnd = new Random();

    private static String[] payers = {"Merchants", "Individuals"};
    private static String[] managers = {"david", "josen", "fab", "simon", "banana", "tom", "scott", "ekrn", "sunshine", "lily", "kudu", "scala", "spark", "flume", "storm", "kafka", "avro", "linux"};
    private static String[] terminalTypes = {"RNM", "CNM", "RNM", "GNM", "CNJ", "GNJ", "RNJ", "GNM", "CNM"};
    private static String[] stores = {"连锁店", "分营店", "工厂店", "会员店", "直销店"};
    private static String[] items = {"面包","酒","油","牛奶","蔬菜","猪肉","牛肉","羊肉","曲奇","手机","耳机","面粉","大米","糖","苹果","茶叶","书","植物","玩具","床","锅","牙膏","洗衣粉","酱油","金鱼","干货"};
    private static String[] itemsType ={"食物","酒水","饮料","日用品","电子","数码","娱乐","家俱"};


    public static void main(String[] args) {

        Timer timer = new Timer();
        timer.schedule(new TimerTask() {
            @Override
            public void run() {

                Random rnd = new Random();

                ProduceReceipts pr = new ProduceReceipts();
                areaJsonString = pr.readJSON(path, "area.json");

                String transactionID = System.currentTimeMillis() + ""+Math.round(Math.random() * 9000 + 1000);
                String transactionDate = System.currentTimeMillis() + "";
                String taxNumber = Math.round(Math.random() * 9000 + 1000) + " " + Math.round(Math.random() * 9000 + 1000) + " " + Math.round(Math.random() * 9000 + 1000) + " " + Math.round(Math.random() * 9000 + 1000);
                String invoiceId = System.currentTimeMillis() + "";
                String invoiceNumber = Math.round(Math.random() * 900000000 + 100000000) + "";
                String invoiceDate = System.currentTimeMillis() + "";
                List<Area> listArea = pr.listArea(areaJsonString);
                int idx = rnd.nextInt(listArea.size());
                String provinceID = listArea.get(idx).getProvinceID();
                String provinceName = listArea.get(idx).getProvinceName();
                String cityID = listArea.get(idx).getCityID();
                String cityName = listArea.get(idx).getCityName();
                String telephone = provinceID + "-" + Math.round(Math.random() * 9000 + 1000) + " " + Math.round(Math.random() * 9000 + 1000);
                int managerSize = managers.length;
                String manger = managers[rnd.nextInt(managerSize)];
                int payerSize = payers.length;
                String payer = payers[rnd.nextInt(payerSize)];
                String operator = "OP" + Math.round(Math.random() * 90000 + 10000);
                int terminalTypeSize = terminalTypes.length;
                String terminalNumber = terminalTypes[rnd.nextInt(terminalTypeSize)] + Math.round(Math.random() * 90000 + 10000);
                String account = pr.StringReplaceWithStar(Math.round(Math.random() * 9000 + 1000) + " " + Math.round(Math.random() * 9000 + 1000) + " " + Math.round(Math.random() * 9000 + 1000) + " " + Math.round(Math.random() * 9000 + 1000));
                String tcNumber = Math.round(Math.random() * 9000 + 1000) + " " + Math.round(Math.random() * 9000 + 1000) + " " + Math.round(Math.random() * 9000 + 1000) + " " + Math.round(Math.random() * 9000 + 1000) + "";


                File file = new File(path + "receipts.avsc");

                String line = null;

                BufferedReader reader = null;
                try {
                    reader = new BufferedReader(new FileReader(file));

                    while ((line = reader.readLine()) != null) {
                        // System.out.println("========>" + line);
                    }
                    reader.close();
                } catch (IOException e) {
                    e.printStackTrace();
                } finally {
                    if (reader != null) {
                        try {
                            reader.close();
                        } catch (IOException e1) {
                        }
                    }
                }
                try {
                    //获得整个Schema
                    Schema schema = new Schema.Parser().parse(new File(path + "receipts.avsc"));

                    GenericRecord record = new GenericData.Record(schema);

                    //获取schema中的字段


                    int storesSize = stores.length;

                    //获取店面的Schema
                    Schema.Field  storeField = schema.getField("store");
                    Schema storeSchema =  storeField.schema();
                    GenericRecord storeRecord = new GenericData.Record(storeSchema);

                    String storeNumber = Math.round(Math.random() * 9000 + 1000) + "";
                    String address = provinceName + cityName;
                    String storeName = provinceName + cityName + stores[rnd.nextInt(storesSize)];

                    storeRecord.put("store_number",storeNumber);
                    storeRecord.put("store_name",storeName);
                    storeRecord.put("address",address);


                    int itemsSize = items.length;
                    int itemsTypeSize = itemsType.length;

                    List<GenericRecord> productRecordList = new ArrayList<GenericRecord>();
                    //获取product的schema
                    Schema.Field productField = schema.getField("products");
                    Schema productSchema = productField.schema();





                    for (int i=0; i< 10; i++){
                        String itemName = items[rnd.nextInt(1000)%itemsSize];
                        String itemType = itemsType[rnd.nextInt(1000)%itemsTypeSize];
                        String quantity = String.valueOf(rnd.nextInt(100));
                        String price = String.valueOf(rnd.nextFloat()*100);
                        String discount = String.valueOf(rnd.nextFloat());


                        GenericRecord productRecord = new GenericData.Record(productSchema);

                        productRecord.put("item",itemName);
                        productRecord.put("item_type",itemType);
                        productRecord.put("quantity",quantity);
                        productRecord.put("price",price);
                        productRecord.put("discount",discount);
                        productRecordList.add(productRecord);
                    }


                    record.put("transaction_id",transactionID);
                    record.put("transaction_date",transactionDate);
                    record.put("invoice_id",invoiceId);
                    record.put("invoice_number",invoiceNumber);
                    record.put("telephone",telephone);
                    record.put("payer",payer);
                    record.put("store",storeRecord);
                    record.put("operator",operator);
                    record.put("terminal_number",terminalNumber);
                    record.put("products",productRecordList);
                    record.put("account",account);
                    record.put("tc_number",terminalNumber);

                    LOG.info(record.toString());

                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }, 0, 1000);
    }
}

6.验证

在kafka机器上执行命令kafka-console-consumer.sh读取topic “dt-receipts”中看是否日志已被kafka收集：

[root@apollo ~]# kafka-console-consumer.sh --bootstrap-server 192.168.56.181:9092,192.168.56.182:9092,192.168.56.183:9092,192.168.56.184:9092 --from-beginning --topic dt-receipts

我们可以看到，数据已经收集到kafka里的dt-receipts的topic里面：
flume-kafka.png

mrdavidtian
关注
  • 0
    点赞
  • 踩
  • 4
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Flume + kafka + log4j构建日志采集系统
08-30
Flume + kafka + log4j构建日志采集系统，附实例及文档。
springboot_log4j2_flume
02-23
Log4j直接发送数据到Flume + Kafka (方式一) 通过flume收集系统日记, 收集的方式通常采用以下. 系统logs直接发送给flume系统, 本文主要记录种方式进行说明. 文章链接,请看:...
基于Flume+LOG4J+Kafka的日志采集架构方案
weixin_30537451的博客
05-08 322
本文将会介绍如何使用 Flume、log4j、Kafka进行规范的日志采集。 Flume 基本概念 Flume是一个完善、强大的日志采集工具，关于它的配置，在网上有很多现成的例子和资料，这里仅做简单说明不再详细赘述。Flume包含Source、Channel、Sink三个最基本的概念： Source——日志来源，其中包括：Avro Source、Thrift Source、...
SpringBoot + Kafka + ELK 完成海量日志收集（超详细）
最新发布
2401_84153193的博客
05-04 653
《一线大厂Java面试题解析+核心总结学习笔记+最新讲解视频+实战项目源码》，点击传送门，即可获取！public class NetUtil {public static String normalizeAddress(String address){String[] blocks = address.split(“[:]”);if(blocks.length > 2){throw new IllegalArgumentException(address + " is invalid");}String h
log4j+flume+kafka+strom整合
liuxiangke0210的专栏
12-31 1901
log4j+flume+kafka+strom整合   整合流程图。 第一步  整合log4j和flume（主机名为hadoop0） (1) 安装flume在/usr/softinstall/下 (2)在/usr/softinstall/flume/conf 目录下建立文件lfks.conf a1.cha
log4j+flume+kafka管理日志，查询日志
zhouyu7373的专栏
09-01 1443
由于系统每天生成日志文件非常庞大所以做了这个日志分类
【十六】Log4j+Flume+Kafka+Spark Streaming（使用Scala语言）
Sid小杰的博客
07-19 521
项目目录 pom.xml &lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apach...
基于spark+flume+kafka+hbase的实时日志处理分析系统.zip
12-24
在这个项目中，Flume 收集的日志数据被发送到 Kafka，作为数据缓冲区。Kafka 提供了高可用性和可扩展性，使得系统可以处理大量实时数据流，同时保证数据不丢失。 3. **Apache Spark**：Spark 是一个用于大规模数据...
Real-time-log-analysis-system:基于spark stream + flume + kafka + hbase的实时日志处理分析系统（分为控制台版本和基于springboot，Echarts等的Web UI可视化版本）
03-11
实时日志分析系统基于spark stream + flume + kafka + hbase的实时日志处理分析系统分为控制台版本（主要由scala完成）和基于springboot，Echarts等的Web UI可视化版本（主要由java完成）
flume+kafka+sparkStream+redis实时日志采集.docx
12-20
在构建实时日志采集系统时，常常会使用到Apache Flume、Apache Kafka、Apache Spark Streaming以及Redis等组件。以下是对这些技术的详细说明： **Apache Flume** 是一个分布式、可靠且可用于有效收集、聚合和移动...
Log4j直接发送数据到Flume + Kafka (方式二)
有所思的专栏
02-24 1024
该文是接着上篇 Log4j直接发送数据到Flume + Kafka (方式一) 所提到的问题进行展开记录的. 主要方式就是通过继承log4j的Appenders, 采用flume rpc动态连接flume服务去发送logs信息. flume sdk包: &lt;dependency&gt; &lt;groupId&gt;org.apache.flume&lt;/groupId&gt; ...
logback日志记录写入kafka
03-02
使用slf4j配置kafkaAppender写入日志到kafka列队 支持日志解析+过滤等扩展接口
flume+Kafka采集log4j数据
ylqdh的博客
11-22 173
简介：测试flume+Kafka集成来采集log4j产生的日志 环境： Flume 1.6.0 Kafka 0.11.0.2 jdk 1.8.0_121 Kafka集群、flume安装服务器上ubuntu16.04.5上 一、log4j生成数据 在window上，编写Java程序，模拟日志生成 import org.apache.log4j.Logger; /* 模拟日志生成 */ ...
基于Flume+Log4j+Kafka的日志采集架构方案（上）
weixin_34221112的博客
01-15 101
2019独角兽企业重金招聘Python工程师标准>>> ...
对接kafka_flume对接kafka多路径同时收集日志，配置怎么写？
weixin_33488806的博客
01-07 134
flume安装目录的conf下配置文件如下：# define#比如我们有8个路径要收集a1.sources = r1 r2 r3 r4 r5 r6 r7 r8a1.sinks = k6a1.channels = c6# sourcea1.sources.r1.type = execa1.sources.r2.type = execa1.sources.r3.type = execa1.source...
flume从log4j收集日志输出到kafka
anxia5150的博客
10-13 301
1. flume安装 (1)下载：wget http://archive.cloudera.com/cdh5/cdh/5/flume-ng-1.6.0-cdh5.7.1.tar.gz (2)解压：tar zxvf flume-ng-1.6.0-cdh5.7.1.tar.gz (3)环境变量： export FLUME_HOME=/xxx/soft/apache-flume-1...
SpringBoot+Kafka+ELK 完成海量日志收集
weixin_42073629的博客
09-04 125
整体流程大概如下： 服务器准备 在这先列出各服务器节点，方便同学们在下文中对照节点查看相应内容 图片 推荐下自己做的 Spring Boot 的实战项目： https://github.com/YunaiV/ruoyi-vue-pro SpringBoot项目准备 引入log4j2替换SpringBoot默认log，demo项目结构如下： 图片 pom <dependencies> <dependency> <gro...
Flume整合log4j和Kafka
lql_h的博客
05-15 217
log4j配置 配置文件 修改log4j.properties # 设置要进行数据跟踪程序所在的名称 log4j.logger.cn.mldn.myflume=INFO ,flume log4j.appender.flume=org.apache.flume.clients.log4jappender.Log4jAppender log4j.appender.flume.layout...
Log4j直接发送数据到Flume + Kafka (方式一)
有所思的专栏
02-24 977
通过flume收集系统日记, 收集的方式通常采用以下两种: flume以监听文件的方式进行收集, 对系统程序代码无入侵. 业务程序的logs直接发送给flume服务系统, 本文针对这种方式进行说明.   主要准备工作: Linux系统一台(可以使用虚拟机), 安装flume, zookeeper, kafka java环境开发机   flume的配置 flume的安装使用...
Flume+Kafka+Hbase集成配置
05-14
Flume、Kafka和HBase都是大数据领域常用的组件，它们可以很好地协同工作来实现数据的实时采集、传输和存储。下面是它们的集成配置。 1. 安装Flume Flume是Apache基金会下的分布式、可靠、高可用的海量日志采集、...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值