Log4j写入日志到Flume

一、log4j配置

依赖包引入

<dependency>
        <groupId>org.apache.flume.flume-ng-clients</groupId>
        <artifactId>flume-ng-log4jappender</artifactId>
        <version>1.6.0</version>
</dependency>

java代码

import org.apache.log4j.Logger;

/**
 * 模拟日志产生
 */
public class LoggerGenerator {

    private static Logger logger = Logger.getLogger(LoggerGenerator.class.getName());

    public static void main(String[] args) throws Exception{

        int index = 0;
        while(true) {
            Thread.sleep(1000);
            logger.info("value : " + index++);
        }
    }
}

配置文件

log4j.rootLogger=INFO,stdout,flume

log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.target = System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} [%t] [%c] [%p] - %m%n


log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = 192.168.62.142
log4j.appender.flume.Port = 41414
log4j.appender.flume.UnsafeMode = true

   具体的配置可以参考flume官网

http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html#log4j-appender

二、flume配置

配置文件

agen1.sources = r1
agen1.sinks = k1
agen1.channels = c1
agent1.sources.r1.type = avro
agent1.sources.r1.bind = 0.0.0.0
agent1.sources.r1.port = 41414

agent1.channels.c1.type = memory
agent1.channels.c1.capacity = 1000
agent1.channels.c1.transactionCapacity = 1000

agent1.sinks.k1.type = logger

# agent1.sinks.k1.channel = c1
# agent1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
# agent1.sinks.k1.topic = streamingtopic
# agent1.sinks.k1.brokerList = mini1:9092
# agent1.sinks.k1.batchSize = 20
# agent1.sinks.k1.requiredAcks = 1

agent1.sources.r1.channels = c1
agent1.sinks.k1.channel = c1

   这里将接收到的数据直接打印出来，如果需要将数据保存至kafka中可以直接把注释的内容作为配置

启动flume

bin/flume-ng agent --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/streaming.conf --name agent1 -Dflume.root.logger=INFO,console &

三、联调测试

   启动java程序：

   flume端数据展示如下:

   最开始项目中启动生成日志时老是报错，错误如下，不知道是什么原因，d按时在其他项目中同样的代码同样的配置都能成功运行，不知道是什么原因，重新导入包也不行。

   flume端的错误信息如下：

   不知道是什么原因引起的，就是无法收集日志信息，目前解决办法就是重新启动一个项目来生成日志，后面如果找到解决办法再来更新。