log4j+flume+kafka模拟Spark Streaming流式处理数据

log4j+flume+kafka模拟Spark Streaming流式处理数据

1. java 编程模拟日志产生

/**
 * 模拟Logger 产生日志
 */
public class LoggerGenerator {

    private static Logger logger = Logger.getLogger(LoggerGenerator.class.getName());
    
    public static void main(String[] args) throws Exception{
        int index = 0;
        while(true) {
            Thread.sleep(1000);
            logger.info("value : " + index++);
        }
    }
}

2. log4j.properties文件配置

log4j.rootLogger=INFO,stdout,flume
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.target = System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} [%t] [%c] [%p] - %m%n

3. 接下来使用log4j将日志输出到flume中，使用Log4J Appender

1）在log4j.properies文件中添加

log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = 192.168.126.31
log4j.appender.flume.Port = 41414
log4j.appender.flume.UnsafeMode = true

2）需要添加jar包

Appends Log4j events to a flume agent’s avro source. A client using this appender must have the flume-ng-sdk in the classpath (eg, flume-ng-sdk-1.6.0.jar).

需要：flume-ng-sdk.jar

    <dependency>
        <groupId>org.apache.flume.flume-ng-clients</groupId>
        <artifactId>flume-ng-log4jappender</artifactId>
        <version>1.6.0</version>
    </dependency>

flume接收数据agent，resource需要avro

 <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.11</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>${spark_version}</version>
        </dependency>

        <!-- sparkstreaming kafka-->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
            <version>${spark_version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flume.flume-ng-clients</groupId>
            <artifactId>flume-ng-log4jappender</artifactId>
            <version>1.6.0</version>
        </dependency>

    </dependencies>

4. log4j_flume.properties 文件配置

log4j_agent.sources = avro_source
log4j_agent.channels = memory_channel
log4j_agent.sinks = kafka_sink

log4j_agent.sources.avro_source.type = avro
log4j_agent.sources.avro_source.bind = 0.0.0.0
log4j_agent.sources.avro_source.port = 41414

log4j_agent.channels.memory_channel.type = memory
log4j_agent.channels.memory_channel.capacity = 10000
log4j_agent.channels.memory_channel.transactionCapacity = 10000

log4j_agent.sinks.kafka_sink.type = org.apache.flume.sink.kafka.KafkaSink
log4j_agent.sinks.kafka_sink.topic = test20
log4j_agent.sinks.kafka_sink.brokerList = 192.168.126.31:9092
log4j_agent.sinks.kafka_sink.requiredAcks = 1
log4j_agent.sinks.kafka_sink.batchSize = 20

log4j_agent.sources.avro_source.channels = memory_channel
log4j_agent.sinks.kafka_sink.channel = memory_channel

5. 启动测试

启动flume

flume-ng agent --name log4j_agent --conf ${FLUME_HOME}/conf --conf-file ${FLUME_HOME}/conf/log4j_flume.properties -Dflume.root.logger=INFO,console

运行产生日志的程序

先启动zookeeper

zkServer.sh start

启动kafka：

kafka-server-start.sh -daemon ${KAFKA_HOME}/config/server.properties

启动kafka消费者，看能否接收到数据

kafka-console-consumer.sh --zookeeper master:2181,slave1:2181,slave2:2181 --topic test20

6. 开发spark Streaming程序接收kafka 消息

object KafkaConsumerMsg {

    def main(args: Array[String]): Unit = {
        val conf = new SparkConf().setMaster("local[2]").setAppName("KafkaConsumerMsg")
        val ssc = new StreamingContext(conf, streaming.Seconds(5))
        val topicParams = Map("test20" -> 1)
        val dstream = KafkaUtils.createStream(ssc, "192.168.121.31:2181,192.168.121.32:2181,192.168.121.33:2181", "testgroup_id", topicParams)
        dstream.map(_._2).count().print()
        ssc.start()
        ssc.awaitTermination()
    }
}

7. 源码

https://github.com/zhmcode/SsfklProject