log4j+flume+kafka模拟Spark Streaming流式处理数据
1. java 编程模拟日志产生
/**
* 模拟Logger 产生日志
*/
public class LoggerGenerator {
private static Logger logger = Logger.getLogger(LoggerGenerator.class.getName());
public static void main(String[] args) throws Exception{
int index = 0;
while(true) {
Thread.sleep(1000);
logger.info("value : " + index++);
}
}
}
2. log4j.properties文件配置
log4j.rootLogger=INFO,stdout,flume
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.target = System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} [%t] [%c] [%p] - %m%n
3. 接下来使用log4j将日志输出到flume中,使用Log4J Appender
1)在log4j.properies文件中添加
log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = 192.168.126.31
log4j.appender.flume.Port = 41414
log4j.appender.flume.UnsafeMode = true
2)需要添加jar包
Appends Log4j events to a flume agent’s avro source. A client using this appender must have the flume-ng-sdk in the classpath (eg, flume-ng-sdk-1.6.0.jar).
需要:flume-ng-sdk.jar
<dependency>
<groupId>org.apache.flume.flume-ng-clients</groupId>
<artifactId>flume-ng-log4jappender</artifactId>
<version>1.6.0</version>
</dependency>
flume接收数据agent,resource需要avro
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>${spark_version}</version>
</dependency>
<!-- sparkstreaming kafka-->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
<version>${spark_version}</version>
</dependency>
<dependency>
<groupId>org.apache.flume.flume-ng-clients</groupId>
<artifactId>flume-ng-log4jappender</artifactId>
<version>1.6.0</version>
</dependency>
</dependencies>
4. log4j_flume.properties 文件配置
log4j_agent.sources = avro_source
log4j_agent.channels = memory_channel
log4j_agent.sinks = kafka_sink
log4j_agent.sources.avro_source.type = avro
log4j_agent.sources.avro_source.bind = 0.0.0.0
log4j_agent.sources.avro_source.port = 41414
log4j_agent.channels.memory_channel.type = memory
log4j_agent.channels.memory_channel.capacity = 10000
log4j_agent.channels.memory_channel.transactionCapacity = 10000
log4j_agent.sinks.kafka_sink.type = org.apache.flume.sink.kafka.KafkaSink
log4j_agent.sinks.kafka_sink.topic = test20
log4j_agent.sinks.kafka_sink.brokerList = 192.168.126.31:9092
log4j_agent.sinks.kafka_sink.requiredAcks = 1
log4j_agent.sinks.kafka_sink.batchSize = 20
log4j_agent.sources.avro_source.channels = memory_channel
log4j_agent.sinks.kafka_sink.channel = memory_channel
5. 启动测试
启动flume
flume-ng agent --name log4j_agent --conf ${FLUME_HOME}/conf --conf-file ${FLUME_HOME}/conf/log4j_flume.properties -Dflume.root.logger=INFO,console
运行产生日志的程序
先启动zookeeper
zkServer.sh start
启动kafka:
kafka-server-start.sh -daemon ${KAFKA_HOME}/config/server.properties
启动kafka消费者,看能否接收到数据
kafka-console-consumer.sh --zookeeper master:2181,slave1:2181,slave2:2181 --topic test20
6. 开发spark Streaming程序接收kafka 消息
object KafkaConsumerMsg {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setMaster("local[2]").setAppName("KafkaConsumerMsg")
val ssc = new StreamingContext(conf, streaming.Seconds(5))
val topicParams = Map("test20" -> 1)
val dstream = KafkaUtils.createStream(ssc, "192.168.121.31:2181,192.168.121.32:2181,192.168.121.33:2181", "testgroup_id", topicParams)
dstream.map(_._2).count().print()
ssc.start()
ssc.awaitTermination()
}
}