使用flume从kafka采集日志

 

1.kafka环境准备

创建topic

bin/kafka-topic.sh --create --topic app_log --partitions 2 --replication-factor 2 --zookeeper localhost:2181

2.部署日志采集器

验证是否成功:

bin/kafka-console-consumer.sh --topic app_log --bootstrap-server doitedu01:9092,doitedu02:9092,doitedu03:9092

3.flume配置文件

a1.sinks.k1.hdfs.path = hdfs://linux01:9000/event_log/%{topic}/%Y-%m-%d/

此处设置为:采集不同topic来源的数据,会根据topic名称生成不同的日志文件夹,当然,前提需要在前面设置sources参数:

a1.sources.r1.setTopicHeader = true
a1.sources.r1.topicHeader = topic

a1.sources = r1
a1.channels = c1
a1.sinks = k1

# source config
a1.sources.r1.channels = c1
a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.kafka.bootstrap.servers = linux01:9092,linux02:9092,linux03:9092
a1.sources.r1.kafka.consumer.group.id = flume
a1.sources.r1.kafka.topics = wxapp_log,app_log
a1.sources.r1.setTopicHeader = true
a1.sources.r1.topicHeader = topic



# channel config
a1.channels.c1.type = file


# sink config
a1.sinks.k1.channel = c1
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://linux01:9000/event_log/%{topic}/%Y-%m-%d/
a1.sinks.k1.hdfs.filePrefix = event_
a1.sinks.k1.hdfs.fileSuffix = .log
a1.sinks.k1.hdfs.rollInterval = 0
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.rollSize = 134217728
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute
a1.sinks.k1.hdfs.fileType = DataStream

4.启动flume采集日志

在后台运行,在浏览器34345端口可查看flume采集情况

bin/flume-ng agent -c conf/ -f agentconf/kafka-hdfs.conf -n a1 -Dflume.monitoring.type=http -Dflume.monitoring.port=34345 1>/dev/null 2>&1 &