Flume监听kafka文件源写入hdfs

环境准备

1，启动hdfs，zookeeper

2，kafka创建topic，并生产数据：

bin/kafka-topics.sh --create --topic app_log --partitions 2 --replication-factor 2 --zookeeper linux01:2181

bin/kafka-console-producer.sh --broker-list linux01:9092 --topic app_log

flume配置文件

配置文件路径在flume安装包下面myconfig/kafka2hdfs.conf

a1.sources = r1
a1.channels = c1
a1.sinks = k1

# source config
a1.sources.r1.channels = c1
a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.kafka.bootstrap.servers = linux01:9092,linux02:9092,linux03:9092
a1.sources.r1.kafka.consumer.group.id = doit
a1.sources.r1.kafka.topics = wx_log,app_log
a1.sources.r1.setTopicHeader = true
a1.sources.r1.topicHeader = topic

# channel config
a1.channels.c1.type = file

# sink config
a1.sinks.k1.channel = c1
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://linux01:8020/event_log/%{topic}/%Y-%m-%d/
a1.sinks.k1.hdfs.filePrefix = event_
a1.sinks.k1.hdfs.fileSuffix = .log
a1.sinks.k1.hdfs.rollInterval = 0
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.rollSize = 134217728
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute
a1.sinks.k1.hdfs.fileType = DataStream

指定配置文件，后台启动flume，并开启监听：

nohup bin/flume-ng agent -c conf/ -f myconfig/kafka2hdfs.conf -n a1 -Dflume.root.logger=DEBUG,console -Dflume.monitoring.type=http -Dflume.monitoring.port=34345 1>/dev/null 2>&1 &

监听地址：linux01:34345

查看结果：