cdh5.16.2配置flume读取kafka文件到hdfs

环境

• kafka_2.13-2.4.1
• cdh5.16.2

flume配置

按需选择

```java
#comonents
a1.sources = r1
a1.sinks = k1
a1.channels = c1
 
#source
#a1.sources.r1.type = netcat
#a1.sources.r1.bind = 127.0.0.1
#a1.sources.r1.port = 8888

a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.batchSize = 1000
a1.sources.r1.batchDurationMillis = 2000
a1.sources.r1.kafka.bootstrap.servers = 192.168.97.212:9093
a1.sources.r1.kafka.topics = adTo
a1.sources.r1.kafka.consumer.group.id = flume-consumer
 
#sink
a1.sinks = k1
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /tmp/flume/%Y%m%d
#上传文件的前缀
a1.sinks.k1.hdfs.filePrefix = ad%H%M
#是否按照时间滚动文件夹
a1.sinks.k1.hdfs.round = true
#多少时间单位创建一个新的文件夹
a1.sinks.k1.hdfs.roundValue = 1
#重新定义时间单位
a1.sinks.k1.hdfs.roundUnit = hour
#是否使用本地时间戳
a1.sinks.k1.hdfs.useLocalTimeStamp = true
#积攒多少个 Event 才 flush 到 HDFS 一次
a1.sinks.k1.hdfs.batchSize = 1000
#设置文件类型，可支持压缩
a1.sinks.k1.hdfs.fileType = DataStream
#多久生成一个新的文件
a1.sinks.k1.hdfs.rollInterval = 3600
#设置每个文件的滚动大小
a1.sinks.k1.hdfs.rollSize = 134217700
#文件的滚动与 Event 数量无关
a1.sinks.k1.hdfs.rollCount = 0
#最小冗余数
a1.sinks.k1.hdfs.minBlockReplicas = 1

#channel
#a1.channels.c1.type=memory
a1.channels.c1.type=file
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.checkpointDir = ./flume/checkpoint
a1.channels.c1.dataDirs = ./flume/data
 
#bind
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1

启动

启动查看日志（目录：/var/log/flume-ng）无错误即可。
监控页面：