flume kafka2hdfs demo

做了个简单消费kafka数据到HDFS

这里配置每5分钟写入一个新文件

配置文件名为：test0.conf

在flume conf目录启动
…/bin/flume-ng agent -n a1 -c ./conf -f ./test0.conf -Dflume.root.logger=INFO,console

a1.sources  =  r1 
a1.sinks  =  k1 
a1.channels  =  c1

#sources
a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.channels = c1
a1.sources.r1.batchSize = 5000
a1.sources.r1.batchDurationMillis = 2000
a1.sources.r1.bootstrap.servers = hadoop:9092
a1.sources.r1.topic = testkafka
a1.sources.r1.zookeeperConnect = hadoop:2181
#a1.sources.r1.kafka.consumer.group.id = flume

#channel
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100

#sinks
a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path = /user/flume/%y-%m-%d/
a1.sinks.k1.hdfs.writeFormat=Text
a1.sinks.k1.hdfs.filePrefix = events-
#a1.sinks.k1.hdfs.round = true
#a1.sinks.k1.hdfs.roundValue = 10
#a1.sinks.k1.hdfs.roundUnit = minute
a1.sinks.k1.hdfs.fileType=DataStream
a1.sinks.k1.hdfs.rollInterval=300
a1.sinks.k1.hdfs.rollSize=0
a1.sinks.k1.hdfs.rollCount=0
a1.sinks.k1.hdfs.idleTimeout=0