kafka连接flume把数据分别推到HDFS和SparkStreaming

kafka没有办法多个消费者重复消费同一个topic，所以就在kafka后面挂载flume，然后利用replicating的selector把数据分别发往HDFS做存储和sparkstreaming中实时分析。下面我贴一下flume的配置文件

a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1 c2
a1.sources.r1.selector.type = replicating
# 配置source
a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.batchSize = 800
a1.sources.r1.batchDurationMillis = 2000
a1.sources.r1.kafka.bootstrap.servers = node03:9092,node04:9092,node05:9092
a1.sources.r1.kafka.topics = startup,event
a1.sources.r1.kafka.consumer.group.id = custom.g.id
# ----------------------------------------------------------------------------------
# 配置sink1
#配置HDFS sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://node02:9000/flume/%Y%m%d/%H
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.writeFormat = TEXT
a1.sinks.k1.hdfs.inUsePrefix=_
a1.sinks.k1.hdfs.round = true
a1.sinks.k2.hdfs.roundUnit = minute
a1.sinks.k2.hdfs.roundValue = 5
a1.sinks.k2.hdfs.useLocalTimeStamp = true
#设置输出文件的前缀
a1.sinks.k1.hdfs.filePrefix = logs
#设置输出文件的后缀
a1.sinks.k1.hdfs.fileSuffix = .log
a1.sinks.k1.hdfs.batchSize = 800
a1.sinks.k1.hdfs.rollSize = 134217700
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.rollInterval = 300
a1.sinks.k1.hdfs.useLocalTimeStamp = true
a2.sinks.k1.hdfs.minBlockReplicas = 1
# 配置sink2
# a1.sinks.k2.channel = c1
# a1.sinks.k2.type = org.apache.spark.streaming.flume.sink.SparkSink
# a1.sinks.k2.hostname=192.168.64.1 # 负责汇总数据的服务器的IP地址或主机名
# a1.sinks.k2.port = 8888
# a1.sinks.k2.batchSize= 2000 # 批处理大小 
a1.sinks.k2.type = logger
# ----------------------------------------------------------------------------------
a1.channels.c1.type = memory
a1.channels.c1.capacity = 2000
a1.channels.c1.transactionCapacity = 1000
#
a1.channels.c2.type = memory
a1.channels.c2.capacity = 2000
a1.channels.c2.transactionCapacity = 1000
# ----------------------------------------------------------------------------------
a1.sources.r1.channels = c1 c2
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2

现在sink2暂时用logger输出了，等需要的时候就放开k2把数据输出到实时分析系统去，对了，这里要注意。如果需要把数据保存到HDFS需要在flume的lib文件夹中放入相关的jar.....这些jar的版本我没有具体研究过，基本上和hadoop一个版本应该没有什么问题了吧

commons-configuration-1.6.jar
commons-io-2.4.jar
hadoop-auth-2.7.2.jar
hadoop-common-2.7.2.jar
hadoop-hdfs-2.7.2.jar
htrace-core-3.1.0-incubating.jar

启动flume

flume-ng agent --conf conf/ --name a1 --conf-file jobs/kafka-hdfs_sparkstreaming.config -Dflume.root.logger=INFO,console

输出结果是按年月日划分文件夹，下一级的是按每小时划分。具体多少时间滚动一次，在上面flume中有设置，这里我设置了128M或者每5分钟滚动一次，尽量减少小文件插入HDFS中