Flume连接HDFS和Hive

记录一下在CDH中配置flume

Flume连接HDFS

1、进入Flume配置

2、配置flume.conf

 # Name the components on this agent
 a1.sources = r1
 a1.sinks = k1
 a1.channels = c1

 # sources
 a1.sources.r1.type = netcat
 a1.sources.r1.bind = 0.0.0.0
 a1.sources.r1.port = 41414

 # sinks
 a1.sinks.k1.type = hdfs
 a1.sinks.k1.hdfs.path = hdfs://slave1/flume/events/%y-%m-%d/%H%M/%S
 a1.sinks.k1.hdfs.filePrefix = events-
 a1.sinks.k1.hdfs.round = true
 a1.sinks.k1.hdfs.roundValue = 10
 a1.sinks.k1.hdfs.roundUnit = minute
 a1.sinks.k1.hdfs.useLocalTimeStamp=true
 a1.sinks.k1.hdfs.batchSize = 10
 a1.sinks.k1.hdfs.fileType = DataStream

 # channels
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100

 # Bind the source and sink to the channel
 a1.sources.r1.channels = c1
 a1.sinks.k1.channel = c1

3、测试telnet通信

telnet slave1 41414