flume案例一

1.access.log 搜集到 hdfs 上并按天存储。

a1.sources=s1
a1.channels=c1
a1.sinks=k1
#设置sources
a1.sources.s1.type=spooldir
a1.sources.s1.spooDir=/root/data/flume
a1.sources.s1.includePattern=access[0-9]{4}-[0-9]{2}-[0-9]{2}.log
a1.sources.s1.deserializer=LINE
a1.sources.s1.deserializer.maxLineLength=60000
#设置channel
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
#设置sink
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.fileType=DataStream
a1.sinks.k1.hdfs.filePrefix=access
a1.sinks.k1.hdfs.fileSuffix=.log
a1.sinks.k1.hdfs.path=hdfs://192.168.155.131:9000/flume/%Y-%m-%d
a1.sinks.k1.hdfs.useLocalTimeStamp=true
a1.sinks.k1.hdfs.batchSize=640
a1.sinks.k1.hdfs.rollCount= 0
a1.sinks.k1.hdfs.rollSize=6400000
a1.sinks.k1.hdfs.rollInterval =30
#设置连接
a1.sources.s1.channels=c1
a1.sinks.k1.channel=c1

**

2.监听一个 tcp 端口 41414 将数据打印在控制台

**
events.sources = eventsSource
events.sinks = eventsSink
events.channels = eventsChannel
#设置sink
events.sinks.eventsSink.type=logger
#设置sources
events.sources.eventsSource.type = avro
events.sources.eventsSource.hostname=localhost
events.sources.eventsSource.port=44444
#设置channel
events.channels.eventsChannel.type =file
events.channels.eventsChannel.checkpointDir=/root/data/flumeFile/checkpoint/test
events.channels.eventsChannel.dataDirs=/root/data/flumeFile/data/test

#设置连接
events.sources.eventsSource.channels=eventsChannel
events.sinks.eventsSink.channel =eventsChannel