Flume Avro配置

一、Flume avro架构模型

二、avro配置

(1) agent foo 配置(配置文件名：log2Avrosink.conf )

#Name the components on this agent
a1.sources=srcLog
a1.channels=chMemory
a1.sinks=avroSink
#source配置
a1.sources.srcLog.type = TAILDIR        (具有断点还原功能，生产中建议使用)
a1.sources.srcLog.positionFile = /root/lb/flume-ng/taildir_position.json
a1.sources.srcLog.filegroups = f1
a1.sources.srcLog.filegroups.f1 = /root/lb/test/.*log
#channels配置
a1.channels.chMemory.type = memory
a1.channels.chMemory.capacity=1000
a1.channels.chMemory.transactionCapacity = 100
#sink配置
a1.sinks.avroSink.type = avro
a1.sinks.avroSink.hostname = 192.168.11.33    (avro sink对接avro source的地址，一定要根据自己环境更改。也可用主机名)
a1.sinks.avroSink.port = 4141      
# Bind the source and sink to the channel
a1.sources.srcLog.channels = chMemory
a1.sinks.avroSink.channel = chMemory

agent运行命令：

/opt/app/apache-flume-1.8.0-bin/bin/flume-ng agent --name a1 --conf /opt/app/apache-flume-1.8.0-bin/conf --conf-file /opt/app/apache-flume-1.8.0-bin/conf/log2Avrosink.conf - Dflume.root.logger=INFO,console

(2) agent bar 配置(配置文件名：AvroSource2HDFS.conf )

#Name the components on this agent
a1.sources=srcAvro
a1.channels=chMemory
a1.sinks=hdfsSink

#source配置
a1.sources.srcAvro.type = avro
a1.sources.srcAvro.bind = 0.0.0.0   (相当于服务器绑定一个本机地址，0.0.0.0表示自动绑定本机地址)
a1.sources.srcAvro.port = 4141      (需要与对端保持一致)

#channels配置
a1.channels.chMemory.type = memory
a1.channels.chMemory.capacity=1000
a1.channels.chMemory.transactionCapacity = 100

#sink配置
a1.sinks.hdfsSink.type = hdfs
a1.sinks.hdfsSink.hdfs.path = /tzt/test/%Y%m%d/%d/%H
a1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true    (由于使用了时间配置符%Y%m%d/%d/%H，不配置会报错)
a1.sinks.hdfsSink.hdfs.codeC = lzop        (使用lzo压缩。注意lzo压缩生成文件后缀为.lzo_deflate ，lzop生成的后缀才是.lzo)
a1.sinks.hdfsSink.hdfs.fileType=CompressedStream
a1.sinks.hdfsSink.hdfs.writeFormat=Text
a1.sinks.hdfsSink.hdfs.rollInterval = 86400
a1.sinks.hdfsSink.hdfs.rollSize = 134217728
a1.sinks.hdfsSink.hdfs.rollCount = 0
a1.sinks.hdfsSink.hdfs.batchSize = 100
a1.sinks.hdfsSink.hdfs.threadsPoolSize=10
# Bind the source and sink to the channel
a1.sources.srcAvro.channels = chMemory
a1.sinks.hdfsSink.channel = chMemory

agent运行命令：

/opt/app/apache-flume-1.8.0-bin/bin/flume-ng agent --name a1 --conf /opt/app/apache-flume-1.8.0-bin/conf --conf-file /opt/app/apache-flume-1.8.0-bin/conf/AvroSource2HDFS.conf - Dflume.root.logger=INFO,console