一、Flume avro架构模型
二、avro配置
(1) agent foo 配置(配置文件名:log2Avrosink.conf )
#Name the components on this agent
a1.sources=srcLog
a1.channels=chMemory
a1.sinks=avroSink
#source配置
a1.sources.srcLog.type = TAILDIR (具有断点还原功能,生产中建议使用)
a1.sources.srcLog.positionFile = /root/lb/flume-ng/taildir_position.json
a1.sources.srcLog.filegroups = f1
a1.sources.srcLog.filegroups.f1 = /root/lb/test/.*log
#channels配置
a1.channels.chMemory.type = memory
a1.channels.chMemory.capacity=1000
a1.channels.chMemory.transactionCapacity = 100
#sink配置
a1.sinks.avroSink.type = avro
a1.sinks.avroSink.hostname = 192.168.11.33 (avro sink对接avro source的地址,一定要根据自己环境更改。也可用主机名)
a1.sinks.avroSink.port = 4141
# Bind the source and sink to the channel
a1.sources.srcLog.channels = chMemory
a1.sinks.avroSink.channel = chMemory
agent运行命令:
/opt/app/apache-flume-1.8.0-bin/bin/flume-ng agent --name a1 --conf /opt/app/apache-flume-1.8.0-bin/conf --conf-file /opt/app/apache-flume-1.8.0-bin/conf/log2Avrosink.conf -
Dflume.root.logger=INFO,console
(2) agent bar 配置(配置文件名:AvroSource2HDFS.conf )
#Name the components on this agent
a1.sources=srcAvro
a1.channels=chMemory
a1.sinks=hdfsSink
#source配置
a1.sources.srcAvro.type = avro
a1.sources.srcAvro.bind = 0.0.0.0 (相当于服务器绑定一个本机地址,0.0.0.0表示自动绑定本机地址)
a1.sources.srcAvro.port = 4141 (需要与对端保持一致)
#channels配置
a1.channels.chMemory.type = memory
a1.channels.chMemory.capacity=1000
a1.channels.chMemory.transactionCapacity = 100
#sink配置
a1.sinks.hdfsSink.type = hdfs
a1.sinks.hdfsSink.hdfs.path = /tzt/test/%Y%m%d/%d/%H
a1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true (由于使用了时间配置符%Y%m%d/%d/%H,不配置会报错)
a1.sinks.hdfsSink.hdfs.codeC = lzop (使用lzo压缩。注意lzo压缩生成文件后缀为.lzo_deflate ,lzop生成的后缀才是.lzo)
a1.sinks.hdfsSink.hdfs.fileType=CompressedStream
a1.sinks.hdfsSink.hdfs.writeFormat=Text
a1.sinks.hdfsSink.hdfs.rollInterval = 86400
a1.sinks.hdfsSink.hdfs.rollSize = 134217728
a1.sinks.hdfsSink.hdfs.rollCount = 0
a1.sinks.hdfsSink.hdfs.batchSize = 100
a1.sinks.hdfsSink.hdfs.threadsPoolSize=10
# Bind the source and sink to the channel
a1.sources.srcAvro.channels = chMemory
a1.sinks.hdfsSink.channel = chMemory
agent运行命令:
/opt/app/apache-flume-1.8.0-bin/bin/flume-ng agent --name a1 --conf /opt/app/apache-flume-1.8.0-bin/conf --conf-file /opt/app/apache-flume-1.8.0-bin/conf/AvroSource2HDFS.conf -
Dflume.root.logger=INFO,console