flume配置记录

flume配置记录

标签（空格分隔）： flume

---

avro写入hdfs

avro传递的字符串数据直接写入hdfs

agent1.sources=source1
agent1.channels=channel1
agent1.sinks=sink1

agent1.sources.source1.type=avro
agent1.sources.source1.bind=0.0.0.0
agent1.sources.source1.port=44444
agent1.sources.source1.channels=channel1

agent1.channels.channel1.type=memory
agent1.channels.channel1.capacity=10000
agent1.channels.channel1.transactionCapacity=1000
agent1.channels.channel1.keep-alive=30

agent1.sinks.sink1.type=hdfs
agent1.sinks.sink1.channel=channel1
agent1.sinks.sink1.hdfs.path=hdfs://dashuju174:9000/flume/%{application}/%{dir}/%Y%m%d
agent1.sinks.sink1.hdfs.fileType=DataStream
agent1.sinks.sink1.hdfs.writeFormat=Text
agent1.sinks.sink1.hdfs.round=true
agent1.sinks.sink1.hdfs.rollInterval=0
agent1.sinks.sink1.hdfs.rollSize=10240
agent1.sinks.sink1.hdfs.rollCount=0

kafka写入hbase

# read from kafka and write to hbase
bft-agent.sources = bft-source
bft-agent.channels = bft-channel-detail
bft-agent.sinks = bft-sink-detail

# source
bft-agent.sources.bft-source.type=avro
bft-agent.sources.bft-source.bind=0.0.0.0
bft-agent.sources.bft-source.port=44443
#bft-agent.sources.bft-source.selector.type = replicating

# detail-channel
bft-agent.channels.bft-channel-detail.type = org.apache.flume.channel.kafka.KafkaChannel
bft-agent.channels.bft-channel-detail.kafka.bootstrap.servers = ceshi185:19092,ceshi186:19092,ceshi185:19092
bft-agent.channels.bft-channel-detail.kafka.topic = flume_wp_detail_channel
bft-agent.channels.bft-channel-detail.kafka.consumer.group.id = flume_wp_detail_channel

# detail-sink
bft-agent.sinks.bft-sink-detail.type = asynchbase
bft-agent.sinks.bft-sink-detail.table = t_invoice_ticket_detail_log
bft-agent.sinks.bft-sink-detail.columnFamily = i
bft-agent.sinks.bft-sink-detail.serializer = com.bwjf.flume.invoice.bft.sink.InvoiceDetailHbaseSerializer

# assemble
bft-agent.sources.bft-source.channels = bft-channel-detail
bft-agent.sinks.bft-sink-detail.channel = bft-channel-detail

单个source、channel，多个sink

这种方式写多个sink对消费同一个channel，source传递来的数据只会在某一个sink中执行，执行成功后另外的sink不会重复执行

# read from kafka and write to hbase
bft-agent.sources = bft-source
bft-agent.channels = bft-channel
bft-agent.sinks = bft-sink bft-sink-detail

# source
bft-agent.sources.bft-source.type=avro
bft-agent.sources.bft-source.bind=0.0.0.0
bft-agent.sources.bft-source.port=44443

# channel
bft-agent.channels.bft-channel.type = org.apache.flume.channel.kafka.KafkaChannel
bft-agent.channels.bft-channel.kafka.bootstrap.servers = ceshi185:19092,ceshi186:19092,ceshi185:19092
bft-agent.channels.bft-channel.kafka.topic = flume_wp_channel
bft-agent.channels.bft-channel.kafka.consumer.group.id = flume_wp_channel

# sink
bft-agent.sinks.bft-sink.type = asynchbase
bft-agent.sinks.bft-sink.table = t_invoice_ticket_log
bft-agent.sinks.bft-sink.columnFamily = i
bft-agent.sinks.bft-sink.serializer = com.bwjf.flume.invoice.bft.sink.BftInvoiceHbaseSerializer

#detail-channel
bft-agent.sinks.bft-sink-detail.type = asynchbase
bft-agent.sinks.bft-sink-detail.table = t_invoice_ticket_detail_log
bft-agent.sinks.bft-sink-detail.columnFamily = i
bft-agent.sinks.bft-sink-detail.serializer = com.bwjf.flume.invoice.bft.sink.BftInvoiceDetailHbaseSerializer

# assemble
bft-agent.sources.bft-source.channels = bft-channel
bft-agent.sinks.bft-sink.channel = bft-channel
bft-agent.sinks.bft-sink-detail.channel = bft-channel

单个source，复制channel、sink