第4.1.2章 flume的拓扑结构_双层flume-CSDN博客

本文链接：https://blog.csdn.net/warrah/article/details/78780260

1 多个应用通过连接同一个Flume Agent
参考logback与flume集成章节的描述，很容想到各个系统使用同一个logback和flume的配置文件，轻松的实现了将日志数据写入HDFS中的效果。

这种设计的问题也很凸显，单点故障。
2 Flume Agent集群
很容易就想到了，使用haproxy做代理

haproxy配置也较为简单

frontend flume_hdfs_front
		bind *:9095
		mode tcp
    log global
    option tcplog
    timeout client 3600s
    backlog 4096
    maxconn 1000000
    default_backend flume_hdfs_back

backend flume_hdfs_back
		mode tcp
		option log-health-checks
   	option redispatch
   	option tcplog
   	balance roundrobin
   	timeout connect 1s
   	timeout queue 5s
   	timeout server 3600s
		balance roundrobin
		server f1 192.168.5.174:44444 check inter 2000 rise 3 fall 3 weight 1 
		server f2 192.168.5.173:44444 check inter 2000 rise 3 fall 3 weight 1

我设置的一个文件大小是10M，当我将其中一台机器174停掉后，我发现每台flume会生成自己的文件，而不是在原来的文件中追加。

当我再把174启动，把173停掉时，一条新的记录又产生了。

方案2中，Flume Agent直连HDFS的方式，当多个Flume Agent写入HDFS的时候，HDFS的namenode就会产生巨大的压力。
3 Flume根据不同的业务写入不同的数据库

这里还是将flume作为hbase的前置节点，探索flume到一定程度，就不用重复造轮子了，前人走过的路，跟着摔跟头是不明智的，flume高并发优化——（2）精简结构，以及 flume高并发优化——（4）kafka channel
我们公司的flume拓扑结构还需要精简
4 使用kafka channel，写入hbase

这种方式写多个sink对消费同一个channel，source传递来的数据只会在某一个sink中执行，执行成功后另外的sink不会重复执行.

# read from kafka and write to hbase
dzm-agent.sources = dzm-source
dzm-agent.channels = dzm-channel dzm-channel-detail
dzm-agent.sinks = dzm-sink dzm-sink-detail

# source
dzm-agent.sources.dzm-source.type=avro
dzm-agent.sources.dzm-source.bind=0.0.0.0
dzm-agent.sources.dzm-source.port=44443
#dzm-agent.sources.dzm-source.selector.type = replicating

# channel
dzm-agent.channels.dzm-channel.type = org.apache.flume.channel.kafka.KafkaChannel
dzm-agent.channels.dzm-channel.kafka.bootstrap.servers = ceshi185:19092,ceshi186:19092,ceshi185:19092
dzm-agent.channels.dzm-channel.kafka.topic = flume_dzm_channel
dzm-agent.channels.dzm-channel.kafka.consumer.group.id = flume_dzm_channel

# sink
dzm-agent.sinks.dzm-sink.type = asynchbase
dzm-agent.sinks.dzm-sink.table = t_invoice_ticket
dzm-agent.sinks.dzm-sink.columnFamily = i
dzm-agent.sinks.dzm-sink.serializer = com.bwjf.flume.invoice.dzm.sink.InvoiceHbaseSerializer

# detail-channel
dzm-agent.channels.dzm-channel-detail.type = org.apache.flume.channel.kafka.KafkaChannel
dzm-agent.channels.dzm-channel-detail.kafka.bootstrap.servers = ceshi185:19092,ceshi186:19092,ceshi185:19092
dzm-agent.channels.dzm-channel-detail.kafka.topic = flume_dzm_detail_channel
dzm-agent.channels.dzm-channel-detail.kafka.consumer.group.id = flume_dzm_detail_channel

# detail-sink
dzm-agent.sinks.dzm-sink-detail.type = asynchbase
dzm-agent.sinks.dzm-sink-detail.table = t_invoice_detail_ticket
dzm-agent.sinks.dzm-sink-detail.columnFamily = i
dzm-agent.sinks.dzm-sink-detail.serializer = com.bwjf.flume.invoice.dzm.sink.InvoiceDetailHbaseSerializer

# assemble
dzm-agent.sources.dzm-source.channels = dzm-channel dzm-channel-detail
dzm-agent.sinks.dzm-sink.channel = dzm-channel
dzm-agent.sinks.dzm-sink-detail.channel = dzm-channel-detail

5 sink组
sink组中所有sink是不会同时被激活，任何时候只有它们中的一个用来发送数据，因此sink组不应该用来更快的清除channel。

# read from kafka and write to hbase
dzm-agent.sources = dzm-source
dzm-agent.channels = dzm-channel dzm-channel-detail
dzm-agent.sinks = dzm-sink1 dzm-sink2 dzm-sink3 dzm-sink-detail1 dzm-sink-detail2 dzm-sink-detail3
dzm-agent.sinkGroups = dzm-sg dzm-sg-detail

# source
dzm-agent.sources.dzm-source.type=avro
dzm-agent.sources.dzm-source.bind=0.0.0.0
dzm-agent.sources.dzm-source.port=44443
#dzm-agent.sources.dzm-source.selector.type = replicating

# channel
dzm-agent.channels.dzm-channel.type = org.apache.flume.channel.kafka.KafkaChannel
dzm-agent.channels.dzm-channel.kafka.bootstrap.servers = ceshi185:19092,ceshi186:19092,ceshi185:19092
dzm-agent.channels.dzm-channel.kafka.topic = flume_dzm_channel
dzm-agent.channels.dzm-channel.kafka.consumer.group.id = flume_dzm_channel

# sink group
dzm-agent.sinkGroups = dzm-sg
dzm-agent.sinkGroups.dzm-sg.sinks = dzm-sink1 dzm-sink2 dzm-sink3
dzm-agent.sinkGroups.dzm-sg.processor.type = load_balance
dzm-agent.sinkGroups.dzm-sg.processor.backoff = load_balance

# sink
dzm-agent.sinks.dzm-sink1.type = asynchbase
dzm-agent.sinks.dzm-sink1.table = t_invoice_ticket
dzm-agent.sinks.dzm-sink1.columnFamily = i
dzm-agent.sinks.dzm-sink1.serializer = com.bwjf.flume.invoice.dzm.sink.InvoiceHbaseSerializer

dzm-agent.sinks.dzm-sink2.type = asynchbase
dzm-agent.sinks.dzm-sink2.table = t_invoice_ticket
dzm-agent.sinks.dzm-sink2.columnFamily = i
dzm-agent.sinks.dzm-sink2.serializer = com.bwjf.flume.invoice.dzm.sink.InvoiceHbaseSerializer

dzm-agent.sinks.dzm-sink3.type = asynchbase
dzm-agent.sinks.dzm-sink3.table = t_invoice_ticket
dzm-agent.sinks.dzm-sink3.columnFamily = i
dzm-agent.sinks.dzm-sink3.serializer = com.bwjf.flume.invoice.dzm.sink.InvoiceHbaseSerializer

# detail-channel
dzm-agent.channels.dzm-channel-detail.type = org.apache.flume.channel.kafka.KafkaChannel
dzm-agent.channels.dzm-channel-detail.kafka.bootstrap.servers = ceshi185:19092,ceshi186:19092,ceshi185:19092
dzm-agent.channels.dzm-channel-detail.kafka.topic = flume_dzm_detail_channel
dzm-agent.channels.dzm-channel-detail.kafka.consumer.group.id = flume_dzm_detail_channel

# detail-sink group
dzm-agent.sinkGroups = dzm-detail-sg
dzm-agent.sinkGroups.dzm-detail-sg.sinks = dzm-sink-detail1 dzm-sink-detail2 dzm-sink-detail3
dzm-agent.sinkGroups.dzm-detail-sg.processor.type = load_balance
dzm-agent.sinkGroups.dzm-detail-sg.processor.backoff = load_balance

# detail-sink
dzm-agent.sinks.dzm-sink-detail1.type = asynchbase
dzm-agent.sinks.dzm-sink-detail1.table = t_invoice_detail_ticket
dzm-agent.sinks.dzm-sink-detail1.columnFamily = i
dzm-agent.sinks.dzm-sink-detail1.serializer = com.bwjf.flume.invoice.dzm.sink.InvoiceDetailHbaseSerializer

dzm-agent.sinks.dzm-sink-detail2.type = asynchbase
dzm-agent.sinks.dzm-sink-detail2.table = t_invoice_detail_ticket
dzm-agent.sinks.dzm-sink-detail2.columnFamily = i
dzm-agent.sinks.dzm-sink-detail2.serializer = com.bwjf.flume.invoice.dzm.sink.InvoiceDetailHbaseSerializer

dzm-agent.sinks.dzm-sink-detail3.type = asynchbase
dzm-agent.sinks.dzm-sink-detail3.table = t_invoice_detail_ticket
dzm-agent.sinks.dzm-sink-detail3.columnFamily = i
dzm-agent.sinks.dzm-sink-detail3.serializer = com.bwjf.flume.invoice.dzm.sink.InvoiceDetailHbaseSerializer

# assemble
dzm-agent.sources.dzm-source.channels = dzm-channel dzm-channel-detail
dzm-agent.sinks.dzm-sink1.channel = dzm-channel
dzm-agent.sinks.dzm-sink2.channel = dzm-channel
dzm-agent.sinks.dzm-sink3.channel = dzm-channel
dzm-agent.sinks.dzm-sink-detail1.channel = dzm-channel-detail
dzm-agent.sinks.dzm-sink-detail2.channel = dzm-channel-detail
dzm-agent.sinks.dzm-sink-detail3.channel = dzm-channel-detail

参考资料：