Spark Streaming实时流处理笔记（6）—— Kafka 和 Flume的整合

1 整体架构

2 Flume 配置

https://flume.apache.org/releases/content/1.6.0/FlumeUserGuide.html
启动kafka kafka-server-start.sh $KAFKA_HOME/config/server.properties

avro-memory-kafka.conf

# Name the components on this agent                        
avro-memory-kafka.sources = avro-source                    
avro-memory-kafka.sinks = kafka-sink                       
avro-memory-kafka.channels = memory-channel                
                                                           
# Describe/configure the source                            
avro-memory-kafka.sources.avro-source.type = avro          
avro-memory-kafka.sources.avro-source.bind = node1         
avro-memory-kafka.sources.avro-source.port = 44444         
                                                           
# Describe the sink                                        
avro-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
avro-memory-kafka.sinks.kafka-sink.brokerList = node1:9092 
avro-memory-kafka.sinks.kafka-sink.topic = hello_topic     
avro-memory-kafka.sinks.kafka-sink.batchSize = 5           
avro-memory-kafka.sinks.kafka-sink.requiredAcks = 1        
                                                           
# Use a channel which buffers events in memory             
avro-memory-kafka.channels.memory-channel.type = memory    
                                                           
                                                           
# Bind the source and sink to the channel                  
avro-memory-kafka.sources.avro-source.channels = memory-channel
avro-memory-kafka.sinks.kafka-sink.channel = memory-channel

2.1 启动 Flume

flume-ng agent --name avro-memory-kafka --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/myconf/avro-memory-kafka.conf -Dflume.root.logger=INFO,console

flume-ng agent --name exec-memory-avro --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/myconf/exec-memory-avro.conf -Dflume.root.logger=INFO,console

2.2 启动 Kafka

启动消费者

kafka-console-consumer.sh --zookeeper node1:2181 --topic hello_topic