整合Flume和Kafka完成实时数据采集

结合我的这三篇博客　Flume应用案例之两台服务器之间进行实时数据采集
和分布式消息队列Kafka简介及环境安装部署
Flume应用案例之监控一个文件实时采集新增的数据输出到控制台
把avro-memory-logger.conf改成avro-memory-kafka.conf
内容如下:

avro-memory-kafka.sources = avro-source
avro-memory-kafka.sinks = kafka.sink
avro-memory-kafka.channels = memory-channel


avro-memory-kafka.sources.avro-source.type = avro
avro-memory-kafka.sources.avro-source.bind = zq
avro-memory-kafka.sources.avro-source.port = 44444


avro-memory-kafka.sinks.kafka.sink.type = org.apache.flume.sink.kafka.KafkaSink
avro-memory-kafka.sinks.kafka.sink.brokerlist = zq:9092
avro-memory-kafka.sinks.kafka.sink.topic = topic_test
avro-memory-kafka.sinks.kafka.sink.batchSize = 5
avro-memory-kafka.sinks.kafka.sink.requiredAcks = 1 


avro-memory-kafka.channels.memory-channel.type = memory


avro-memory-kafka.sources.avro-source.channels = memory-channel
avro-memory-kafka.sinks.kafka-sink.channel = memory-channel

最先开始启动zookeeper,再是kafka
然后再分别启动avro-memory-kafka.conf和exec-memory-avro.conf
再启动kafka的消费者
详情见开头说的我的三篇博客
最后开始测试：
再开一个终端，往data.log文件里面传输数据
开始会有一点慢，因为上文批处理数量是５,batchSize = 5
如果每次５条数据，就会立马响应并显示在控制台
像这样就完成了整合Flume和Kafka完成实时数据采集的任务