Flume与Kafka整合

Flume与Kakfa整合

flume官方下载地址：https://flume.apache.org/download.html

建议下载最新的1.6.0版本的，因为1.6.0版本的集成了整合kafka的插件包可以直接配置使用

1、下载并解压apache-flume-1.6.0-bin.tar.gz包

通过tar –zxvf apache-flume-1.6.0-bin.tar.gz命令解压压缩文件

Flume解压既安装成功，配置conf/flume-conf.properties文件启动完成相应的功能

2、配置flume连接kafka

前面kafka集群已经成功，这里只需要配置好conf/flume-conf.properties文件，配置如下

a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type=avro
a1.sources.r1.bind=master
a1.sources.r1.port=41414

# Describe the sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink  
a1.sinks.k1.topic = testflume  
a1.sinks.k1.brokerList = 192.168.230.129:9092,192.168.230.130:9092,192.168.230.131:9092  
a1.sinks.k1.requiredAcks = 1  
a1.sinks.k1.batchSize =