一、说明
版本介绍:apache-flume-1.6.0-bin + kafka_2.10-0.8.2.0
场景说明:把flume中的数据sink到Kafka
集群说明:
192.168.215.90 broker、consumer、zookeeper 、flume
192.168.215.110 broker、zookeeper
192.168.215.120 broker、zookeeper
二、分别到三个节点启动zookepper节点
zkServer.sh start
使用命令zkServer.sh status检查zookepper节点启动情况
三、下载整合插件JAR包 flumeng-kafka-plugin.jar
下载地址--http://yunpan.cn/cF33J42EsRaDH (提取码:23bf)
四、启动Kafka集群
三个节点执行命令,启动kafka服务 bin/kafka-server-start.sh config/server.properties >/dev/null 2>&1 &
在90节点上执行:
创建消息主题topic bin/kafka-topics.sh --create --zookeeper 192.168.215.90:2181 --replication-factor 2 --partitions 4 --topic test
新开窗口,创建消费者consumer bin/kafka-console-consumer.sh --zookeeper 192.168.215.90:2181 --topic test --from-beginning
五、配置flume
cd /usr/local/apache-flume-1.5.2-bin
vi conf/producer.conf ,加入如下内容:
具体配置,如下所示:
#agent
producer.sources = s
producer.channels = c
producer.sinks = r
#source
producer.sources.s.type = spooldir
producer.sources.s.channels = c
producer.sources.s.spoolDir = /root/hmbbs
producer.sources.s.fileHeader = true
producer.sinks.r.type = org.apache.flume.plugins.KafkaSink
producer.sinks.r.metadata.broker.list=192.168.215.90:9092
producer.sinks.r.partitioner.class=org.apache.flume.plugins.SinglePartition
producer.sinks.r.partition.key=1
producer.sinks.r.serializer.class=kafka.serializer.StringEncoder
producer.sinks.r.request.required.acks=0
producer.sinks.r.max.message.size=1000000
producer.sinks.r.producer.type=async
producer.sinks.r.custom.encoding=UTF-8
producer.sinks.r.custom.topic.name=test
#Specify the channel for the sink
producer.sinks.r.channel = c
producer.channels.c.type = memory
producer.channels.c.capacity = 1000
附注:需要在/root目录下创建hmbbs目录
六、启动agent服务
在flume的home目录下执行命令 flume-ng agent -n producer conf -f conf/producer.conf -Dflume.root.logger=DEBUG,console
七、检验
在hmbbs中新建一个文件,并输入“this is a test”,在90节点的消费者窗口可见“this is a test”,说明整合成功!
八、接收流数据
在flume目录下修改producer.conf文件即可,修改如下:
vi conf/producer.conf