一、服务器配置
版本介绍:apache-flume-1.6.0-bin + kafka_2.9.2-0.8.2.2
集群说明:
192.168.10.66 broker、consumer、zookeeper 、flume
192.168.10.67 broker、zookeeper
192.168.10.68 broker、zookeeper
二、分别到三个节点启动zookepper节点
zkServer.sh start
使用命令zkServer.sh status检查zookepper节点启动情况
三、下载整合插件JAR包 flumeng-kafka-plugin.jar
这里就不提供了,网站自己找吧
四、启动Kafka集群
这里kafka的集群配置参照博主本系列另一篇文章 点击打开链接
三个节点执行命令,启动kafka服务 bin/kafka-server-start.sh config/server.properties >/dev/null 2>&1 &
主节点上执行:
创建消息主题topic bin/kafka-topics.sh --create --zookeeper 192.168.10.66:2181 --replication-factor 3 --partitions 3 --topicwjTopic
新开窗口,创建消费者consumer bin/kafka-console-consumer.sh --zookeeper 192.168.10.66:2181 --topic wjTopic --from-beginning
五、配置flume
cd /usr/local/apache-flume-1.6.0-bin
vi conf/flume-conf.properties ,加入如下内容:
#agent
producer.sources = s
producer.channels = c
producer.sinks = r
#source
producer.sources.s.type = spooldir
producer.sources.s.channels = c
producer.sources.s.spoolDir = /var/ceshi
producer.sources.s.fileHeader = true
producer.sinks.r.type = org.apache.flume.plugins.KafkaSink
producer.sinks.r.metadata.broker.list=192.168.10.66:9092
producer.sinks.r.partitioner.class=org.apache.flume.plugins.SinglePartition
producer.sinks.r.partition.key=1
producer.sinks.r.serializer.class=kafka.serializer.StringEncoder
producer.sinks.r.request.required.acks=0
producer.sinks.r.max.message.size=1000000
producer.sinks.r.producer.type=async
producer.sinks.r.custom.encoding=UTF-8
producer.sinks.r.custom.topic.name=test
#Specify the channel for the sink
producer.sinks.r.channel = c
producer.channels.c.type = memory
producer.channels.c.capacity = 1000
六、启动agent服务
在flume的home目录下执行命令 flume-ng agent -n producer conf -f conf/flume-conf.properties -Dflume.root.logger=DEBUG,console
七、检验
在hmbbs中新建一个文件,并输入“hello”,在主节点的消费者窗口可见“hello”,到此整合成功
八、更换不同的source
例如监听log,可将配置改为:
#agent
producer.sources = s
producer.channels = c
producer.sinks = r
#source
producer.sources.s.type= exec
producer.sources.s.command= tail -F /usr/hadoop/hadoop-2.6.0/logs/hadoop-root-namenode-master.log
producer.sources.s.channels = c
producer.sinks.r.type = org.apache.flume.plugins.KafkaSink
producer.sinks.r.metadata.broker.list=master:9092
producer.sinks.r.partitioner.class=org.apache.flume.plugins.SinglePartition
producer.sinks.r.partition.key=1
producer.sinks.r.serializer.class=kafka.serializer.StringEncoder
producer.sinks.r.request.required.acks=0
producer.sinks.r.max.message.size=1000000
producer.sinks.r.producer.type=async
producer.sinks.r.custom.encoding=UTF-8
producer.sinks.r.custom.topic.name=wjTopic
#Specify the channel for the sink
producer.sinks.r.channel = c
producer.channels.c.type = memory
producer.channels.c.capacity = 1000