从无到有系列之flume-kafka整合01

一、服务器配置

版本介绍：apache-flume-1.6.0-bin + kafka_2.9.2-0.8.2.2

集群说明：

192.168.10.66   broker、consumer、zookeeper 、flume

192.168.10.67   broker、zookeeper

192.168.10.68   broker、zookeeper

二、分别到三个节点启动zookepper节点

zkServer.sh  start

使用命令zkServer.sh  status检查zookepper节点启动情况

三、下载整合插件JAR包 flumeng-kafka-plugin.jar

这里就不提供了，网站自己找吧

四、启动Kafka集群

  这里kafka的集群配置参照博主本系列另一篇文章 点击打开链接

   三个节点执行命令，启动kafka服务    bin/kafka-server-start.sh config/server.properties >/dev/null 2>&1 &

主节点上执行：

创建消息主题topic     bin/kafka-topics.sh --create --zookeeper 192.168.10.66:2181 --replication-factor 3 --partitions 3 --topicwjTopic

新开窗口，创建消费者consumer      bin/kafka-console-consumer.sh --zookeeper 192.168.10.66:2181 --topic  wjTopic  --from-beginning

五、配置flume

cd /usr/local/apache-flume-1.6.0-bin

vi conf/flume-conf.properties ，加入如下内容：

#agent
producer.sources = s
producer.channels = c
producer.sinks = r
#source
producer.sources.s.type = spooldir
producer.sources.s.channels = c
producer.sources.s.spoolDir = /var/ceshi
producer.sources.s.fileHeader = true

producer.sinks.r.type = org.apache.flume.plugins.KafkaSink
producer.sinks.r.metadata.broker.list=192.168.10.66:9092
producer.sinks.r.partitioner.class=org.apache.flume.plugins.SinglePartition
producer.sinks.r.partition.key=1
producer.sinks.r.serializer.class=kafka.serializer.StringEncoder
producer.sinks.r.request.required.acks=0
producer.sinks.r.max.message.size=1000000
producer.sinks.r.producer.type=async
producer.sinks.r.custom.encoding=UTF-8
producer.sinks.r.custom.topic.name=test

#Specify the channel for the sink
producer.sinks.r.channel = c
producer.channels.c.type = memory
producer.channels.c.capacity = 1000

六、启动agent服务

在flume的home目录下执行命令   flume-ng agent -n producer conf -f conf/flume-conf.properties -Dflume.root.logger=DEBUG,console

七、检验

在hmbbs中新建一个文件，并输入“hello”，在主节点的消费者窗口可见“hello”，到此整合成功

八、更换不同的source

例如监听log，可将配置改为：

 #agent

producer.sources = s
producer.channels = c
producer.sinks = r
#source
producer.sources.s.type= exec
producer.sources.s.command= tail -F /usr/hadoop/hadoop-2.6.0/logs/hadoop-root-namenode-master.log
producer.sources.s.channels = c
producer.sinks.r.type = org.apache.flume.plugins.KafkaSink
producer.sinks.r.metadata.broker.list=master:9092
producer.sinks.r.partitioner.class=org.apache.flume.plugins.SinglePartition
producer.sinks.r.partition.key=1
producer.sinks.r.serializer.class=kafka.serializer.StringEncoder
producer.sinks.r.request.required.acks=0
producer.sinks.r.max.message.size=1000000
producer.sinks.r.producer.type=async
producer.sinks.r.custom.encoding=UTF-8
producer.sinks.r.custom.topic.name=wjTopic
#Specify the channel for the sink
producer.sinks.r.channel = c
producer.channels.c.type = memory
producer.channels.c.capacity = 1000