flume集成kafka入门案例

kafka安装与flume安装网上很多，这里不进行讲解

1.在flume conf目录下创建配置文件

#配置文件信息
a1.sources = s1
a1.sinks = k1
a1.channels = c1

a1.sources.s1.type = exec
# 配置监控文件
a1.sources.s1.command = tail -F /export/servers/taillogs/info.log


#配置管道信息
a1.channels.c1.type = memory
a1.channels.c1.capactity = 10000
a1.channels.c1.transactionCapactity =100

#设置kafka接收器
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
#设置kafka的broker地址与端口号
a1.sinks.k1.brokerList = node01:9092,node02:9092,node03:9092
#设置kafka的topic
a1.sinks.k1.topic = flumetest
#设置kafka序列化方式
#a1.sinks.k1.serializer.class = kafka.serializer.StringEncoder
a1.sinks.k1.flumeBatchSize = 20
a1.sinks.k1.kafka.producer.acks = 1
a1.sinks.k1.kafka.producer.compression.type = snappy


a1.sources.s1.channels = c1
a1.sinks.k1.channel = c1

2.自动向监控文件中添加数据用于测试

#!/bin/bash

while true
do
 date >> /export/servers/taillogs/info.log;
 sleep 0.5;
done

3.在任一机器下创建topic

./kafka-topics.sh --create --zookeeper node01:2181,node02:2181,node03:2181 --replication-factor 2 --partitions 3 --topic flumetest

验证一下是否创建成功

./kafka-topics.sh --list --zookeeper node01:2181,node02:2181,node03:2181

4.启动消费者，进行数据消费

./kafka-console-consumer.sh --bootstrap-server node01:9092,node02:9020,node03:9020 --topic flumetest --from-beginning

5.启动flume,进行文件监控

flume-ng agent -n a1 -c conf -f ../conf/jiankong.conf -Dflume.root.logger=DEBUG,console

遇到问题？？？？

1.版本问题

[ERROR - org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:427)] Sink k1 has been removed due to an error during configuration
org.apache.flume.conf.ConfigurationException: brokerList must contain at least one Kafka broker
        at org.apache.flume.sink.kafka.KafkaSinkUtil.addDocumentedKafkaProps(KafkaSinkUtil.java:55)
        at org.apache.flume.sink.kafka.KafkaSinkUtil.getKafkaProperties(KafkaSinkUtil.java:37)

所以配置里的a1.sinks.k1.kafka.bootstrap.servers = master:9092,slave1:9092,slave2:9092换成a1.sinks.k1.brokerList= master:9092,slave1:9092,slave2:9092