flume集成kafka完整案例

flume集成kafka官网案例：http://flume.apache.org/FlumeUserGuide.html#kafka-sink

一：flume集成kafka配置文件

a1.sources = r1

a1.channels = c1

a1.sinks = k1

 

a1.sources.r1.type = exec

#tail -F 根据文件名进行追踪

a1.sources.r1.command = tail -F /export/servers/taillogs/access.log

a1.sources.r1.channels = c1

 

a1.channels.c1.type=memory

a1.channels.c1.capacity=10000

a1.channels.c1.transactionCapacity=100

 

a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink

#指定kafka类型

a1.sinks.k1.topic = system_log

#kafka集群地址

a1.sinks.k1.brokerList = node01:9092,node02:9092,node03:9092

a1.sinks.k1.requiredAcks = 1

a1.sinks.k1.batchSize = 20

a1.sinks.k1.channel = c1

二：准备数据

1）创建access.log文件

mkdir -p /export/servers/taillogs/

touch access.log

2）创建往access.log日志文件添加数据脚本 tail-file.sh 文件

#!/bin/bash

while true

do

date >> /export/servers/taillogs/access.log;

sleep 0.5;

done

3）创建system_log的topic

./kafka-topics.sh --create --zookeeper node01:2181,node02:2181,node03:2181 --topic itcast_13 --partitions 3 --replication-factor 1

三：启动

1）进入flume安装目录下启动flume。（kafka.conf是flume集成kafka的配置内容）

bin/flume-ng agent -c conf -f conf/kafka.conf -n a1  -Dflume.root.logger=INFO,console

2）启动模拟生产日志数据脚本

./tail-file.sh

3）在kafka安装的bin目录下启动kafka消费者查看数据

kafka-console-consumer.sh --from-beginning --topic system_log --zookeeper node01:2181,node02:2181,node03:218