Flume应用案例之整合Flume 和 kafka完成实时数据采集

一、需求：整合Flume 和 kafka完成实时数据采集

说明：如下图
机器Agent1对应本人hadoop01虚拟机，
机器Agent2对应本人hadoop02虚拟机
1、机器Agent1监控日志文件，向机器Agent1中输入数据
2、avro sink把新产生的日志输出到对应的avro source 指定的hostname 和port上
3、通过avro source对应的agent2将日志输出到kafka（生产者）
4、由kafka consumer（消费者输出数据）

二、技术选型：

1、jdk1.8
2、zookeeper-3.4.14.tar.gz Zookeeper安装及配置
3、kafka_2.12-2.4.0.tgz Kafka集群部署
4、apache-flume-1.9.0-bin.tar.gz Flume安装部署

flume 技术选型：
exec source + memory channel + avro sink hadoop01服务器
avro source + memory channel + kafka sink hadoop02服务器

vim exec-memory-avro.conf hadoop01机器

#定义这个agent中各组件的名字
exec-memory-avro.sources = exec-sources
exec-memory-avro.sinks = avro-sink
exec-memory-avro.channels = memory-channel

exec-memory-avro.sources.exec-sources.type = exec
exec-memory-avro.sources.exec-sources.command = tail -f /opt/bigdatas/flumedata.log
exec-memory-avro.sources.exec-sources.shell = /bin/sh -c

exec-memory-avro.sinks.avro-sink.type = avro
exec-memory-avro.sinks.avro-sink.hostname = hadoop02
exec-memory-avro.sinks.avro-sink.port = 44444

exec-memory-avro.channels.memory-channel.type = memory

exec-memory-avro.sources.exec-sources.channels = memory-channel
exec-memory-avro.sinks.avro-sink.channel = memory-channel

vim avro-memory-kafka.conf hadoop02服务器

#定义这个agent中各组件的名字
avro-memory-kafka.sources = avro-sources
avro-memory-kafka.sinks = kafka-sink
avro-memory-kafka.channels = memory-channel

avro-memory-kafka.sources.avro-sources.type = avro
avro-memory-kafka.sources.avro-sources.bind = hadoop02
avro-memory-kafka.sources.avro-sources.port = 44444

# sink 配置参见http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html#kafka-sink
avro-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
avro-memory-kafka.sinks.kafka-sink.kafka.bootstrap.servers = hadoop01:9092,hadoop02:9092,hadoop03:9092
# kafka topic：test
avro-memory-kafka.sinks.kafka-sink.kafka.topic = test
avro-memory-kafka.sinks.kafka-sink.flumeBatchSize= 5
avro-memory-kafka.sinks.kafka-sink.kafka.producer.acks = 1

avro-memory-kafka.channels.memory-channel.type = memory

avro-memory-kafka.sources.avro-sources.channels = memory-channel
avro-memory-kafka.sinks.kafka-sink.channel = memory-channel

三、启动Zookeeper集群

依次启动：

zkServer.sh start
zkServer.sh status  -- 查看启动状态

四、启动kafka集群

cd kafka
bin/kafka-server-start.sh config/server.properties

五、先启动hadoop02机器 avro-memory-kafka

./flume-ng agent --name avro-memory-kafka --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/avro-memory-kafka.conf -Dflume.root.logger=INFO,console

六、启动hadoop01机器 exec-memory-avro

./flume-ng agent --name exec-memory-avro --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/exec-memory-avro.conf -Dflume.root.logger=INFO,console

七、打开hadoop02，启动kafka消费端

/opt/kafka/kafka_2.12-2.4.0/bin/kafka-console-consumer.sh --bootstrap-server hadoop01:9092 --topic test 

八、打开hadoop01,向/opt/bigdatas/flumedata中输入数据

九、可以在hadoop02，kafka消费端看到数据输出