一、需求:整合Flume 和 kafka完成实时数据采集
说明:如下图
机器Agent1对应本人hadoop01虚拟机,
机器Agent2对应本人hadoop02虚拟机
1、机器Agent1监控日志文件,向机器Agent1中输入数据
2、avro sink把新产生的日志输出到对应的avro source 指定的hostname 和port上
3、通过avro source对应的agent2将日志输出到kafka(生产者)
4、由kafka consumer(消费者输出数据)
二、技术选型:
1、jdk1.8
2、zookeeper-3.4.14.tar.gz Zookeeper安装及配置
3、kafka_2.12-2.4.0.tgz Kafka集群部署
4、apache-flume-1.9.0-bin.tar.gz Flume安装部署
flume 技术选型:
exec source + memory channel + avro sink hadoop01服务器
avro source + memory channel + kafka sink hadoop02服务器
vim exec-memory-avro.conf hadoop01机器
#定义这个agent中各组件的名字
exec-memory-avro.sources = exec-sources
exec-memory-avro.sinks = avro-sink
exec-memory-avro.channels = memory-channel
exec-memory-avro.sources.exec-sources.type = exec
exec-memory-avro.sources.exec-sources.command = tail -f /opt/bigdatas/flumedata.log
exec-memory-avro.sources.exec-sources.shell = /bin/sh -c
exec-memory-avro.sinks.avro-sink.type = avro
exec-memory-avro.sinks.avro-sink.hostname = hadoop02
exec-memory-avro.sinks.avro-sink.port = 44444
exec-memory-avro.channels.memory-channel.type = memory
exec-memory-avro.sources.exec-sources.channels = memory-channel
exec-memory-avro.sinks.avro-sink.channel = memory-channel
vim avro-memory-kafka.conf hadoop02服务器
#定义这个agent中各组件的名字
avro-memory-kafka.sources = avro-sources
avro-memory-kafka.sinks = kafka-sink
avro-memory-kafka.channels = memory-channel
avro-memory-kafka.sources.avro-sources.type = avro
avro-memory-kafka.sources.avro-sources.bind = hadoop02
avro-memory-kafka.sources.avro-sources.port = 44444
# sink 配置参见http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html#kafka-sink
avro-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
avro-memory-kafka.sinks.kafka-sink.kafka.bootstrap.servers = hadoop01:9092,hadoop02:9092,hadoop03:9092
# kafka topic:test
avro-memory-kafka.sinks.kafka-sink.kafka.topic = test
avro-memory-kafka.sinks.kafka-sink.flumeBatchSize= 5
avro-memory-kafka.sinks.kafka-sink.kafka.producer.acks = 1
avro-memory-kafka.channels.memory-channel.type = memory
avro-memory-kafka.sources.avro-sources.channels = memory-channel
avro-memory-kafka.sinks.kafka-sink.channel = memory-channel
三、启动Zookeeper集群
依次启动:
zkServer.sh start
zkServer.sh status -- 查看启动状态
四、启动kafka集群
cd kafka
bin/kafka-server-start.sh config/server.properties
五、先启动hadoop02机器 avro-memory-kafka
./flume-ng agent --name avro-memory-kafka --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/avro-memory-kafka.conf -Dflume.root.logger=INFO,console
六、启动hadoop01机器 exec-memory-avro
./flume-ng agent --name exec-memory-avro --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/exec-memory-avro.conf -Dflume.root.logger=INFO,console
七、打开hadoop02,启动kafka消费端
/opt/kafka/kafka_2.12-2.4.0/bin/kafka-console-consumer.sh --bootstrap-server hadoop01:9092 --topic test