架构图如上
【kafka端】
1.启动kafa
./kafka-server-start.sh ../config/server.properties 1>/dev/null 2>&1 & # kafka-server-start.sh 脚本在 kafka_2.12-2.2.0/bin路径下
2.查看topic
./bin/kafka-topics.sh --list --zookeeper localhost:2181 | grep flume_log
3.新建topic
./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic flume_log
【filebeat端】
1.编辑文件vim filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /data/log/purchase/user_purchase_log_*
#=========================== user_purchase_log 商品访问日志 h5+buz =============================
##------------------------------kafka-----------------------------------
output.kafka:
hosts: ["IP:9092"]
topic: flume_log
keep_alive: 10s
2.运行:
./filebeat -e -c filebeat.yml
[报错]
在运行./filebeat时报出error错误:
Exiting: error loading config file: yaml: line 197: did not find expected key
Exiting: error loading config file: yaml: line 197: did not find expected key
发现是格式错误,空格多了,去除点多余的空格即可
【运行完之后可以再kafak端查看topic有没有数据】
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic flume_log --from-beginning