Filebeat 传输数据到Kafka无法连接

最新推荐文章于 2024-01-04 17:28:30 发布

Peppa·Pig

最新推荐文章于 2024-01-04 17:28:30 发布

阅读量2.9k

点赞数 1

分类专栏： ELK 文章标签： ELK

本文链接：https://blog.csdn.net/u011940366/article/details/112007948

版权

ELK 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

最近开始研究ELK，准备做个日志系统，数据采集使用Filebeat组件，输出源配好kafka后，启动filebeat报如下日志：

2020-12-31T13:54:53.203+0800    INFO    kafka/log.go:53 client/metadata fetching metadata for [flink-log-test] from broker 172.30.79.135:6667

2020-12-31T13:54:53.204+0800    INFO    kafka/log.go:53 producer/broker/1007 starting up

2020-12-31T13:54:53.204+0800    INFO    kafka/log.go:53 producer/broker/1007 state change to [open] on flink-log-test/0

2020-12-31T13:54:53.204+0800    INFO    kafka/log.go:53 producer/leader/flink-log-test/0 selected broker 1007

2020-12-31T13:54:53.204+0800    INFO    kafka/log.go:53 producer/leader/flink-log-test/0 state change to [flushing-1]

2020-12-31T13:54:53.204+0800    INFO    kafka/log.go:53 producer/leader/flink-log-test/0 state change to [normal]

2020-12-31T13:54:53.204+0800    INFO    kafka/log.go:53 Connected to broker at bg134:6667 (registered as #1007)

2020-12-31T13:54:53.205+0800    INFO    kafka/log.go:53 producer/broker/1007 state change to [closing] because EOF

2020-12-31T13:54:53.205+0800    INFO    kafka/log.go:53 Closed connection to broker bg134:6667

2020-12-31T13:54:53.205+0800    INFO    kafka/log.go:53 producer/leader/flink-log-test/0 state change to [retrying-2]

2020-12-31T13:54:53.205+0800    INFO    kafka/log.go:53 producer/leader/flink-log-test/0 abandoning broker 1007

2020-12-31T13:54:53.205+0800    INFO    kafka/log.go:53 producer/broker/1007 shut down

2020-12-31T13:54:53.305+0800    INFO    kafka/log.go:53 client/metadata fetching metadata for [flink-log-test] from broker 172.30.79.135:6667

由日志信息可看出，filebeat一直在做连接kafka的动作。对比了网上其他人的模板，发现自己配的确实没有差异，大家都只配了“hosts”和“topic”。于是怀疑是不是参数不全导致。

到官网查看了指导手册，发现：

Configuration optionsedit
You can specify the following options in the kafka section of the filebeat.yml config file:
***
versionedit
Kafka version filebeat is assumed to run against. Defaults to 1.0.0.

Event timestamps will be added, if version 0.10.0.0+ is enabled.

Valid values are all kafka releases in between 0.8.2.0 and 2.0.0.

原来这个版本的Filebeat默认使用1.0.0版本的kafka客户端，我的kafka客户端版本是0.10.1，需要特殊指定“version”。遂在配置文件加上：