切换架构
1、filebeat向kafka报数据
1、修改配置文件
vim /usr/local/filebeat/filebeat.yml 注释掉之前向es报数据的配置部分,添加向kafka报数据配置,如下 output.kafka: hosts: ["10.61.2.44:9092"] enabled: true topic: test |
- 测试
复制服务器窗口,一个用于添加数据,另外一个用于查看kafka侧结果: 窗口1、向filebeat所采集的文件之中添加数据,kafka自动会创建topic: 窗口2、执行命令查看结果/usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server 10.61.2.44:9092 --topic test --from-beginning |
2、Logstash
1、下载
下载wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.3.tar.gz,解压tar -xzvf /usr/local/logstash-6.2.3.tar.gz,并修改文件名mv logstash-6.2.3 logstash,并删除压缩文件rm -rf logstash-6.2.3.tar.gz |
2、修改配置
1、创建文件 cd /usr/local/logstash touch logstash.conf chmod 755 logstash.conf
vim /usr/local/logstash/logstash.conf
input { kafka{ group_id => "test-consumer-group" codec => "json"
topics => ["test"] bootstrap_servers => "10.61.2.44:9092" } }
output { elasticsearch { codec =>"json" hosts => ["10.61.2.44:9200"] index =>"test-%{+YYYY.MM.dd}" } } 3、测试配置文件: /usr/local/logstash/bin/logstash -t -f /usr/local/logstash/logstash.conf |
3、启动
/usr/local/logstash/bin/logstash -f /usr/local/logstash/logstash.conf & 后台启动 /usr/local/logstash/bin/logstash -f /usr/local/logstash/logstash.conf --debug 启动时会显示详细日志 |
- 日志格式化
- filebeat配置
报单个日志
vim /usr/local/filebeat/filebeat.yml 打开如下配置,并添加自己的配置进去 修改如下: fields: ip: 10.61.2.44 belong: other (注意缩进) 修改output.kafka:的topic属性为: topic: 'cloudlink-%{[fields.belong]}'
|
报多个日志
2.1在2.44上修改kafka的配置文件,新增如下属性(用于外部机器访问) advertised.listeners=PLAINTEXT://10.61.2.44:9092 2.2在filebeat.inputs:下面新增如下内容: - type: log enabled: true paths: - /var/log/*.log fields: ip: 10.61.2.44 belong: dmz |
2、logstash配置文件
参考地址:https://www.jianshu.com/p/d02e460cc4da |
input { kafka{ group_id => "test-consumer-group" codec => "json" bootstrap_servers => "10.61.2.44:9092" topics_pattern => "cloudlink-.*" consumer_threads => 5 auto_offset_reset => "latest" } }
output { elasticsearch { codec =>"json" hosts => ["10.61.2.44:9200"] index =>"%{[@metadata][topic]}-%{+YYYY.MM.dd}" } } |