一、前言
本篇环境
Elasticsearch6.0 , Logstash6.0 , Kibana6.0 , filebeat6.0
这种索引模式建多了,比较吃内存请注意自己服务器的消耗
流程filebeat ⇒ redis ==>Logstash
流程图 请参考:https://blog.csdn.net/liyyzz33/article/details/86701949
二、目的
filebeat多日志传输,经Logstash过滤后输出多索引。
三、配置文件
filebeat.yml
主要通过log_type来判断
filebeat.prospectors:
- input_type: log
paths:
- /data/logs/php/*.log
fields:
log_source: php
log_type: php
tail_files: true
scan_frequency: 60s
backoff: 10s
- input_type: log
paths:
- /data/logs/web/*.log
fields:
log_source: web
log_type: web
tail_files: true
scan_frequency: 60s
backoff: 10s
output.redis:
hosts: ["192.168.1.123"]
port: 6379
key: "filebeat"
db: 0
timeout: 5
Logstash
input-output.conf
input {
redis {
data_type => "list"
key => "filebeat"
host => "192.168.1.123"
port => 6379
threads => 5
codec => "json"
}
}
filter {
}
output {
if [fields][log_type] == "php" {
elasticsearch {
hosts => ["192.168.1.60:9200","192.168.1.61:9200","192.168.1.62:9200"]
index => "php-%{+YYYY.MM.dd}"
action => "index"
}
}
if [fields][log_type] == "web" {
elasticsearch {
hosts => ["192.168.1.60:9200","192.168.1.61:9200","192.168.1.62:9200"]
index => "web-%{+YYYY.MM.dd}"
action => "index"
}
stdout {
}
}
效果