参考：
 https://www.jianshu.com/p/d889aae7c72e

filebeat日志通过redis传递至logstash在输出至elasticsearch参考

  1. 场景需求说明
    在同一台主机有多个日志需要区分不同index输出至elasticsearch
  2. filebeat配置
# cat /etc/filebeat/filebeat.yml
# 给不同日志打不同tags用于区分
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/supervisor/fastchat/*.log
  tags: ["psych-log-0388"]
- type: log
  enabled: true
  paths:
    - /var/log/supervisor/bert/*.log
  tags: ["bert-log-0388"]
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
# 输出至redis 
# key是自定义
output.redis:
  hosts: ["192.168.3.65:46379"]
  db: "3"
  password: "password"
  key: "0388"
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  1. logstash配置
# cat /etc/logstash/conf.d/psych-bert-log-redis-to-elasticsearch.conf 
# key和filebeat设置为一致
input{
    redis {
    host => "192.168.3.65"
    port => "46379"
    password => "password"
    db => "3"
    data_type => "list"
    key => "0388"
 
}
}

# 通过在filebeat中自定义tags来区分不同日志并使用不同的index输出至elasticsearch 
output{
    if "psych-log-0388" in [tags]  {
       elasticsearch {
            hosts => ["192.168.3.59:9200"]
            index => "psych-log-0388-%{+YYYY.MM.dd}"
        }
        #stdout{
        #   codec => rubydebug
        #}
    }
    if "bert-log-0388" in [tags]  {
       elasticsearch {
            hosts => ["192.168.3.59:9200"]
            index => "bert-log-0388-%{+YYYY.MM.dd}"
        }
        #stdout{
        #   codec => rubydebug
        #}
    }
}
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.

启动logstash

# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/psych-bert-log-redis-to-elasticsearch.conf
  • 1.

查看elasticsearch是否收到日志

# curl http://192.168.3.59:9200/_cat/indices|grep psych
  • 1.

在kibana添加对应日志不详述