filebeat+redis+logstash+es+kibana 多日志收集

最新推荐文章于 2024-07-30 18:23:48 发布

xiangzilong

最新推荐文章于 2024-07-30 18:23:48 发布

阅读量246

点赞数

分类专栏：中间件

本文链接：https://blog.csdn.net/qq_36270681/article/details/118679331

版权

中间件专栏收录该内容

34 篇文章 0 订阅

订阅专栏

同时收集本地系统日志和docker 日志

[root@k8-node2-dc ~]#cat    /etc/filebeat/filebeat.yml  
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
    - /var/log/messages
  tags: ["system-log-239"]
  exclude_lines: ['^DBG','^$']

- type: log
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*.log
  tags: ["docker-log-239"]
  exclude_lines: ['^DBG','^$']

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
setup.template.settings:
  index.number_of_shards: 1
output.redis:
  hosts: ["10.1.1.242:6679"]
  db: "10"
  password: "MkdGH*3ldf"
  key: "log-239"
  timeout: 5

enabled: true 开启多路径收集

[root@logstash-28 conf.d]# cat logstas-docker.conf 
input{
    redis {
    host => "10.1.1.242"
    port => "6679"
    password => "MkdGH*3ldf"
    db => "10"
    data_type => "list"
    key => "log-239"
  
}
}
  
output{
    if "docker-log-239" in [tags]  {
       elasticsearch {
            hosts => ["10.1.234.21:9200"]
            index => "docker-log-239-%{+YYYY.MM.dd}"
        }
       stdout{
           codec => rubydebug
       }
    }
}

[root@logstash-28 conf.d]# cat logstas-simple.conf 
input{
    redis {
    host => "10.1.1.242"
    port => "6679"
    password => "MkdGH*3ldf"
    db => "10"
    data_type => "list"
    key => "log-239"
  
}
}
  
output{
    if "system-log-239" in [tags]  {
       elasticsearch {
            hosts => ["10.1.234.21:9200"]
            index => "system-log-239-%{+YYYY.MM.dd}"
        }
       stdout{
           codec => rubydebug
       }
    }
}

创建索引后可以看到如下效果