elk7.4+filebeat收集日志

已于 2023-09-28 14:59:46 修改
阅读量39 收藏
点赞数
分类专栏： Elasticsearch（elk） 文章标签： jenkins 运维
于 2020-04-23 15:58:00 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/huningfei/article/details/133024146
版权

一简介

流程图如下：每个客户端安装filebeat
filebeat--->>logstash-->elasticsearch--->kibana
安装es前提要安装java环境

二 elk三个插件安装

1 配置yum源

[root@elk xs]# cat /etc/yum.repos.d/elk.repo 
[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

2安装

sudo yum install logstash kibana elasticsearch

三 更改kibana和elasticsearch配置文件

1 elasticsearch配置文件如下

cluster.name: elk
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.17.199.231
http.port: 9200
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
cluster.initial_master_nodes: ["node-1"]

说明：当时没有启用cluster.initial_master_nodes: ["node-1"] ，启动一直报错，报错信息如下：

#[1] bootstrap checks failed
# [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

2kibana更改如下

vim /etc/kibana/kibana.yml
server.port: 5601 #监听端口
server.host: "172.17.199.231"
elasticsearch.hosts: ["http://172.17.199.231:9200"] #es地址

3启动kibana和elasticsearch

/etc/init.d/kibana start
/etc/init.d/elasticsearch start

四 设置logstash

说明：我这里收集的日志比较多，所以写了很多

1 配置文件

input {
  beats {
     port => 5044
  }
}


filter{
    grok {
       match => ["message", "%{SYSLOGBASE} %{GREEDYDATA:message}"]
       overwrite => ["message"]
    }
}

output {
    if [app] == "www" {
        if [type] == "tiantian-system-service-node2" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-system-service-node2-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-system-service-node4" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-system-service-node4-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-assets-service-node2" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-assets-service-node2-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-assets-service-node4" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-assets-service-node4-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-user-service-node2" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-user-service-node2-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-user-service-node4" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-user-service-node4-%{+YYYY.MM.dd}"
           }  
        }
        else if [type] == "tiantian-order-service-node2" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-order-service-node2-%{+YYYY.MM.dd}"
           }  
        }
        else if [type] == "tiantian-order-service-node4" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-order-service-node4-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-audit-service-node5" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-audit-service-node5-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-audit-service-node6" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-audit-service-node6-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-admin-service-node2" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-admin-service-node2-%{+YYYY.MM.dd}"
           }
        }
          
        else if [type] == "tiantian-admin-service-node4" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-admin-service-node4-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-collection-service-node5" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-collection-service-node5-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-collection-service-node6" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-collection-service-node6-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-collection-admin-node5" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-collection-admin-node5-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-collection-admin-node6" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-collection-admin-node6-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-collection-job-node5" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-collection-job-node5-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-channel-api-node2" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-channel-api-node2-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-channel-api-node4" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-channel-api-node4-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-job-node4" {
           elasticsearch { 
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-job-node4-%{+YYYY.MM.dd}"
           }  
        }  
        else if [type] == "tiantian-mq-service-node4" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-mq-service-node4-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-mq-service-node2" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-mq-service-node2-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "jz-asset-node1" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "jz-asset-node1-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-channel-admin-node4" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-channel-admin-node4-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-channel-admin-node2" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-channel-admin-node2-%{+YYYY.MM.dd}"
           }
        }

        else if [type] == "tiantian-admin-api-node5" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-admin-api-node5-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-admin-api-node6" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-admin-api-node6-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-api-v2-node2" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-api-v2-node2-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-api-v2-node4" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-api-v2-node4-%{+YYYY.MM.dd}"
           }  
        } 
        else if [type] == "tiantian-audit-admin-node5" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-audit-admin-node5-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-audit-admin-node6" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-audit-admin-node6-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "xxd-credit-service-node2" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "xxd-credit-service-node2-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "xxd-jinbaodai-api-node4" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "xxd-jinbaodai-api-node4-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "xxd-jinbaodai-api-node7" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "xxd-jinbaodai-api-node7-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "xxd-jinbaodai-service-node7" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "xxd-jinbaodai-service-node7-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "xxd-jinbaodai-service-node4" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "xxd-jinbaodai-service-node4-%{+YYYY.MM.dd}"
           }
        }
         else if [type] == "xxd-ops-service-node5" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "xxd-ops-service-node5-%{+YYYY.MM.dd}"
           }
        }
         else if [type] == "xxd-ops-service-node6" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "xxd-ops-service-node6-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "xxd-ops-web-node5" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "xxd-ops-web-node5-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-warning-node5" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-warning-node5-%{+YYYY.MM.dd}"
           }
        }
        else if [type] == "tiantian-warning-node6" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "tiantian-warning-node6-%{+YYYY.MM.dd}"
           }
        }
    }
  stdout { codec=> rubydebug }
}

2启动

设置服务自启动：systemctl enable logstash
启动服务：systemctl start logstash
停止服务：systemctl stop logstash
重启服务：systemctl restart logstash
查看服务状态：systemctl status logstash
报错1

启动报错 could not find java;set JAVA_HOME or ensure java报错，是因为java环境变量的问题，解决办法:

# 把你实际的java环境变量做个软连接即可
ln -s /usr/local/java/jdk1.8.0_111/bin/java /usr/bin/java
报错2

logstash收集日志报错 Logstash 报错: A plugin had an unrecoverable error. Will restart this plugin
解决办法，/etc/logstash/conf.d目录下面不要有多个conf配置文件
参考：ELK-生产测试遇到的问题及解决 - Jamin Zhang

五 filebeat

在客户端(产生日志的服务器上面安装)

1 安装

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.4.0-linux-x86_64.tar.gz
tar xzvf filebeat-7.4.0-linux-x86_64.tar.gz

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.4.0-x86_64.rpm
sudo rpm -vi filebeat-7.4.0-x86_64.rpm

2 filebeat配置文件

grep -v "^ *#" /etc/filebeat/filebeat.yml |grep -v "^$"

[root@node1 filebeat]# grep -v "^[[:space:]]*#" /etc/filebeat/filebeat.yml |grep -v "^$"
filebeat.inputs:
- type: log
  enabled: true
  paths:
    
     - /datalog/service/tiantian-system-service/node2/nohup.out
  encoding: utf-8
  tail_files: true
  
  fields:
    app: www
    type: tiantian-system-service-node2
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  enabled: true
  paths:
     - /datalog/service/tiantian-system-service/node4/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-system-service-node4
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-user-service/node2/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-user-service-node2
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-user-service/node4/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-user-service-node4
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-assets-service/node4/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-assets-service-node4
  fields_under_root: true   
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-assets-service/node2/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-assets-service-node2
  fields_under_root: true 
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-order-service/node2/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-order-service-node2
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-order-service/node4/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-order-service-node4
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-audit-service/node5/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-audit-service-node5
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-audit-service/node6/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-audit-service-node6
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-admin-service/node2/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-admin-service-node2
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-admin-service/node4/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-admin-service-node4
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-collection-service/node5/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-collection-service-node5
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-collection-service/node6/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-collection-service-node6
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-collection-admin/node5/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-collection-admin-node5
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
 
- type: log
  paths:
     - /datalog/service/tiantian-collection-admin/node6/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-collection-admin-node6
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-collection-job/node5/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-collection-job-node5
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-channel-api/node2/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-channel-api-node2
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-channel-api/node4/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-channel-api-node4
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-job/node4/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-job-node4
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-mq-service/node2/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-mq-service-node2
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-mq-service/node4/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-mq-service-node4
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/jz-asset/node1/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: jz-asset-node1
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-channel-admin/node2/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-channel-admin-node2
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-channel-admin/node4/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-channel-admin-node4
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-admin-api/node5/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-admin-api-node5
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
 
 
- type: log
  paths:
     - /datalog/service/tiantian-admin-api/node6/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-admin-api-node6
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-api-v2/node2/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-api-v2-node2
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-api-v2/node4/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-api-v2-node4
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-audit-admin/node5/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-audit-admin-node5
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-audit-admin/node6/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-audit-admin-node6
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/xxd-credit-service/node2/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: xxd-credit-service-node2
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/xxd-jinbaodai-api/node4/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: xxd-jinbaodai-api-node4
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/xxd-jinbaodai-api/node7/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: xxd-jinbaodai-api-node7
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/xxd-jinbaodai-service/node4/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: xxd-jinbaodai-service-node4
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/xxd-jinbaodai-service/node7/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: xxd-jinbaodai-service-node7
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/xxd-ops-service/node5/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: xxd-ops-service-node5
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/xxd-ops-service/node6/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: xxd-ops-service-node6
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-warning/node5/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-warning-node5
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/tiantian-warning/node6/nohup.out
  encoding: utf-8
  tail_files: true
  fields:
    app: www
    type: tiantian-warning-node6
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
- type: log
  paths:
     - /datalog/service/xxd-ops-web/node6/nohup.out
  encoding: utf-8
  tail_files: true
  backoff: 1s
  fields:
    app: www
    type: xxd-ops-web-node5
  fields_under_root: true
  multiline:
     pattern: '^[0-2][0-9]:[0-5][0-9]:[0-5][0-9]'
     negate: true
     match: after
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.logstash:
  hosts: ["172.17.199.231:5044"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

3 配置文件解释

filebeat.prospectors:

input_type: log
paths:

'/mnt/iss/service/order/nlogs/.log' #==监听日志文件全路径 全部监听用，否则直接写具体文件名，也可模糊匹配。多个目录时增加一行配置即可
encoding: utf-8 #==编码格式
exclude_files: [".gz$ | .gc."] #==排除监听的文件

fields:
serverName: 'user-service-006' ##==额外添加的字段，用于区分服务器名称
type: service-log ##==多种服务日志时的区分字段
fields_under_root: true ##==是否直接添加这些字段到日志内容中
scan_frequency: 3s ##==扫描文件的频率
#===== Multiline options
multiline: ##==多行日志的合并配置，用于异常堆栈内容时的处理
pattern: '^{"date":'
negate: true
match: after
timeout: 2s
backoff: 1s
max_backoff: 3s

close_renamed: false ##文件重命名后是否停止监听
close_removed: true ##文件被删除后是否停止监听
tail_files: true ## 是否从文件末尾读取(启动时)
enabled: true
filebeat.spool_size: 2048 ## 事件发送的阀值，超过阀值，强制刷新网络连接
filebeat.idle_timeout: 2s ## 事件发送的超时时间，即使没有超过阀值，也会强制刷新网络连接

ignore_older: 24h ##日志文件监听超时时间阀值

六 加密访问

默认kibana直接可以访问，但是这样不太安全，这里我们采用nginx反向代理，并且设置密码访问

1 安装加密工具

需要安装httpd的密码文件工具
yum -y install httpd-tools
htpasswd -c -b /etc/kibana/kibana.passwd kibana 111111

2 安装nginx并配置

server {
	listen 80;
	server_name kibana.tiantianjiedao.com;
        access_log /var/log/nginx/kibana/kinaba_access.log main;
        error_log /var/log/nginx/kibana/kinaba_error.log;

        auth_basic "Kibana Auth";
        auth_basic_user_file /etc/kibana/kibana.passwd;
	index  index.html index.htm;
	location / {
		proxy_set_header Host      $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_pass http://172.17.199.231:5601;
	}
}

最后启动nginx，直接访问nginxip即可

huningfei
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
容器日志采集利器：Filebeat深度剖析与实践
Docker的专栏
08-01 2369
在云原生时代和容器化浪潮中，容器的日志采集是一个看起来不起眼却又无法忽视的重要议题。对于容器日志采集我们常用的工具有Filebeat和Fluentd，两者对比各有优劣，相...
ELK+Filebeat+Kafka+ZooKeeper构建日志分析平台
01-23
ELK+Filebeat+Kafka+ZooKeeper构建日志分析平台，架构图解
Beats：使用 Filebeat 中的 filestream 输入更快速、更轻松地读取活动日志文件
Elastic 中国社区官方博客
08-16 3884
在 Elastic 7.14 中，filestream input（ log input 的后继者）现在在 Filebeat 中普遍可用。 这种新的、卓越的输入为读取活动日志文件提供了更好的支持，在系统中存在背压时具有更快的反应时间、更快的注册表更新、与外部日志轮换工具的更好合作等等。 改进的注册表性能 以前，当注册表文件（用于保存发布事件进度的文件）包含许多条目时，状态更新会变慢，即使 Filebeat 只收集少数文件。主要问题是完整的注册表在来自输出的每个 ACK​​ 之后被序列化。这需要每次注
ELK日志分析平台之filebeat读取日志
兔纸先森的后花园
07-04 3008
1 软件环境说明本次安装部署所用的软件均为官网上目前的最新版本。操作系统软件Java环境windows 10logstash-6.2.4 jdk 1.8.0_171filebeat-6.3.01.2.2           Elasticsearch安装a.        解压tar包（tar -zxvf elasticsearch-6.2.4.tar.gz）；b.       修改elastic...
使用fileBeat7.9.2读取日志文件到KAFKA
螺蛳粉不加葱的微博
11-10 2535
目录一、背景二、安装三、启用kafka模块四、修改配置四、启用脚本五、格式化springBoot日志格式七、替换@timestamp为日志时间六、遇到的问题六、参考 一、背景 需要将微服务日志传输到elk系统，需要先将日志传到kafka做缓冲； filebeat-7.9.2.tar下载 百度云盘链接：戳我 提取码：iefm 二、安装 tar -zxvf filebeat-7.9.2-linux-x86_64.tar.gz 三、启用kafka模块 进入modules.d文件夹,修改kafka.yml.dis
filebeat采集linux日志,部署filebeat收集nginx日志-Go语言中文社区
weixin_36052776的博客
05-08 280
搭建filebeat自动发现日志在上一篇博客我们部署了logstash去读取日志，但是logstash需要消耗的资源较大。在每台客户端安装logstash不现实。Filebeat是一个轻量级的日志采集器，我们可以使用filebeat去自动发现日志(1)、下载解压filebeat下载filebeat-6.5.4-linux-x86_64.tar.gz，解压(2)、配置filebeat监控nginx日...
ELK7.1.1+FileBeat 收集日志.pdf
07-22
基础ElasticSearch7.1.1版本搭建日志收集框架；采用FileBeat采集log日志，通过logstash过滤后导入ES
Elk+filebeat搭建日志系统1
08-03
1、Filebeat负责收集应用写到磁盘上的日志，并将日志发送给logstash 2、logstash处理来自filebeat的日志，并将处理后的日志保存ela
elk+filebeat+kafka日志系统搭建.docx
07-05
linux环境下搭建全套 elk+filebeat+kafka日志系统 详细步骤。
elk + kafka + filebeat搭建日志分析系统-附件资源
03-05
elk + kafka + filebeat搭建日志分析系统-附件资源
使用 Filebeat 监听日志文件
wml
03-15 946
优势Logstash 主要的有点就是它的灵活性，主要因为它有很多插件，详细的文档以及直白的配置格式让它可以在多种场景下应用。我们基本上可以在网上找到很多资源，几乎可以处理任何问题。劣势Logstash 致命的问题是它的性能以及资源消耗(默认的堆大小是 1GB)。尽管它的性能在近几年已经有很大提升，与它的替代者们相比还是要慢很多的。
轻量级的日志采集组件 Filebeat 讲解与实战操作
最新发布
匠人精神，持之以恒！
09-24 2077
Filebeat是一个轻量级的日志数据收集工具，属于Elastic公司的Elastic Stack（ELK Stack）生态系统的一部分。它的主要功能是从各种来源收集日志数据，将数据发送到Elasticsearch、Logstash或其他目标，以便进行搜索、分析和可视化。轻量级：Filebeat是一个轻量级的代理，对系统资源的消耗非常低。它设计用于高性能和低延迟，可以在各种环境中运行，包括服务器、容器和虚拟机。多源收集。
filebeat读取日志存入elasticsearch
SpringHASh的博客
08-26 547
【代码】filebeat读取日志存入elasticsearch。
玩日志的你不了解 Filebeat ，就像搞结拜不认识关二爷！深度解析 Filebeat 工作原理，轻松玩转大数据！
魏小言的博客
10-11 1502
或翻译 可认为99.999准确实时，但是并不会100十分精确，至少一次发送，存在多发的场景、且日志旋转速度超过时，存在数据丢失、Linux inode 重用也可能导致跳过行发送 Logstash 占用资源高，不适合线上业务部署。 官方文档：https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html Filebeat is a lightweight shipper for forwarding and centr
【Filebeat】
源圈
08-25 1563
文章目录1. Filebeat简介 1. Filebeat简介 Filebeat是 Elasticsearch 公司开发并提供的一款开源轻量级日志收集器，它是Beats 的一种，主要用来收集日志文件数据。   当你启动 Filebeat 时，它将启动一个或多个输入源(inputs)用来监控事先指定的日志文件位置，对于 Filebeat 定位到的每个日志文件，Filebeat 将为其启动一个收集...
日志文件转运工具Filebeat笔记|万字长文
NewTyun的博客
10-23 1994
一、概述与简介Filebeat是一个日志文件转运工具，在服务器上以轻量级代理的形式安装客户端后，Filebeat会监控日志目录或者指定的日志文件，追踪读取这些文件（追踪文...
轻量级日志收集 FileBeat + ElasticSearch
lovo1的专栏
07-29 1121
轻量级日志收集 FileBeat + ElasticSearch 前言 轻量级的服务，未构建独立的spring cloud 体系，如单体spring boot 使用ELK组件进行日志收集，整体过于复杂繁琐，推荐轻量级日志收集框架：spring boot logback json 格式输出 +FileBeat + ElasticSearch +kibana（查询展示也可忽略）。 一、Filebeat是什么？ Filebeat是轻量级日志收集转发插件。作为服务器上的代理安装，Filebeat监视您指定的
ELK 做日志分析(filebeat+logstash+elasticsearch)配置
dianjingsha6726的博客
09-20 321
利用 Filebeat去读取日志发送到 Logstash ,再由 Logstash 处理后发送给 Elasticsearch 。 一、Filebeat 项目日志文件： 利用 Filebeat 去读取文件，paths 下面配置路径地址，Filebeat 会自动去读取 /data/share/business_log/TA-*/debug.log 文件 #================...
Filebeat概述
随风飘雁
07-02 488
Filebeat是用于收集和转发日志数据的轻量级传送程序。 作为服务器上的代理安装，Filebeat监视您指定的日志文件或位置，收集日志事件，并将其转发到Elasticsearch或Logstash进行索引。 Filebeat的工作方式如下：启动Filebeat时，它将启动一个或多个输入，这些输入将在指定的位置中查找日志数据。 对于Filebeat所找到的每个日志，Filebeat都会启动收集器。 每个收集器都读取一个日志以获取新内容，并将新日志数据发送到libbeat，libbeat会汇总事件并将汇总的数
源码安装ELK+filebeat+sentinl
05-19
以下是ELK+filebeat+sentinl的源码安装步骤： 1. 安装Java ELK需要Java运行环境，所以首先需要安装Java。可以在Oracle官网上下载适合自己系统的Java安装包，然后按照提示进行安装。 2. 安装Elasticsearch 在Elasticsearch官网上下载对应系统的安装包，解压后进入bin目录，执行以下命令启动Elasticsearch： ``` ./elasticsearch ``` 3. 安装Kibana 在Kibana官网上下载对应系统的安装包，解压后进入bin目录，执行以下命令启动Kibana： ``` ./kibana ``` 4. 安装Logstash 在Logstash官网上下载对应系统的安装包，解压后进入bin目录，执行以下命令启动Logstash： ``` ./logstash -f logstash.conf ``` 其中，logstash.conf是Logstash的配置文件。 5. 安装Filebeat 在Filebeat官网上下载对应系统的安装包，解压后进入bin目录，编辑filebeat.yml配置文件，配置日志收集的路径和输出到Logstash的地址，然后执行以下命令启动Filebeat： ``` ./filebeat -e -c filebeat.yml ``` 6. 安装Sentinl Sentinl是一个基于Kibana的插件，用于实现告警功能。首先需要安装Kibana插件管理工具elasticsearch-plugin，执行以下命令进行安装： ``` ./kibana-plugin install elasticsearch-plugin ``` 然后再安装Sentinl插件，执行以下命令进行安装： ``` ./kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-6.4.2-0/sentinl-v6.4.2.zip ``` 7. 配置Sentinl 编辑Kibana的配置文件kibana.yml，修改以下配置： ``` sentinl:admin_email: admin@example.com ``` 其中，admin@example.com是管理员邮箱。 8. 启动ELK+filebeat+sentinl 依次启动Elasticsearch、Kibana、Logstash和Filebeat。然后在Kibana中打开Sentinl插件，配置告警规则即可。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值