实现elk日志分离

elk实现对日志的分离

修改filebeat服务器filebeat的配置文件如下所示

[root@data-node2 filebeat]# cat filebeat.yml | grep -v "#" | grep -v "^$"

filebeat.prospectors:

- input_type: log

  enabled: true

  paths:

     - /var/log/messages

  document_type: system_messages

  fields:

    logsource: data-node2

    logtype: system_messages

- input_type: log

  enabled: true

  paths:

     - /var/log/*.log

  document_type: system_log

  fields:

    logsource: data-node2

    logtype: system_log

filebeat.config.modules:

  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

setup.template.settings:

  index.number_of_shards: 3

setup.kibana:

output.logstash:

  hosts: ["192.168.101.20:5044","192.168.101.19:5044"]

  enabled: true

修改logstsh的文件logstash.conf文件

[root@data-node1 conf.d]# cat logstash.conf

input {

  beats {

    host => "0.0.0.0"

    port => 5044

  }

}

output {

  if[fields][logtype] == "system_messages" {

  elasticsearch {

    hosts => ["192.168.101.17:9200"]

    index => "messages-%{+YYYY.MM.dd}"

  }

}

  if[fields][logtype] == "system_log" {

  elasticsearch {

    hosts => ["192.168.101.17:9200"]

    index => "log-%{+YYYY.MM.dd}"

  }

}

}

查看是否在es服务器上存储相关日志信息

[root@master-node1 ~]# curl '172.31.125.105:9200/_cat/indices?v' | egrep "log-2018.*|messages" | grep -v "sys"

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100  4182  100  4182    0     0  11861      0 --:--:-- --:--:-- --:--:-- 11880

green  open   messages-2018.12.24         tTcHgT7tSa6jlpMPn_GeJw   5   1         86            0        1mb        559.9kb

green  open   log-2018.12.24              OSB4Zya7Q661EBEAqNpoKg   5   1        657            0      1.3mb        718.7kb