4、Filebeat

一、Filebeat安装

1、下载解压

tar -zxf filebeat-6.6.0-linux-x86_64.tar.gz
mv filebeat-6.6.0-linux-x86_64 /usr/local/filebeat-6.6.0
mv filebeat-6.6.0-linux-x86_64 /usr/local/filebeat-6.6.0
mv /usr/local/filebeat-6.6.0/filebeat.yml /usr/local/filebeat-6.6.0/filebeat.yml.bak

2、修改配置文件，将日志直接发送到elasticsearch，不经过处理
vim /usr/local/filebeat-6.6.0/filebeat.ym

[root@elk-node2-51 local]# cat /usr/local/filebeat-6.6.0/filebeat.yml
filebeat.inputs:
- type: log
  tail_files: true
  backoff: "1s"
  paths:
      - /usr/local/nginx/logs/access.log

output:
  elasticsearch:
    hosts: ["10.0.0.50:9200"]

3、启动Filebeat

前台启动

/usr/local/filebeat-6.6.0/filebeat  -e -c /usr/local/filebeat-6.6.0/filebeat.yml

后台启动

nohup /usr/local/filebeat-6.6.0/filebeat  -e -c /usr/local/filebeat-6.6.0/filebeat.yml >/tmp/filebeat.log 2>&1 &

4、访问nginx页面然后查看filebeat日志，可以看到连接到了elasticsearch
tail /tmp/filebeat.log
5、在kibana上可以看到有filebeat的索引

6、创建索引然后查看

二、Filebeat+Logstash

Filebeat -> Logstash -> Elasticsearch -> Kibana

1、修改filebeat配置文件，将output改为将日志发送到logstash

vim /usr/local/filebeat-6.6.0/filebeat.yml

[root@elk-node2-51 local]# cat /usr/local/filebeat-6.6.0/filebeat.yml
filebeat.inputs:
- type: log
  tail_files: true
  backoff: "1s"
  paths:
      - /usr/local/nginx/logs/access.log

output:
  logstash:
    hosts: ["10.0.0.51:5044"]

2、修改logstash配置文件，监听5044端口，接收Filebeat发送过来的日志，可以在logstash上对日志进行过滤等处理，然后将日志发送到elasticsearch。

logstash的日志有报错 Could not index event to Elasticsearch，因为host内包含name引起的。暂时的解决办法就是添加配置进行过滤，添加一个下面的过滤器。
vim /usr/local/logstash-6.6.0/config/logstash.conf

[root@elk-node2-51 local]# cat /usr/local/logstash-6.6.0/config/logstash.conf 
input {
  beats {
    host => '0.0.0.0'
    port => 5044
  }
}

filter {
  mutate {
    rename => { "[host][name]" => "host" }
  }
}

output{
  elasticsearch{
    hosts => ["http://10.0.0.50:9200"]
  }
}

然后在kibana查看日志

三、Filebeat采集多个日志

1、filebeat配置介绍
https://www.cnblogs.com/smile361/p/7688545.html
https://blog.csdn.net/a464057216/article/details/51233375

2、修改filebeat配置文件，采集nginx日志和secure登录日志

增加一个fields字段，也可以添加tags标签，在前面K8s有用到
fields:
type: access
fields_under_root: true

vim /usr/local/filebeat-6.6.0/filebeat.yml

[root@elk-node2-51 ~]# cat /usr/local/filebeat-6.6.0/filebeat.yml
filebeat.inputs:
- type: log
  tail_files: true
  backoff: "1s"
  paths:
      - /usr/local/nginx/logs/access.log
  fields:
    type: access
  fields_under_root: true

- type: log
  tail_files: true
  backoff: "1s"
  paths:
      - /var/log/secure
  fields:
    type: secure
  fields_under_root: true

output:
  logstash:
    hosts: ["10.0.0.51:5044"]
    enabled: true

3、Logstash如何判断日志文件

• Filebeat加入一字段用来区别
• Logstash使用区别字段来区分

4、修改Logstash文件
Logstash通过fields中的type字段进行判断

vim /usr/local/logstash-6.6.0/config/logstash.conf

[root@elk-node2-51 ~]# cat /usr/local/logstash-6.6.0/config/logstash.conf
input {
  beats {
    host => '0.0.0.0'
    port => 5044
  }
}

filter {
  mutate {
    rename => { "[host][name]" => "host" }
  }
}

output{

  if [type] == "access" {
    elasticsearch {
      hosts => ["http://10.0.0.50:9200"]
      index => "access-%{+YYYY.MM.dd}"
    }
  }

  if [type] == "secure" {
    elasticsearch {
      hosts => ["http://10.0.0.50:9200"]
      index => "secure-%{+YYYY.MM.dd}"
    }
  }

}

5、重启filebeat和logstash，在kibana上添加access索引和secure索引然后查看