20.ELK-filebeat模块功能收集nginx的普通格式日志

filebeat收集nginx的普通格式日志

开通nginx的模块功能
1.开通nginx的模块功能

filebeat modules enable nginx

#使用最简单的方式
cd /etc/filebeat/modules.d/
mv nginx.yml.disabled nginx.yml

2、检查是否开启成功
filebeat modules list

修改nginx模块的配置文件
cd /etc/filebeat/modules.d/
vim nginx.yml

把nginx的日志格式还原成普通的日志格式
1、把之前的老日志清空一下
cat /dev/null > /var/log/nginx/access.log
2、修改nginx的主配置文件(日志格式应用为原来的老模板)
vim /etc/nginx/nginx.conf
access_log /var/log/nginx/access.log main;
3、检查nginx配置文件是否有误
nginx -t
4、重启nginx服务
systemctl restart nginx
5、重新产生一些访问日志

修改filebeat的主配置文件
vim /etc/filebeat/filebeat.yml

output.elasticsearch:
  hosts: ["192.168.51.189"]
  indices:
    - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        source: "/var/log/nginx/access.log"
    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        source: "/var/log/nginx/error.log"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

在head界面，清除之前的老索引信息
重启filebeat的服务
systemctl restart filebeat

发现filebeat无法收集nginx的日志,报错如下，需要在es服务器上安装插件才行，如果是es集群的话，每个节点都要安装插件才行

2020-04-20T12:54:00.848+0800    ERROR   pipeline/output.go:100  
Failed to connect to backoff(elasticsearch(http://192.168.51.189:9200)): 
Connection marked as failed because the onConnect callback failed: Error l
oading pipeline for fileset nginx/access: This module requires the following 
Elasticsearch plugins: ingest-user-agent, ingest-geoip. 
You can install them by running the following commands on all the Elasticsearch nodes:
sudo bin/elasticsearch-plugin install ingest-user-agent
sudo bin/elasticsearch-plugin install ingest-geoip

上传两个插件的安装包到该目录
cd /opt/es-software

#安装这两个插件
/usr/share/elasticsearch/bin/elasticsearch-plugin install file:///opt/es-software/ingest-user-agent-6.6.0.zip

/usr/share/elasticsearch/bin/elasticsearch-plugin install file:///opt/es-software/ingest-geoip-6.6.0.zip

重启es的服务
systemctl restart elasticsearch
在head插件界面检查索引是否生成