02-elk创建多个日志索引---filebeat配置文件

A,不修改日志格式的收集配置：

filebeat配置文件：

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
output.elasticsearch:
  hosts: ["10.4.7.11:9200"]

B,收集日志（单个日志并修改日志索引名称）

1，在配置文件中 ，修改索引名称，并修改待收集软件的日志格式nginx

1，filebeat配置文件：

/etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true 
    
output.elasticsearch:
   hosts: ["http://10.4.7.11:9200"]
   index: "nginx-access-%{[agent.version]}-%{+yyyy.MM}"
logging.level:   info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644 
setup.template.enabled: false
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.ilm.enabled: false   
      

2，修改nginx的日志文件格式（log_format）：

在http模块下修改nginx生产的日志格式：
log_format json '{ "time_local": "$time_local", '
                          '"remote_addr": "$remote_addr", '
                          '"referer": "$http_referer", '
                          '"request": "$request", '
                          '"status": $status, '
                          '"bytes": $body_bytes_sent, '
                          '"agent": "$http_user_agent", '
                          '"x_forwarded": "$http_x_forwarded_for", '
                          '"up_addr": "$upstream_addr",'
                          '"up_host": "$upstream_http_host",'
                          '"upstream_time": "$upstream_response_time",'
                          '"request_time": "$request_time"'
    ' }';
    access_log  /var/log/nginx/access.log  json;

如示例：

在一个配置文件中收集多个日志文件
方法一：

cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true 
    
- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  json.keys_under_root: true
  json.overwrite_keys: true 
- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log  
output.elasticsearch:
   hosts: ["http://10.4.7.11:9200"]
   indices:
     - index: "nginx-access-%{[agent.version]}-%{+yyyy.MM}"
       when.contains:
         log.file.path: "/var/log/nginx/access.log"           ######特别是log.file.path的由来
     - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM.}"
       when.contains:
         log.file.path: "/var/log/nginx/error.log"            ######特别是log.file.path的由来 
logging.level:   info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644 
setup.template.enabled: false
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.ilm.enabled: false
EOF

方法二：打个标签

打个标签：
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true 
  tags: ["access"]
    
- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]
  json.keys_under_root: true
  json.overwrite_keys: true 
- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log  
processors:               ###删除无用的字段
  - drop_fields:
      fields: ["ecs","log","log.offset"]   
output.elasticsearch:
   hosts: ["http://10.4.7.11:9200"]
   indices:
     - index: "nginx-access-%{[agent.version]}-%{+yyyy.MM}"
       when.contains:
          tags: "access"
     - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM.}"
       when.contains:
          tags: "error"
logging.level:   info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644 
setup.template.enabled: false
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.ilm.enabled: false

删除无用的字段：

processors:
  - drop_fields:
      fields: ["ecs","log","log.offset"]

无用字段删除前与删除后的比较：

C,在不修改nginx默认日志格式的情况下，用grok收集nginx日志，

不需要修改日志的模式，通过定义grok 来收集日志
普通格式的日志，转换成json格式的形式

%{IP:clientip} - - \[%{HTTPDATE:nginx.access.time}\] \"%{DATA:nginx.access.info}\" %{NUMBER:http.response.status_code:log} %{NUMBER:http.response.boby.bytes:log} \"(-|%{DATA:http.request.referrer})\" \"(-|%{DATA:user_agent.original})\" \"(-|%{IP:http_x_forwarded_for})\"

在elasticsearch 中创建一个grok

PUT _ingest/pipeline/pipeline-nginx-access
{
  "description": "nginx access log",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{IP:clientip} - - \\[%{HTTPDATE:nginx.access.time}\\] \"%{DATA:nginx.access.info}\" %{NUMBER:http.response.status_code:log} %{NUMBER:http.response.boby.bytes:log} \"(-|%{DATA:http.request.referrer})\" \"(-|%{DATA:user_agent.original})\" \"(-|%{IP:http_x_forwarded_for})\""]
      }
    },{
      "remove": {
        "field": "message"
      }
    }
  ]
}

3，修改filebeat配置文件

cat /etc/filebeat/filebeat.yml |egrep -v "#|^$"
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  tags: ["access"]
- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]
processors:
  - drop_fields:
      fields: ["ecs","log"]
output.elasticsearch:
   hosts: ["http://10.4.7.11:9200"]
   pipelines:
    - pipeline: "pipeline-nginx-access"
      when.contains:
        tags: "access"
   indices:
     - index: "nginx-access-%{[agent.version]}-%{+yyyy.MM}"
       when.contains:
         tags: "access"
     - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM.}"
       when.contains:
         tags: "error"
setup.template.enabled: false
setup.ilm.enabled: false

logging.level:   info
logging.to_files: true

D:利用官方的module来进行日志收集的配置工作

首先配置filebeat配置文件

filebeat.config.modules:      #这个指定了模块的位置，并激活模块功能
  path: ${path.config}/modules.d/*.yml
  reload.enabled: enable        
filebeat.modules:                   
- module: nginx
output.elasticsearch:
   hosts: ["http://10.4.7.11:9200"]
   indices:
     - index: "nginx-access-%{[agent.version]}-%{+yyyy.MM}"
       when.contains:
     - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM.}"
       when.contains:
setup.template.enabled: false
setup.ilm.enabled: false

其次：修改filebeat的modules设置

cd /usr/share/filebeat/module/        ####  这个是filebeat的模板文件所在地
filebeat modules list
filebeat modules enable nginx    #### 激活nginx模板
# 在当前目录下编辑nginx.yml
vim nginx.yml
var.paths: ["/var/log/nginx/access.log"]
var.paths: ["/var/log/nginx/error.log"]  # 修改这两个地址
# 有一个报错算是可能是bug
在/usr/share/filebeat/module/nginx/access下有个ingress_controller目录，配置文件已经禁止了但还是可以开启，需要把这个文件删除掉。
mv  ingress_controller  /opt/   #移除   一劳永逸

最后就是清空日志文件，重启filebeat服务

echo > /var/log/nginx/access.log
echo > /var/log/nginx/error.log
systemctl restart filebeat