ELK logstash json对你的nginx进行解码

收集Nginx访问日志

---

将Nginx访问日志格式改为JSON

log_format json '{ "@timestamp": "$time_iso8601", '
'"remote_addr": "$remote_addr", '
'"remote_user": "$remote_user", '
'"body_bytes_sent": "$body_bytes_sent", '
'"request_time": "$request_time", '
'"status": "$status", '
'"request_uri": "$request_uri", '
'"request_method": "$request_method", '
'"http_referrer": "$http_referer", '
'"http_x_forwarded_for": "$http_x_forwarded_for", '
'"http_user_agent": "$http_user_agent"}';

 access_log  /var/log/nginx/access.log json;

 Logstash再使用JSON过滤插件解析：

这里filebeat采集的就是json格式，不需要特殊配置filebeat,在logstash这里还要使用json插件

[root@localhost ~]# cat /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/*.log
  tags: ["nginx"]
  fields_under_root: true
  fields:
    project: microservice
    app: product

output.logstash:
  hosts: ["192.168.179.102:5044"]

[root@localhost ~]# cat /usr/local/logstash/conf.d/test.conf
input {
  beats {
  host => "0.0.0.0"
  port => 5044
 }
}

filter {
  json {
    source => "message"
}
  if [app] == "product" {
    mutate { 
      add_field => { 
        "[@metadata][target_index]" => "microservice-product-%{+YYYY.MM}" 
   } 
  }
} else if [app] == "gateway" {
    mutate { 
      add_field => { 
        "[@metadata][target_index]" => "microservice-gateway-%{+YYYY.MM.dd}" 
   } 
  }
} else {
    mutate { 
      add_field => { 
        "[@metadata][target_index]" => "unknown-%{+YYYY}" 
   } 
  } 
 } 
}

output {
  elasticsearch {
    hosts => "192.168.179.102:9200"
    index => "%{[@metadata][target_index]}"
 } 
}

 现在就可以对日志的字段做可视化统计