[问题已处理]-[elk]-filebeat收集json格式的nginx日志

filebeat收集json格式的nginx日志

 

由于画蛇添足 用了下面的配置导致nginx的json日志一直显示在message里

log_format access_json '{ "@timestamp": "$time_local", ' '"remote_addr": "$remote_addr", ' '"referer": "$http_referer", ' '"request": "$request", ' '"status": $status, ' '"bytes": $body_bytes_sent, ' '"agent": "$http_user_agent", ' '"x_forwarded": "$http_x_forwarded_for", ' '"up_addr": "$upstream_addr",' '"up_host": "$upstream_http_host",' '"up_resp_time": "$upstream_response_time",' '"request_time": "$request_time"' '"remote_user ": "$remote_user "' '"http_host ": "$http_host "' '"upstream_status": "$upstream_status"' '"http_referer": "$http_referer"' '"ssl_protocol": "$ssl_protocol"' '"ssl_cipher  ": "$ssl_cipher  "' ' }';

 

下面是nginx输出

导致filebeat报错

修改nginx.conf中https模块

log_format access_json '{ "@timestamp": "$time_iso8601", '                          '"time": "$time_iso8601", '                          '"remote_addr": "$remote_addr", '                          '"remote_user": "$remote_user", '                          '"body_bytes_sent": "$body_bytes_sent", '                          '"request_time": "$request_time", '                          '"status": "$status", '                          '"host": "$host", '                          '"request": "$request", '                          '"request_method": "$request_method", '                          '"uri": "$uri", '                          '"http_referrer": "$http_referer", '                          '"body_bytes_sent":"$body_bytes_sent", '                          '"http_x_forwarded_for": "$http_x_forwarded_for", '                          '"http_user_agent": "$http_user_agent" '                     '}';

下面是nginx输出

然后修改filebeat.yml

加入 json.keys_under_root: true
     json.overwrite_keys: true

json.keys_under_root： 默认这个值是FALSE的，也就是我们的json日志解析后会被放在json键上。设为TRUE，所有的keys就会被放到根节点
json.overwrite_keys: 是否要覆盖原有的key，这是关键配置，将keys_under_root设为TRUE后，再将overwrite_keys也设为TRUE，就能把filebeat默认的key值给覆盖了

filebeat.prospectors: - input_type: log  paths:    - /var/log/nginx/access.log  json.keys_under_root: true  json.overwrite_keys: true

 

然后reload nginx  重启filebeat 之后的结果