Logstash 处理json格式日志文件

处理nginx日志输出到docker容器日志

ex：

{"log":"192.168.10.20 - - [19/Jun/2018:09:08:07 +0000] \"POST /backend/web/index.php?r=products/care/assessment/quick-assessment/assessment-info HTTP/1.1\" 200 60 \"-\" \"PlanMaker/4.3.4 (iPad; iOS 11.4; Scale/2.00)\" \"-\"\n",
"stream":"stdout",
"time":"2018-06-19T09:08:07.929679127Z"}

logstash配置文件：

1、自定义字段
input {
       stdin{
        codec => json
          }
}

filter {
      grok {
       match => { "log" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}"
       }
    }
      mutate {
       remove_field => "log"
           }
}

output {

      stdout {
       codec => rubydebug
     }
}

运行结果：

2、采用系统自带的格式化模板
input {
      stdin{
        codec => json
          }
}

filter {
      grok {
       match => { "log" => "%{HTTPD_COMMONLOG} %{QS:x_forwarded_for}" }
    }
      mutate {
       remove_field => "log"
           }
}

output {
     stdout {
       codec => rubydebug
     }
}

运行结果：