logstash收集nginx日志实战

目的：收集access.log日志文件信息,收集ip, 时间 , 请求行.
nginx数据源例子：
36.157.150.1 - - [07/Jan/2020:12:59:27 +0800] “GET /phpmyadmin_8c1019c9c0de7a0f/js/messages.php?lang=zh_CN&db=&collation_connection=utf8_unicode_ci&token=6a44d72481633c90bffcfd42f11e25a1 HTTP/1.1” 200 8131 “-” "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.3636.157.150.1 - - [07/Jan/2020:12:59:27 +0800] “GET /phpmyadmin_8c1019c9c0de7a0f/js/messages.php?lang=zh_CN&db=&collation_connection=utf8_unicode_ci&token=6a44d72481633c90bffcfd42f11e25a1 HTTP/1.1” 200 8131 “-” "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36

配置文件
vi from_nginx.conf

input{
file{
path => “/xiaozhang/datas/access.log”
type => “log”
start_position => “beginning”
}
}

filter {
grok {

match => {
“message” => “%{IPORHOST:clientip} - - [%{HTTPDATE:time_local}] “(?:%{WORD:method} %{NOTSPACE:request}(?:HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})” %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS:http_referer} %{QS:agent}”
}
}
}

output{
file{
path => “/xiaozhang/datas/%{+YYYY-MM-dd}-%{host}.txt”
codec=>line{
format=>"%{clientip}|%{time_local}|%{rawrequest}"
}
flush_interval => 0
}
}

启动logstash

bin/logstash -f config/from_nginx.conf