Logstash+Redis+Elasticsearch+Kibana Nginx日志查询系统
环境
elasticsearch-0.90.5.zip
kibana-latest.zip
redis-2.6.16.tar.gz
logstash-1.2.2-flatjar.jar
nginx.conf配置
log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" ';
nginx日志
172.16.201.174 - - [25/Mar/2014:16:39:13 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1772.0 Safari/537.36"
表达式
%{IPORHOST:source_ip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}
collection
hadoop@stormspark:~/log/logstash$ cat sp.conf
input {
file {
type => "nginx-access"
path => "/var/log/nginx/access.log"
}
}
output {
stdout {
debug => true
debug_format => json
}
redis {
host => "127.0.0.1"
port => 6379
data_type => "list"
key => "logstash"
}
}
index配置
hadoop@stormspark:~/log/logstash$ cat index.conf
input {
redis {
host => "127.0.0.1"
port => "6379"
data_type => "list"
key => "logstash"
type => "redis-input"
}
}
filter {
grok {
type => "nginx-access"
pattern => "%{IPORHOST:source_ip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}"
}
}
output {
elasticsearch {
host => "127.0.0.1"
}
}
分别启动logstash,redis,es等。
java -jar logstash-1.2.2-flatjar.jar agent -f sp.conf
java -jar logstash-1.2.2-flatjar.jar agent -f index.conf
最后来个截图: