使用kibana分析nginx日志

1）nginx日志格式

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
}

2)nginx服务器安装logstash

链接：https://pan.baidu.com/s/1bSYnZwc3zaSuh0cddP8QKg 
提取码：7dk6 
yum install logstash-7.0.0.rpm -y 

3）准备nginx正则文件

#cat /etc/logstash/patterns/nginx 
WZ ([^ ]*)
NGINXACCESS %{IP:remote_ip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{WZ:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} %{NUMBER:bytes} %{QS:referer} %{QS:agent} %{QS:xforward}

4）客户端的logstash配置文件内容如下

# cat /etc/logstash/logstash.conf 
input {
    file {
        path => [ "/var/log/nginx/access.log" ]
        start_position => "beginning"
        ignore_older => 0
                type => "nginx"
    }
}


filter{
        grok {
                patterns_dir => "/etc/logstash/patterns/nginx"   #自定义正则
                match => { "message" => "%{NGINXACCESS}" }
        }
        date {
                match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
                target => "@timestamp"
        }
        mutate {
                remove_field => ["timestamp"]
        }
        geoip {
                source => "remote_ip"
        }
}

output {
    if [type] == "nginx" {
        redis {
            host => "192.168.1.42"
            password => 'redis@2019'
            port => "6379"
            db => "2"
            data_type => "list"
            key => 'nginx'
        }
    }
   stdout { codec => rubydebug }
   }

input插件的file 表示从文件中读取内容
filter插件的grok插件作用是将非结构化数据转换成结构化数据
filter插件的date插件可以从日志消息中提取时间和日期，并将其定义为日志的时间戳字段（@timestamp） 一旦定义 这个时间戳字段将以正确的时间顺序排列日志
remove_field表示插件timestamp时间戳字段
geoip插件能够抓取ip的归属地信息
output插件这里我输出到redis中

5）服务器端的logstash配置如下

# cat /etc/logstash/logstash.conf
input{
   redis {
         type => "nginx"
                    host => "192.168.1.42"
                    port => "6379"
                    password => 'redis@2019'
                    db => '2'
                    data_type => "list"
                    key => 'nginx'
            }

}

output {
      if [type] ==  "nginx" {
        elasticsearch {
                hosts => ['elasticsearch:9200']
                index => 'nginx-%{+YYYY.MM.dd}'
        }
    }
    }

6）创建图形
pv: visualize --> metric

uv: visualize --> metric

状态码排行： visualize --> data table

接口请求排行:visualize --> data table

ip地址接口请求排行:visualize --> data table

最后创建一个dashborad