EFK收集nginx日志，kibana可视化统计访问前10IP，PV、UV数据，配置nginx正则匹配，手把手教会每一位新兴的程序猿~@^Y^

在上一篇的基础上 【153端】操作

1.安装epel和ab压测

yum -y install epel-release  httpd-tools

2.安装nginx并启动

yum -y install nginx
nginx

3.验证端口

ss -nlpt |grep 80

4.编辑filebeat配置文件

vim /etc/filebeat/filebeat.yml

内容如下:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/messages
  fields:
    log_topics: msg007
###nginx##
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  fields:
    log_topics: nginx007

output.kafka:
  enabled: true
  hosts: ["192.168.112.153:9092","192.168.112.154:9092","192.168.112.155:9092"]
  topic: '%{[fields][log_topics]}'

5. 启动filebeat

systemctl  restart filebeat

6.添加nginx配置文件【154端】

vim /etc/logstash/conf.d/nginx.conf

内容如下:

input{
        kafka{
                bootstrap_servers => ["192.168.10.130:9092,192.168.10.131:9092,192.168.10.132:9092"]
                group_id => "logstash"
                topics => "nginx007"
                consumer_threads => 5
        }

}
filter {
        json{
                source => "message"
        }

        mutate {
                remove_field => ["host","prospector","fields","input","log"]
        }
        grok {
                match => { "message" => "%{NGX}" }
        }

}

output{
        elasticsearch {
                hosts => "192.168.10.130:9200"
                index => "nginx-%{+YYYY.MM.dd}"
            }      
}

7.修改管道【化一为二】

vim /etc/logstash/pipelines.yml

8.编写正则匹配

路径

vim /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/nginx

内容

NGX %{IPORHOST:client_ip} (%{USER:ident}|- ) (%{USER:auth}|-) \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} (%{NOTSPACE:request}|-)(?: HTTP/%{NUMBER:http_version})?|-)" %{NUMBER:status} (?:%{NUMBER:bytes}|-) "(?:%{URI:referrer}|-)" "%{GREEDYDATA:agent}"

9.重启/验证

systemctl restart logstash
ss -nlpt|grep 9600

10.【153端】

触发生成数据

chmod -R  777 /var/log/*                              【加权限】
 ab -n 100 -c 100 http://1923168.112.153/index.html  【压测数据】

11.登录测试

返回上一层