1)nginx日志格式
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
}
2)nginx服务器安装logstash
链接:https://pan.baidu.com/s/1bSYnZwc3zaSuh0cddP8QKg
提取码:7dk6
yum install logstash-7.0.0.rpm -y
3)准备nginx正则文件
#cat /etc/logstash/patterns/nginx
WZ ([^ ]*)
NGINXACCESS %{IP:remote_ip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{WZ:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} %{NUMBER:bytes} %{QS:referer} %{QS:agent} %{QS:xforward}
4)客户端的logstash配置文件内容如下
# cat /etc/logstash/logstash.conf
input {
file {
path => [ "/var/log/nginx/access.log" ]
start_position => "beginning"
ignore_older => 0
type => "nginx"
}
}
filter{
grok {
patterns_dir => "/etc/logstash/patterns/nginx" #自定义正则
match => { "message" => "%{NGINXACCESS}" }
}
date {
match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
target => "@timestamp"
}
mutate {
remove_field => ["timestamp"]
}
geoip {
source => "remote_ip"
}
}
output {
if [type] == "nginx" {
redis {
host => "192.168.1.42"
password => 'redis@2019'
port => "6379"
db => "2"
data_type => "list"
key => 'nginx'
}
}
stdout { codec => rubydebug }
}
input插件的file 表示从文件中读取内容
filter插件的grok插件作用是将非结构化数据转换成结构化数据
filter插件的date插件可以从日志消息中提取时间和日期,并将其定义为日志的时间戳字段(@timestamp) 一旦定义 这个时间戳字段将以正确的时间顺序排列日志
remove_field表示插件timestamp时间戳字段
geoip插件能够抓取ip的归属地信息
output插件这里我输出到redis中
5)服务器端的logstash配置如下
# cat /etc/logstash/logstash.conf
input{
redis {
type => "nginx"
host => "192.168.1.42"
port => "6379"
password => 'redis@2019'
db => '2'
data_type => "list"
key => 'nginx'
}
}
output {
if [type] == "nginx" {
elasticsearch {
hosts => ['elasticsearch:9200']
index => 'nginx-%{+YYYY.MM.dd}'
}
}
}
6)创建图形
pv: visualize --> metric
uv: visualize --> metric
状态码排行: visualize --> data table
接口请求排行:visualize --> data table
ip地址接口请求排行:visualize --> data table
最后创建一个dashborad