首先安装nginx,不会安装的可以参考我的文章https://blog.csdn.net/weixin_39614177/article/details/108790113
修改nginx的配置文件nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format json '{ "@timestamp": "$time_iso8601", '
'"remote_addr": "$remote_addr", '
'"remote_user": "$remote_user", '
'"body_bytes_sent": "$body_bytes_sent", '
'"request_time": "$request_time", '
'"status": "$status", '
'"request_uri": "$request_uri", '
'"request_method": "$request_method", '
'"http_referrer": "$http_referer", '
'"http_x_forwarded_for": "$http_x_forwarded_for", '
'"http_user_agent": "$http_user_agent"}';
access_log logs/access.log json;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
server {
listen 80;
server_name localhost;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
root html;
index index.html index.htm;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
添加的地方
log_format json '{ "@timestamp": "$time_iso8601", '
'"remote_addr": "$remote_addr", '
'"remote_user": "$remote_user", '
'"body_bytes_sent": "$body_bytes_sent", '
'"request_time": "$request_time", '
'"status": "$status", '
'"request_uri": "$request_uri", '
'"request_method": "$request_method", '
'"http_referrer": "$http_referer", '
'"http_x_forwarded_for": "$http_x_forwarded_for", '
'"http_user_agent": "$http_user_agent"}';
access_log logs/access.log json;
nginx启动命令
/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
编写filebeat的收集文件filebeat-nginx.yaml,在里面指定收集日志的路径和将路径存储在redis的位置
filebeat.prospectors:
- type: log
paths:
- /usr/local/nginx/logs/access.log
# tags: ["access"]
fields:
app: www
type: nginx-access
fields_under_root: true
- type: log
paths:
- /usr/local/nginx/logs/error.log
# ags: ["error"]
fields:
app: www
type: nginx-error
fields_under_root: true
output.redis:
hosts: ["192.168.88.101:6382"]
key: "filebeat"
db: 0
datatype: list
filebeat启动命令
./filebeat -e -c filebeat-nginx.yaml
编写logstash的配置文件logstash-to-es-nginxjson.conf,在此指定输入和输出的地址,以及对数据进行过滤处理
input {
redis {
host => "192.168.88.101"
port => 6382
db => "0"
data_type => "list"
key => "filebeat"
}
}
filter {
if [app] == "www" {
if [type] == "nginx-access" {
grok {
match => {
"message" => "%{IPV4:remote_addr} - (%{USERNAME:remote_user}|-) \[%{HTTPDATE:time_local}\] \"%{WORD:request_method} %{URIPATHPARAM:request_uri} HTTP/%{NUMBER:http_protocol}\" %{NUMBER:http_status} %{NUMBER:body_bytes_sent} \"%{GREEDYDATA:http_referer}\" \"%{GREEDYDATA:http_user_agent}\" \"(%{IPV4:http_x_forwarded_for}|-)\""
}
overwrite => ["message"]
}
geoip {
source => "remote_addr"
target => "geoip"
database => "/haoke/beats/GeoLite2-City_20200324/GeoLite2-City.mmdb"
add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}"]
add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}"]
}
date {
locale => "en"
match => ["time_local", "dd/MMM/yyyy:HH:mm:ss Z"]
}
mutate {
convert => ["[geoip][coordinates]", "float"]
}
}
}
}
output {
elasticsearch {
hosts => ["http://192.168.88.101:9200","http://192.168.88.101:9201"]
index => "logstash-%{type}-%{+YYYY.MM.dd}"
}
stdout{codec => rubydebug }
}
注意:在上面的配置中需要指定geoip的数据库,如果在网上通过http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
下载不到,可以从我的网盘下载,解压,将路径拷贝过去即可
链接:https://pan.baidu.com/s/1EtDZ4AYKhPkWyuD-6FrPcw
提取码:y8w7
logstash启动时,可以先启动测试下,如果配置不报错,则执行启动命令
#启动 --config.test_and_exit 用于测试配置文件是否正确
bin/logstash -f logstash-to-es-nginxjson.conf --config.test_and_exit
#正式启动 --config.reload.automatic 热加载配置文件,修改配置文件后无需重新启动
bin/logstash -f logstash-to-es-nginxjson.conf --config.reload.automatic
filebeat,redis,elk都启动完成即可访问检查效果