ELK+redis+filebeat配置nginx日志及geoip

添加的地方

log_format json '{ “@timestamp”: “KaTeX parse error: Double superscript at position 22: …so8601", ' '̲"remote_addr": …remote_addr”, ’
'“remote_user”: “KaTeX parse error: Double superscript at position 21: …e_user", ' '̲"body_bytes_sen…body_bytes_sent”, ’
'“request_time”: “KaTeX parse error: Double superscript at position 22: …t_time", ' '̲"status": "status”, ’
'“request_uri”: “KaTeX parse error: Double superscript at position 21: …st_uri", ' '̲"request_method…request_method”, ’
'“http_referrer”: “KaTeX parse error: Double superscript at position 22: …eferer", ' '̲"http_x_forward…http_x_forwarded_for”, ’
‘“http_user_agent”: “$http_user_agent”}’;
access_log logs/access.log json;

nginx启动命令

/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf

编写filebeat的收集文件filebeat-nginx.yaml，在里面指定收集日志的路径和将路径存储在redis的位置

filebeat.prospectors:

• List item

type: log paths:
- /usr/local/nginx/logs/access.log # tags: [“access”] fields:
app: www
type: nginx-access fields_under_root: true

• type: log paths:
  • /usr/local/nginx/logs/error.log # ags: [“error”] fields:
    app: www
    type: nginx-error fields_under_root: true

output.redis: hosts: [“192.168.88.101:6382”] key: “filebeat” db: 0 datatype: list

filebeat启动命令

./filebeat -e -c filebeat-nginx.yaml 1 编写logstash的配置文件logstash-to-es-nginxjson.conf，在此指定输入和输出的地址，以及对数据进行过滤处理

input {
redis {
host => “192.168.88.101”
port => 6382
db => “0”
data_type => “list”
key => “filebeat”
} }

filter { if [app] == “www” {
if [type] == “nginx-access” {
grok {
match => {
“message” => “%{IPV4:remote_addr} - (%{USERNAME:remote_user}|-) [%{HTTPDATE:time_local}] “%{WORD:request_method} %{URIPATHPARAM:request_uri} HTTP/%{NUMBER:http_protocol}” %{NUMBER:http_status} %{NUMBER:body_bytes_sent} “%{GREEDYDATA:http_referer}” “%{GREEDYDATA:http_user_agent}” “(%{IPV4:http_x_forwarded_for}|-)””
}
overwrite => [“message”]
}
geoip {
source => “remote_addr”
target => “geoip”
database => “/haoke/beats/GeoLite2-City_20200324/GeoLite2-City.mmdb”
add_field => ["[geoip][coordinates]", “%{[geoip][longitude]}”]
add_field => ["[geoip][coordinates]", “%{[geoip][latitude]}”]
}
date {
locale => “en”
match => [“time_local”, “dd/MMM/yyyy:HH:mm:ss Z”]
}
mutate {
convert => ["[geoip][coordinates]", “float”]
}
} } }

output { elasticsearch {
hosts => [“http://192.168.88.101:9200”,“http://192.168.88.101:9201”]
index => “logstash-%{type}-%{+YYYY.MM.dd}” } stdout{codec => rubydebug } }

注意：在上面的配置中需要指定geoip的数据库，如果在网上通过http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz下载不到，可以从我的网盘下载，解压，将路径拷贝过去即可

链接：https://pan.baidu.com/s/1EtDZ4AYKhPkWyuD-6FrPcw 提取码：y8w7 logstash启动时，可以先启动测试下，如果配置不报错，则执行启动命令
#启动 --config.test_and_exit 用于测试配置文件是否正确

bin/logstash -f logstash-to-es-nginxjson.conf --config.test_and_exit
#正式启动 --config.reload.automatic 热加载配置文件，修改配置文件后无需重新启动 bin/logstash -f logstash-to-es-nginxjson.conf --config.reload.automatic filebeat，redis，elk都启动完成即可访问检查效果