filebeat+redis+elk实现nginx日志收集

首先安装nginx，不会安装的可以参考我的文章https://blog.csdn.net/weixin_39614177/article/details/108790113
修改nginx的配置文件nginx.conf

worker_processes  1;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                     '"$http_user_agent" "$http_x_forwarded_for"';

	log_format json '{ "@timestamp": "$time_iso8601", '
			 '"remote_addr": "$remote_addr", '
			 '"remote_user": "$remote_user", '
			 '"body_bytes_sent": "$body_bytes_sent", '
			 '"request_time": "$request_time", '
			 '"status": "$status", '
			 '"request_uri": "$request_uri", '
			 '"request_method": "$request_method", '
			 '"http_referrer": "$http_referer", '
			 '"http_x_forwarded_for": "$http_x_forwarded_for", '
			 '"http_user_agent": "$http_user_agent"}';
			 
    access_log  logs/access.log  json;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    server {
        listen       80;
        server_name  localhost;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            root   html;
            index  index.html index.htm;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

添加的地方

log_format json '{ "@timestamp": "$time_iso8601", '
			 '"remote_addr": "$remote_addr", '
			 '"remote_user": "$remote_user", '
			 '"body_bytes_sent": "$body_bytes_sent", '
			 '"request_time": "$request_time", '
			 '"status": "$status", '
			 '"request_uri": "$request_uri", '
			 '"request_method": "$request_method", '
			 '"http_referrer": "$http_referer", '
			 '"http_x_forwarded_for": "$http_x_forwarded_for", '
			 '"http_user_agent": "$http_user_agent"}';		 
    access_log  logs/access.log  json;

nginx启动命令

/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf

编写filebeat的收集文件filebeat-nginx.yaml，在里面指定收集日志的路径和将路径存储在redis的位置

filebeat.prospectors:
- type: log
  paths:
    - /usr/local/nginx/logs/access.log
  # tags: ["access"]
  fields:
    app: www
    type: nginx-access
  fields_under_root: true
  
- type: log
  paths:
    - /usr/local/nginx/logs/error.log
  # ags: ["error"]
  fields:
    app: www
    type: nginx-error
  fields_under_root: true

output.redis:
  hosts: ["192.168.88.101:6382"]
  key: "filebeat"
  db: 0
  datatype: list

filebeat启动命令

./filebeat -e -c filebeat-nginx.yaml

编写logstash的配置文件logstash-to-es-nginxjson.conf，在此指定输入和输出的地址，以及对数据进行过滤处理

input {
    redis {
        host => "192.168.88.101"
        port => 6382
        db => "0"
        data_type => "list"
        key => "filebeat"
    }
}

filter {
  if [app] == "www" {
    if [type] == "nginx-access" {
      grok {
        match => {
          "message" => "%{IPV4:remote_addr} - (%{USERNAME:remote_user}|-) \[%{HTTPDATE:time_local}\] \"%{WORD:request_method} %{URIPATHPARAM:request_uri} HTTP/%{NUMBER:http_protocol}\" %{NUMBER:http_status} %{NUMBER:body_bytes_sent} \"%{GREEDYDATA:http_referer}\" \"%{GREEDYDATA:http_user_agent}\" \"(%{IPV4:http_x_forwarded_for}|-)\""
        }
        overwrite => ["message"]
      }
      geoip {
          source => "remote_addr"
          target => "geoip"
          database => "/haoke/beats/GeoLite2-City_20200324/GeoLite2-City.mmdb"
          add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}"] 
          add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}"]
      }
      date {
          locale => "en"
          match => ["time_local", "dd/MMM/yyyy:HH:mm:ss Z"]
      }
      mutate {
          convert => ["[geoip][coordinates]", "float"]  
      }
    }
  }
}

output {
  elasticsearch {
      hosts  => ["http://192.168.88.101:9200","http://192.168.88.101:9201"]
      index  => "logstash-%{type}-%{+YYYY.MM.dd}"
  }
  stdout{codec => rubydebug }
}

注意：在上面的配置中需要指定geoip的数据库，如果在网上通过http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz下载不到，可以从我的网盘下载，解压，将路径拷贝过去即可

链接：https://pan.baidu.com/s/1EtDZ4AYKhPkWyuD-6FrPcw 
提取码：y8w7 

logstash启动时，可以先启动测试下，如果配置不报错，则执行启动命令
#启动 --config.test_and_exit 用于测试配置文件是否正确

bin/logstash -f logstash-to-es-nginxjson.conf --config.test_and_exit

#正式启动 --config.reload.automatic 热加载配置文件，修改配置文件后无需重新启动

bin/logstash -f logstash-to-es-nginxjson.conf --config.reload.automatic

filebeat，redis，elk都启动完成即可访问检查效果