同步nginx日志到elasticsearch

编写nginx的配置文件 json格式

log_format access_json 
 '{"@timestamp":"$time_iso8601",'
 '"host":"$server_addr",'
 '"request_method": "$request_method", '
 '"clientip":"$remote_addr",'
 '"size":$body_bytes_sent,'
 '"responsetime":$request_time,'
 '"upstreamtime":"$upstream_response_time",'
 '"upstreamhost":"$upstream_addr",'
 '"http_host":"$host",'
 '"url":"$uri",'
 '"xff":"$http_x_forwarded_for",'
 '"referer":"$http_referer",'
 '"agent":"$http_user_agent",'
 '"status":"$status"}';

  access_log  /var/log/nginx/access.log  access_json;

logstash配置文件所需路径需与nginx一致

 

创建相应的日志仓库

查看日志格式

[root@node01 ~]# tail /var/log/nginx/access.log

出现以下格式为正确
{"@timestamp":"2019-07-09T11:21:28+08:00","host":"192.168.1.30","clientip":"192.168.1.144","size":33,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.1.30","url":"/index.html","domain":"192.168.1.30","xff":"-","referer":"-","status":"200"}

编辑logstash的配置文件

input{

  file{

    #日志存放路径

    path => "/var/log/nginx/access.log"

    #从日志文件开头读取

    start_position => "beginning"

    #设置多长时间检测文件是否修改 默认是1s

    stat_interval => "2"

    codec => "json"

  }

}

filter {}

output {  # 定义日志输出

    elasticsearch {

        hosts => ["localhost:9200"]       # 定义es服务器的ip，这里使用本地

        index => "nginx-log"              #定义索引名称

    }

    stdout {

        codec => json_lines

    }

}

错误：Logstash could not be started because there is already another instance usin

多次启动logstash的话会出现缓存，导致无法启动，删除logstash下data的缓存文件即可

注意缓存文件为隐藏文件.lock 

 

rm -rf data/.lock

。Lock是隐藏文件需要ls -a查看后删除