【日志审计】Ngnix日志filter记录

Nginx日志配置

1、修改Nginx配置/usr/local/nginx/conf/nginx.conf文件，确定要记录的日志。

http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" $http_host $uri ';

    access_log  logs/access.log  main;

参考文章：

1、nginx日志输出配置json格式
	https://blog.csdn.net/zx77588023/article/details/123079087
2、Logstash配置Nginx日志分析
	https://www.zhihu.com/tardis/bd/art/406543347?source_id=1001
3、grok语法定义
	https://www.cnblogs.com/zhangan/p/11395056.html
4、使用logstash-input-kafka消费信息并根据topic写入不同的es索引
	https://code84.com/772385.html
5、Logstash解析嵌套JSON格式数据&常见时间操作
	https://blog.csdn.net/XMZHSY/article/details/121550687

Sample Data

10.172.192.20 - - [13/Jun/2023:13:34:54 +0800] "GET /1 HTTP/1.1" 404 555 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-" 172.32.10.17 /1 

Grok Pattern

%{IP:remote_addr} - %{DATA:remote_user} \[%{HTTPDATE:request_time}\] \"%{WORD:request_method} %{URIPATHPARAM:url_args} %{URIPROTO:protocol}/%{DATA:treaty}\" %{NUMBER:status} %{NUMBER:body_sent_bytes} \"%{DATA:http_referer}\" \"%{DATA:http_user_agent}\" \"%{DATA:X_Forwarded_For}\" %{DATA:host} %{URIPATH:uri}

Structured Data

{
  "remote_addr": "10.172.192.20",
  "request_method": "GET",
  "uri": "/1",
  "url_args": "/1",
  "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36",
  "remote_user": "-",
  "protocol": "HTTP",
  "request_time": "13/Jun/2023:13:34:54 +0800",
  "http_referer": "-",
  "treaty": "1.1",
  "body_sent_bytes": "555",
  "host": "172.32.10.17",
  "X_Forwarded_For": "-",
  "status": "404"
}

---

Sample Data

2023/06/14 15:05:50 [error] 10446#0: *117 open() "/usr/local/nginx/html/123" failed (2: No such file or directory), client: 10.172.192.20, server: localhost, request: "GET /123 HTTP/1.1", host: "172.32.10.17"

Grok Pattern

%{DATA:error_time} \[%{DATA:error_type}\] %{DATA:error_info}, client: %{DATA:client}, server: %{DATA:server}, request: \"%{DATA:request}\", host: \"%{DATA:host}\"

Structured Data

{
  "server": "localhost",
  "request": "\"GET /123 HTTP/1.1\"",
  "error_time": "2023/06/14 15:05:50",
  "error_info": "10446#0: *117 open() \"/usr/local/nginx/html/123\" failed (2: No such file or directory)",
  "error_type": "error",
  "host": "172.32.10.17",
  "client": "10.172.192.20"
}