filebeat ES Kibana 日志可视化统计

NGINX 配置

NGINX 配置日志格式化

参考nginx内置变量
格式化定义在http块内

log_format ucp '{ "time_local": "$time_local", '
                    '"remote_addr": "$remote_addr", '
                    '"referer": "$http_referer", '
                    '"method": "$request_method", '
                    '"protocol": "$server_protocol", '
                    '"addr": "$server_addr", '
                    '"name": "$server_name", '
                    '"port": "$server_port", '
                    '"uri": "$uri", '
                    '"args": "$args", '
                    '"content_type": "$content_type", '
                    '"code": $status, '
                    '"bytes": $body_bytes_sent, '
                    '"user_agent": "$http_user_agent", '
                    '"x_forwarded": "$http_x_forwarded_for", '
                    '"up_addr": "$upstream_addr",'
                    '"up_host": "$upstream_http_host",'
                    '"upstream_time": "$upstream_response_time",'
                    '"request_time": "$request_time"}';

    access_log D:\\soft\\openresty27\\logs\\access.log ucp;

重启nginx生效

ELK配置

1. 先启动ElasticSearch
2. 启动Kibana
3. 启动filebeat

通过filebeat采集log日志—>ES—>Kibana

启动ES

如果是本地测试不用修改配置直接 到bin目录 ./elasticsearch

启动Kibana

测试时不用修改配置 ./kibana

Filebeat配置

全局配置

这里只配置ES地址和kibana地址
nginx配置到module中配置

filebeat.inputs:
# - type: log
#   id: openresty #唯一id
#   enabled: true
#   paths:
#     - D:\\soft\\openresty27\\logs\\access.log 
#   tags: ["test", "nginx"]
#   fields:
#     log_type: nginx_access_log
  


filebeat.config.modules:
  path: C:/Program Files/Elastic/Beats/7.11.2/filebeat/modules.d/*.yml

  reload.enabled: true

setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
  host: "localhost:5601"

output.elasticsearch:
  hosts: ["127.0.0.1:9200"]
  preset: balanced

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  # - decode_json_fields:
  #     fields: ["message"]           # 指定要解析的字段
  #     target: ""                    # 解析后的数据将被放置在根级别
  #     overwrite_keys: true          # 允许覆盖现有的键

启用nginx模块

./filebeat modules enable nginx

# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.11/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true
    var.paths: ["D:/soft/openresty27/logs/*access.log*"]
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
  ingress_controller:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

修改Nginx pipeline

nginx pipeline 是 日志数据转换的工具

修改 C:\Program Files\Elastic\Beats\7.11.2\filebeat\module\nginx\access\ingest\pipeline.yml

1. filebeat 默认的nginx pipeline 只能解析默认的日志格式我们已经自定义为json了
2. 所以需要自定义pipeline 参考 [pipeline] (https://www.elastic.co/guide/en/elasticsearch/reference/7.11/remove-processor.html)
3. pipeline 流水线由一道道工序组成

这里定义了三个处理器

description: Pipeline for parsing Nginx access logs. Requires the geoip and user_agent
  plugins.
processors:
- set:
    field: event.ingested
    value: '{{_ingest.timestamp}}'
- json:
    field: message
    target_field: json_msg
- remove:
    field: ["message","host"]

• set 设置 一个字段
• json 解析 message 字段为 json_msg （message字段默认就是nginx日志的全部，原始格式）
• remove 删除字段可以删除中间字段
• grok 使用 grok匹配模式提取字段
• spilt 分割
• 。。。。

使用grok提取json

#用grok提取这一段json
{"remote_addr":"127.0.0.1","remote_user":"","time_local":"03/Dec/2024:20:44:16 +0800","request":"GET /markmap/01.html HTTP/1.1","status":200,"body_bytes_sent":978,"http_referer":"http://localhost:8000/pages/dd027d/","request_time":0.000,"upstream_response_time":"","upstream_addr":"","server_name":"localhost"}

#grok配置
description: my-pipeline
processors:
  - set:
      field: event.ingested
      value: '{{_ingest.timestamp}}'
  - grok:
      field: message
      patterns:
        - '{"remote_addr":"%{IP:remote_addr}","remote_user":"%{DATA:remote_user}","time_local":"%{HTTPDATE:time_local}","request":"%{WORD:method} %{URIPATHPARAM:uri} HTTP/%{NUMBER:http_version}","status":%{NUMBER:code},"body_bytes_sent":%{NUMBER:body_bytes},"http_referer":"%{DATA:http_refer}","request_time":%{NUMBER:time},"upstream_response_time":"%{DATA:up_time}","upstream_addr":"%{DATA:up_addr}","server_name":"%{DATA:server_name}"}'

可以看到grok就是一组预定义的正则表达式 ：
文档 https://help.aliyun.com/zh/sls/user-guide/grok-patterns?spm=a2c4g.11186623.help-menu-28958.d_2_6_1_12_7_4.1acf5d4f23Z5Pd
在线调试：https://grokconstructor.appspot.com/do/match#result

因为设置nginx日志为json 就不需要写复杂的grok处理器但grok更灵活可以匹配更多字段，但会消耗cpu可能成为性能瓶颈
grok 在logstash 中也用到
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/grok-processor.html
https://www.bookstack.cn/read/logstash-best-practice-cn/filter-grok.md
可以在kibana 中调试 grok 表达式

运行filebeat

filebeat.exe run -e  # 这种会打印信息在黑窗口

成功状态

DELETE _ingest/pipeline/filebeat*

GET _ingest/pipeline # 看到nginx的pipeline

GET _cat/indices # 看到nginx的索引

DELETE filebeat-7.11.2*

配置kibana面板