docker+elk7.8实战之filebeat收集nginx日志

最新推荐文章于 2024-05-31 10:32:17 发布

泽济天下

最新推荐文章于 2024-05-31 10:32:17 发布

阅读量1.6k

点赞数 2

分类专栏： elk7+docker 文章标签： filebeat nginx nginx日志 elk

本文链接：https://blog.csdn.net/u010361276/article/details/108089027

版权

elk7+docker 专栏收录该内容

8 篇文章 3 订阅

订阅专栏

0. 前置条件

需要自行安装nginx环境,通过yum -y install nginx安装即可。

1.安装filebeat

如果没有安装kibana可以参考之前的文章https://blog.csdn.net/u010361276/article/details/107695787
如果安装有kibana可以在如下地址获取到安装filebeat及其他beat的方式和下载地址。（ip换成实际使用的ip）

http://192.168.0.203:5601/app/kibana#/home/tutorial/nginxLogs

具体安装方式，以centos7为例

# 通过rpm包安装
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.8.1-x86_64.rpm
sudo rpm -vi filebeat-7.8.1-x86_64.rpm

注意：

安装前可通过 rpm -qa|grep filebeat查看是否已经安装过filebeat(如果安装过会打印一系列已安装的文件名供查看和卸载使用)

如果需要卸载使用rpm -e xxx命令(xxx代表上一步输出的文件名)

2.配置filebeat

通过rpm文件安装的filebeat对应的配置文件目录在/etc/filebeat/,对应的日志目录在/var/log/filebeat/

主要的配置文件是filebeat.yml 和 modules.d文件夹中的一系列yml文件

在配置文件目录中存在一个filebeat.reference.yml文件，其中列举了几乎filebeat相关的所有配置及说明。

1) 修改filebeat.yml文件

# 这里是配置加载modules.d里面的哪些文件 以及是否定时加载配置文件
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 30s

# 这里配置这个filebeat的相关信息
name: filebeat-202
tags: ["filebeat", "203"]

# filebeat提供了多种输出可供选择,目前我们这里选择的是logstash 后续会做相关改造
# 这里是具体的logstash的IP地址和端口,请对应修改 
output.logstash:
  hosts: ["192.168.0.203:5044"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
# 下面的是配置日志相关的信息 包括日志级别及日志输出位置
logging.level: error
logging.to_stderr: false
logging.to_syslog: false
logging.to_eventlog: false
logging.to_files: true
logging.path: /var/log/filebeat
logging.name: filebeat

2) 修改模块配置

在/etc/filebeat/modules.d文件夹中默认存在了很多以.yml.disabled结尾的文件，我们可以参考这些配置yml文件

也可以直接重命名为yml文件

cd  /etc/filebeat/modules.d
cp nginx.yml.disabled nginx.yml

说明: 开启模块的方式有多种，不局限于提到的这种。

3) logstash配置修改

在步骤2中我们设置了输出是logstash的5044端口，自然logstash的配置也要做相关修改。

在之前搭建的elk环境的logstash目录/opt/elk7/logstash/pipeline中我们新建一个conf文件，命名为logstash-beat-es.conf

# 输入配置 注意是beats不是beat 这里的beats是插件名
input {
  beats {
    port => 5044
  }
}
 
# 过滤器 将日志解析成对应的具体字段值  使用了grok插件和内置的COMBINEDAPACHELOG格式匹配器
filter { 
  # 这里是ruby语法
  if "nginx" == [service][type] {
    grok { 
      # 注意这里的=> 和 ""
      match => {"message" => "%{IP:req_ip} - (%{USERNAME:req_user}|-) \[%{HTTPDATE:req_timestamp}\] \"%{WORD:request_verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion
}\" %{NUMBER:resp_status:int} %{NUMBER:body_sent:int} \"-\" \"%{GREEDYDATA:httpagent}\" \"-\""}
    }
  }
}

# 这里是输出设置 
output {
  # 如果需要调试或者验证日志数据是否到了logstash
  # 如下几行代码便是将收到的日志信息打印到控制台
  stdout {
    codec => rubydebug
  } 
  
  # 设置输出到es的配置 和 之前的一样只是重新定义了索引名称
  elasticsearch {
    index => "beat-%{+YYYY.MM.dd}"
    hosts => "http://192.168.0.203:9200"
     # 注意这里是具体的es配置的用户名和密码 注意这里的""
    user => "elastic"
    password => "xxxxx"
  }
}

添加了配置文件我们需要在pipeline.yml文件中创建对应的信息

添加如下代码即可

- pipeline.id: beat
  path.config: "/usr/share/logstash/pipeline/logstash-beat.conf"

3.启动并验证

logstash的启动由于之前创建logstash容器的时候挂载的是pipeline文件夹 ,所以直接重启容器即可。

docker restart logstash

启动filebeat

# 设置filebeat随机启动
systemctl enable filebeat
# filebeat的启动
systemctl start filebeat 
# 查看filebeat状态
systemctl status filebeat -l

如果看到如下输出表示正常

[root@localhost pipeline]# systemctl status filebeat -l
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
   Active: active (running) since 日 2020-08-16 17:15:55 CST; 39min ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 4206 (filebeat)
    Tasks: 21
   Memory: 80.8M
   CGroup: /system.slice/filebeat.service
           └─4206 /usr/share/filebeat/bin/filebeat -environment systemd -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

8月 16 17:15:55 localhost.localdomain systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
8月 16 17:15:55 localhost.localdomain systemd[1]: Starting Filebeat sends log files to Logstash or directly to Elasticsearch....

注意：

Active的值必须是active(running)，如果是failed或者exits的话需要通过日志排查原因。

如果开启了stdout标准控制台输出则可以通过logstash容器的日志查看具体内容

docker logs -f logstash --tail 200

访问nginx 如 curl http://192.168.0.203

观察logstash的日志输出，可以看到如下内容

{
        "service" => {
        "type" => "nginx"
    },
          "ident" => "-",
      "timestamp" => "16/Aug/2020:18:10:41 +0800",
           "tags" => [
        [0] "filebeat",
        [1] "203",
        [2] "beats_input_codec_plain_applied"
    ],
        "message" => "192.168.0.203 - - [16/Aug/2020:18:10:41 +0800] \"GET / HTTP/1.1\" 200 3700 \"-\" \"curl/7.29.0\" \"-\"",
            "log" => {
          "file" => {
            "path" => "/var/log/nginx/access.log"
        },
        "offset" => 0
    },
       "referrer" => "\"-\"",
     "@timestamp" => 2020-08-16T10:10:44.765Z,
            "ecs" => {
        "version" => "1.5.0"
    },
          "agent" => {
                "name" => "filebeat-202",
             "version" => "7.8.1",
        "ephemeral_id" => "c33421ab-e2d9-4a6b-8bd5-94d3560c7c9b",
            "hostname" => "localhost.localdomain",
                "type" => "filebeat",
                  "id" => "219a0e91-c6f5-47d9-8539-00e89e163d9c"
    },
           "host" => {
                   "os" => {
                "name" => "CentOS Linux",
             "version" => "7 (Core)",
              "kernel" => "3.10.0-862.9.1.el7.x86_64",
              "family" => "redhat",
            "platform" => "centos",
            "codename" => "Core"
        },
         "architecture" => "x86_64",
                 "name" => "filebeat-202",
                   "ip" => [
            [0] "192.168.0.203",
            [1] "fe80::1c6b:d091:a6b1:d3e2",
            [2] "172.17.0.1",
            [3] "fe80::42:b4ff:fe99:5a47",
            [4] "172.18.0.1",
            [5] "fe80::fc89:7aff:fe78:5712",
            [6] "fe80::20e6:5dff:fedb:7db1",
            [7] "fe80::98d8:3fff:febb:1508",
            [8] "fe80::6894:1cff:fed4:8f90"
        ],
             "hostname" => "localhost.localdomain",
                  "mac" => [
            [0] "00:0c:29:81:4c:19",
            [1] "02:42:b4:99:5a:47",
            [2] "02:42:8a:0b:e2:63",
            [3] "fe:89:7a:78:57:12",
            [4] "22:e6:5d:db:7d:b1",
            [5] "9a:d8:3f:bb:15:08",
            [6] "6a:94:1c:d4:8f:90"
        ],
        "containerized" => false,
                   "id" => "35f66aae9fd84ef3af624c81426f1427"
    },
        "request" => "/",
       "@version" => "1",
          "event" => {
        "timezone" => "+08:00",
          "module" => "nginx",
         "dataset" => "nginx.access"
    },
       "response" => "200",
           "verb" => "GET",
          "bytes" => "3700",
    "httpversion" => "1.1",
        "fileset" => {
        "name" => "access"
    },
          "input" => {
        "type" => "log"
    },
       "clientip" => "192.168.0.203",
           "auth" => "-"
}

同样的我们通过浏览器访问nginx首页然后再看下日志输出,可以看到如下内容:

/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
      "httpversion" => "1.1",
          "request" => "/",
          "fileset" => {
        "name" => "access"
    },
        "httpagent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36 Edg/84.0.522.59",
             "tags" => [
        [0] "filebeat",
        [1] "203",
        [2] "beats_input_codec_plain_applied"
    ],
    "req_timestamp" => "18/Aug/2020:21:56:32 +0800",
        "body_sent" => 0,
         "@version" => "1",
              "log" => {
          "file" => {
            "path" => "/var/log/nginx/access.log"
        },
        "offset" => 885
    },
     "request_verb" => "GET",
          "service" => {
        "type" => "nginx"
    },
         "req_user" => "-",
           "req_ip" => "192.168.0.108",
              "ecs" => {
        "version" => "1.5.0"
    },
            "input" => {
        "type" => "log"
    },
      "resp_status" => 304,
       "@timestamp" => 2020-08-18T13:56:33.268Z
}
{
      "httpversion" => "1.1",
          "request" => "/",
          "fileset" => {
        "name" => "access"
    },
        "httpagent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36 Edg/84.0.522.59",
             "tags" => [
        [0] "filebeat",
        [1] "203",
        [2] "beats_input_codec_plain_applied"
    ],
    "req_timestamp" => "18/Aug/2020:21:56:32 +0800",
        "body_sent" => 0,
         "@version" => "1",
              "log" => {
          "file" => {
            "path" => "/var/log/nginx/access.log"
        },
        "offset" => 673
    },
     "request_verb" => "GET",
          "service" => {
        "type" => "nginx"
    },
         "req_user" => "-",
           "req_ip" => "192.168.0.108",
              "ecs" => {
        "version" => "1.5.0"
    },
            "input" => {
        "type" => "log"
    },
      "resp_status" => 304,
       "@timestamp" => 2020-08-18T13:56:32.268Z
}
{
      "httpversion" => "1.1",
          "request" => "/",
          "fileset" => {
        "name" => "access"
    },
        "httpagent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36 Edg/84.0.522.59",
             "tags" => [
        [0] "filebeat",
        [1] "203",
        [2] "beats_input_codec_plain_applied"
    ],
    "req_timestamp" => "18/Aug/2020:21:56:31 +0800",
        "body_sent" => 0,
         "@version" => "1",
              "log" => {
          "file" => {
            "path" => "/var/log/nginx/access.log"
        },
        "offset" => 461
    },
     "request_verb" => "GET",
          "service" => {
        "type" => "nginx"
    },
         "req_user" => "-",
           "req_ip" => "192.168.0.108",
              "ecs" => {
        "version" => "1.5.0"
    },
            "input" => {
        "type" => "log"
    },
      "resp_status" => 304,
       "@timestamp" => 2020-08-18T13:56:32.268Z
}