ELK之LogStash接收Filebeat的数据

ELK之LogStash接收Filebeat的数据

1、修改filebeat的配置文件filebeat.yml

修改filebeat的配置文件filebeat.yml， 将output输出到logstash，由于filebeat只能output到一个位置，故需要注释掉output.console相关配置。

filebeat.inputs:
- type: filestream
  id: nginx-access-log
  enabled: true
  paths:
  # 表示去/tmp下读取所有.log文件
    - /tmp/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  
setup.template.settings:
  index.number_of_shards: 1
  
# 注释掉output.console相关配置
# output to console
#output.console:
#  codec.format:
#    string: '%{[@timestamp]} %{[log]} %{[agent]}  %{[service]}  %{[message]}'
#  pretty: true

# 打开output.logstash配置并添加logstash地址
output.logstash:
  # The Logstash hosts：你Logstash的地址:端口，多个用英文逗号隔开
  hosts: ["172.x.x.x:5044"]
  
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# 暂时注释掉原来默认打开的elasticsearch的配置
#output.elasticsearch:
#  hosts: ["localhost:9200"]

2、Logstash的配置文件修改：

input{
  #stdin{type => stdin}
  #file {
  #  path => ["/tmp/access.log"]
  #  start_position => "beginning"
  #}
  # 输入方式改为beats方式，监听5044端口
  beats{
    port => 5044
  }
}

filter{
  grok{
    match => {"message" => "%{COMBINEDAPACHELOG}" }
  }
  mutate{
    # 重命名字段
    rename => {"clientip" => "cip"}
  }
  mutate{
    # 移出特定字段
    remove_field => ["timestamp","agent"]
  }
  geoip{
    # 由于上面将clientip修改为了cip，故此处配置cip，如果没有rename字段则用clientip
    source => "cip"
  }
}

output{
  stdout{codec => rubydebug}
}

3、启动 Filebeat 和 Logstash

启动filebeat ./filebeat -c filebeat.yml

启动logstash ./logstash -f ../config/first-pipeline.conf

如果您启动报错（Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.）了， 则可以去这里ELK之Logstash启动异常：Logstash could not be started because there is already…查看解决办法。

往/tmp/access.log 日志文件中追加日志：

175.30.108.245 - - [25/Jun/2022:05:11:33 +0800] "GET /api/ss/api/v1/login/getBaseUrl HTTP/1.1" 200 103 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b"

logstash 控制台打印如下，发现已经读取了新追加的日志。
根据filebeat和logstash的配置文件可以发现，文件是由filebeat从/tmp/access.log读取的，并输出到logstash(ip:5044)中，而logstash监听5044端口的数据，并输出到控制台。

{
           "tags" => [
		        [0] "beats_input_codec_plain_applied"
		    ],
          "input" => {
		        "type" => "filestream"
		   },
           "auth" => "-",
        "request" => "/api/ss/api/v1/login/getBaseUrl",
          "ident" => "-",
            "ecs" => {
        "version" => "1.12.0"
    },
     "@timestamp" => 2022-06-27T02:47:09.394Z,
        "service" => {
          "id" => "62145682378000001",
        "name" => "service-01"
    },
    "httpversion" => "1.1",
          "bytes" => "103",
       "referrer" => "\"-\"",
           "host" => {
        "containerized" => false,
                   "os" => {
                "type" => "linux",
              "kernel" => "3.10.0-1127.19.1.el7.x86_64",
             "version" => "7 (Core)",
                "name" => "CentOS Linux",
            "platform" => "centos",
              "family" => "redhat",
            "codename" => "Core"
        },
                   "id" => "20200914151306980406746494236010",
                 "name" => "nb002",
                   "ip" => [
            [ 0] "fe80::fc36:e9ff:fe28:71f5",
            [ 1] "172.x.x.x"
        ],
                  "mac" => [
            [ 0] "fe:36:e9:28:71:f5",
            [ 1] "00:16:3e:30:c8:94"
        ],
             "hostname" => "nb002",
         "architecture" => "x86_64"
    },
            "cip" => "175.30.108.245",
          "geoip" => {
           "region_code" => "JL",
         "country_code3" => "CN",
         "country_code2" => "CN",
             "city_name" => "Changchun",
              "timezone" => "Asia/Shanghai",
              "latitude" => 43.88,
              "location" => {
            "lat" => 43.88,
            "lon" => 125.3247
        },
             "longitude" => 125.3247,
           "region_name" => "Jilin",
          "country_name" => "China",
                    "ip" => "175.30.108.245",
        "continent_code" => "AS"
    },
           "verb" => "GET",
        "message" => "175.30.108.245 - - [25/Jun/2022:05:11:33 +0800] \"GET /api/ss/api/v1/login/getBaseUrl HTTP/1.1\" 200 103 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
            "log" => {
          "file" => {
            "path" => "/tmp/access.log"
        },
        "offset" => 2607
    },
       "response" => "200",
       "@version" => "1"
}

END