FileBeat 搭建说明

官网参考

https://www.elastic.co/guide/en/logstash/current/index.html

官网下载

https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.13.1-linux-x86_64.tar.gz

解压

lxp@lxp:~$ tar -zvxf  filebeat-7.13.1-linux-x86_64.tar.gz

lxp@lxp:~/filebeat-7.13.1-linux-x86_64$ ls
fields.yml  filebeat.reference.yml  kibana       module     NOTICE.txt
filebeat    filebeat.yml            LICENSE.txt  modules.d  README.md
lxp@lxp:~/filebeat-7.13.1-linux-x86_64$

修改配置文件filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/lxp/logs/*.log
  multiline.pattern: '^\[20[0-9]{2}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3}Z\]'
  multiline.negate: true
  multiline.match: after

processors:
- convert:
    fields:
        - {from: "host.name", to: "host", type: "string"}
        - {from: "log.file.path", to: "file", type: "string"}
    ignore_missing: true
    fail_on_error: false
- drop_fields:
    fields: ["log", "ecs","agent","input","metadata"] 
- add_fields:
    target: ''
    fields:
        type: 'service-service_LNTAdapter-log'
# ---------------------------- redis Output ----------------------------
#output.redis:
  #hosts: ["10.0.56.100:6379"]
  #key: "logstash_service"
  #datatype: list
  #db: 0
  #timeout: 5

# ---------------------------- console Output ----------------------------
output.console:
  pretty: true

参考

https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html

重点了解

• Inputs

支持如下输入

• AWS CloudWatch
• AWS S3
• Azure Event Hub
• Cloud Foundry
• Container
• Docker
• GCP Pub/Sub
• HTTP Endpoint
• HTTP JSON
• Kafka
• Log
• MQTT
• NetFlow
• Office 365 Management Activity API
• Redis
• Stdin
• Syslog
• TCP
• UDP

• Output

支持如下输出

• Elasticsearch Service
• Elasticsearch
• Logstash
• Kafka
• Redis
• File
• Console

• Processors

参考：

https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html

样例：

input {
 redis{ 
  host => "elk1-red-cn1a-prd" 
  port => 46379 
  db => 0 
  codec => "json" 
  batch_count => 1000 
  data_type => "list" 
  key => "logstash:elk" 
  threads => 8 
 }
}

filter {
 if [type] == "service-service_LNTAdapter-log" {
  grok {
   match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{WORD:service}\]\[%{LOGLEVEL:level}\]\[%{NOTSPACE:thread}\]\[%{WORD:logId}:%{NOTSPACE:traceId}\]" }
  } 
  date{ 
   match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] 
  }
   }
    if [type] in [ "bss-log", "service-log", "wildfly-log"] {
 grok { 
  match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{NUMBER:relativetime}%{SPACE}[%{GREEDYDATA:threadname}]%{SPACE}%{LOGLEVEL:loglevel }%{SPACE}%{DATA:classname}% {SPACE}-%{SPACE}%{GREEDYDATA:log}" }
 }
 date{ 
  match => [ "timestamp", "ISO8601", "YYYY-MM-dd HH:mm:ss", "YYYY-MM-dd HH:mm:ss.SSS", "dd/MMM/YYYY:HH:mm:ss", "dd/MMM/YYYY:HH:mm:ss +0000" ] }
 }

 if [type] == "access-log" {
  grok {
   match => { "message" => "%{COMMONAPACHELOG}%{NUMBER:responsetime}%{GREEDYDATA:headers}" }
  }
  date{ 
   match => [ "timestamp", "ISO8601", "dd/MMM/YYYY:HH:mm:ss", "dd/MMM/YYYY:HH:mm:ss +0000" ] 
  }
 }

 if [type] == "s3-access-log" {
  grok {
   patterns_dir => "/usr/share/logstash/patterns/"
   match => { "message" => "%{S3ACCESSLOG}" }
  }
  date{ 
   match => [ "timestamp", "ISO8601", "dd/MMM/YYYY:HH:mm:ss", "dd/MMM/YYYY:HH:mm:ss +0000" ] 
  }
 }
 if [type] == "wildfly-access-log" {
  grok {
   patterns_dir => "/usr/share/logstash/patterns/"
   match => { "message" => "%{WILDFLYACCESSLOG}" }
  }
  date{
   match => [ "timestamp", "ISO8601", "dd/MMM/YYYY:HH:mm:ss", "dd/MMM/YYYY:HH:mm:ss +0000" ] 
  }
 }

 if [type] == "smps-access-log" {
  grok {
   patterns_dir => "/usr/share/logstash/patterns/"
   match => { "message" => "%{SMPSACCESSLOG}" }
  }
  date{ 
   match => [ "timestamp", "ISO8601", "dd/MMM/YYYY:HH:mm:ss", "dd/MMM/YYYY:HH:mm:ss +0000" ] 
  }
 }

 if [type] == "service-pmt-cn-log" {
  grok {
   patterns_dir => "/usr/share/logstash/pat

启动

./filebeat -e -c filebeat.yml