ELK架构搭建实践

先看一下本项目搭建的ELK(version 6.2.4)架构图

本文暂时只分析filebeat，logstash的配置。
filebeat的配置主要在filebeat.yml中

#读取日志的配置
#=========================== Filebeat inputs =============================
filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /Users/nikohuang/Documents/workspace/tst1/logs/*.log
  fields:
    log_source: tst1
  
  exclude_files: ['.gz$']

  multiline.pattern: '^\d{4}-\d{2}-\d{2}'

  multiline.negate: true

  multiline.match: after

- type: log
  enabled: true
  paths: 
    - /Users/nikohuang/Documents/workspace/tst2/logs/*.log
  fields:
    log_source: tst2
  
  exclude_files: ['.gz$']

  multiline.pattern: '^\d{4}-\d{2}-\d{2}'   #多行合并规则

  multiline.negate: true

  multiline.match: after
#=========================== Kafka output =============================
output.kafka:
  hosts: ["localhost:9092"]
  topic: elk-log
  required_acks: 1

在Linux环境中启动filebeat服务：nohup ./filebeat -c filebeat.yml --path.logs ./logs/ &>/var/null &

在logstash的主目录中新建文件first-pipeline.cof，添加如下内容

input {
   kafka {
    auto_offset_reset => "latest"
    group_id => "wxdx-elk"
    topics => ["wxdx-elk-log"]
    bootstrap_servers => "localhost:9092"
  }
}
filter {
  grok {
    patterns_dir => ["/usr/local/etc/logstash/patterns/java_pattern"]
    match => { "message" => "(?<time>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) %{IP:clientip} %{LOGLEVEL:level}"}
  }
  json {
        source => "message"
        target => "jsoncontent"
    }
}
output {
   elasticsearch {
     hosts => "127.0.0.1:9201"    #定义输出的es的url。
     index => "%{[fields][log_source]}-%{+YYYY.MM.dd}"   #定义标题索引
   }
   stdout { codec => rubydebug }
}

在/usr/local/etc/logstash/patterns/java_pattern文件中添加log内容解析

JAVACLASS (?:[a-zA-Z$_][a-zA-Z$_0-9]*\.)*[a-zA-Z$_][a-zA-Z$_0-9]*
JAVATHREAD (?:[A-Z]{2}-Processor[\d]+)
LOGLEVEL (?:DEBUG|FATAL|ERROR|WARN|INFO)

在linux环境下启动logstash
nohup ./logstash -f …/config/first-pipeline.cof -w 4 -l …/logs -b 1000 -u 1000 --http.port 9600 &>/var/null &