Logstash+FileBeat+MongoDB+Flask打造的日志系统（一）-CSDN博客

2019独角兽企业重金招聘Python工程师标准>>>

1、MongoDB安装，运行（略）

2、JDK8 安装（略）

-- LogStash依赖Java.

3、LogStash 安装（略）

1）安装logstash-output-mongodb插件

bin/logstash-plugin install logstash-output-mongodb

（如果不行，好像下载logstash-output-mongodb插件解压到vendor/bundle/jruby/1.9/gems/logstash-output-mongodb-master/ 貌似就有反应了。。。）

2）配置文件：

input {
    beats {
        port => 4560
    }
}

filter {
#  if [type] == "register" or [type] == "configserver" {
    grok {
        match=>{"message" => "%{DATESTAMP_CHS2:logtime}\s+%{LOGLEVEL:loglevel} %{GREEDYDATA:msg}"}
    }
    date {
       match => ["logtime", "yyyy-MM-dd HH:mm:ss.SSS" ]
       target => "logtime"
    }
#  }
#  else {
#    grok {
#         match=>{"message" => "%{DATESTAMP_CHS:logtime} \[%{LOGLEVEL:loglevel}\] %{GREEDYDATA:msg}"}
#    }
#    date {
#        match => ["logtime", "yyyy-MM-dd HH:mm:ss" ]
#        target => "logtime"
#    }
#  }
}


output {
#    stdout {
#        codec => rubydebug
#    }
    if "_grokparsefailure" not in [tags] {
       mongodb {
          collection => "log"
          database => "longzhu"
          uri => "mongodb://127.0.0.1:27017"
       }
    }
}

其中 DATESTAMP_CHS2， DATESTAMP_CHS是我添加的，在这个文件里：vendor/bundle/jruby/1.9/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns。

DATESTAMP_CHS %{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}

DATESTAMP_CHS2 %{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}\.\d{3}

5、FileBeat 安装

配置如下，增加一个document_type:tomcat，LogStash就会出现 "type"=>"tomcat"

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /nfsdata/log/xxx.log
  document_type: tomcat
- input_type: log
  paths:
    - /nfsdata/log/xxx.log
  document_type: nginx


#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:4560"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

环境就配置好了！

转载于:https://my.oschina.net/huanghongqiao/blog/1545289