一、安装elasticsearch
可以查看前篇博客 elasticsearch安装、elasticsearch-head 安装
二、安装 配置 logstash
1.安装logstash
下载地址:https://www.elastic.co/downloads/logstash
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.5.4.tar.gz
tar -zxvf logstash-6.5.4.tar.gz
2.为了方便统计,此处使用了自定义模板 springlog.json
{ "springboot-logback": { "order": 1, "index_patterns": [ "sspringboot-logback-*" ] }, "settings": { "number_of_shards": 5, "number_of_replics": 1 }, "mappings": { "properties": { "@timestamp": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis" }, "thread": { "type": "text" }, "level": { "type": "text" }, "class": { "type": "text" }, "messge": { "type": "text" } } } }
3.配置logstash.conf 可以在config目录中创建此文件
input { beats { port => 10515 ssl => false } } filter { if [fields][logtype] == "springboot-logback" { grok { match => { "message" => "%{TIMESTAMP_ISO8601:time} %{GREEDYDATA:thread} %{LOGLEVEL:level} (?<class>\bcom.idelan\S*) - %{GREEDYDATA:message}" } } mutate { remove_field => ["host","tags","beat","@version","prospector","input"] } } } output { if [fields][logtype] == "springboot-logback"{ elasticsearch { hosts => ["192.168.30.242:9200"] index => "logstash-%{+YYYY.MM.dd}" template => "/home/tools/logstash-6.5.4/template/springlog.json" #此文件中的内容即上面自定义的模板json template_name => "springboot-logback" template_overwrite => true } stdout { codec => rubydebug } #将日志打印到控制台调试,放在if中顺便检测 是否获取到了[fields][logtype] 正式环境删除此行配置
} }
4.启动 logstash 进入logstash bin目录
./logstash -f ../config/logstash.conf
后台启动
./logstash -f ../config/logstash.conf &
5.grok 过滤规则配置
调试工具:http://grokdebug.herokuapp.com/
grok正则:https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
6.日志说明 springboot日志配置
日志格式
2019-01-04 16:08:36.487 [http-nio-8200-exec-10] INFO com.idelan.test.controller.TestController - hello info
grok配置
%{TIMESTAMP_ISO8601:time} %{GREEDYDATA:thread} %{LOGLEVEL:level} %{JAVACLASS:class} - %{GREEDYDATA:message}
由于我只想看属于我自己包中的日志,所以此处我过滤了com.idelan包下的日志
%{TIMESTAMP_ISO8601:time} %{GREEDYDATA:thread} %{LOGLEVEL:level} (?<class>\bcom.idelan\S*) - %{GREEDYDATA:message}
自定义grok表达式
语法:(?<field_name>the pattern here)
三、安装配置filebeat
1.下载filebeat
地址:https://www.elastic.co/downloads/beats/filebeat
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.5.4-linux-x86_64.tar.gz
tar -zxvf filebeat-6.5.4-linux-x86_64.tar.gz
2.配置filebeat.yml 如果没有可以手动创建
filebeat.prospectors: - input_type: log paths: - /usr/local/logs/springboot-demo/*.log include_lines: [".*INFO.*",".*ERROR.*"] exclude_lines: [".*DEBUG.*",".*WARN.*"] exclude_files: [".*debug.*",".*warn.*"] multiline: pattern: '^\s*(\d{4}|\d{2})\-(\d{2}|[a-zA-Z]{3})\-(\d{2}|\d{4})' # 指定匹配的表达式(匹配以 2017-11-15 08:04:23:889 时间格式开头的字符串) #pattern: '^\s*("{)' # 指定匹配的表达式(匹配以 "{ 开头的字符串) negate: true # 是否匹配到 match: after # 合并到上一行的末尾 max_lines: 1000 # 最大的行数 timeout: 30s # 如果在规定的时候没有新的日志事件就不等待后面的日志 fields: logtype: springboot-logback output.logstash: hosts: ["192.168.30.242:10515"]
3.启动filebeat
调试命令:./filebeat -e -c filebeat.yml -d "publish"
后台启动:./filebeat -e -c filebeat.yml &
四、成果展示