ELK+Redis+Filebeat构建日志系统

声明： ELK版本为5.5.3

• 使用beat的原因为logstash需要依赖java环境，对cpu和内存消耗都比较大，所以filebeat应运而生，对cpu和内存基本没有消耗。
• 使用redis 的原因是当多个clinet同时写入到logstash或者elasticsearch 时候，有io瓶颈，所以选择了redis ,当然可以使用kafka，rabbitmq等消息中间件。（使用redis是防止大批量日志的时候logstash无法及时处理）

• 软件官网下载即可
• 启动ElasticSearch
• 启动redis
• 启动filebeat

  filebeat -e -c filebeat2.yml
  

• 启动logstash

  logstash -f logstash-sample.conf
  

filebeat2.yml

#=========================== Filebeat prospectors =============================
filebeat.prospectors:
- type: log
  #开启监视，不开不采集
  enabled: true
  paths:
    - D:\logs\projectManagement\*.log
  encoding: utf-8
  # logstash判断日志来源
  fields:
    log_source: project_management 
  multiline:  
    pattern: '^\|'  
    negate: true 
    match: after
    
- type: log
  #开启监视，不开不采集
  enabled: true
  paths:
    - D:\logs\logManagement\*.log
  encoding: utf-8
  # logstash判断日志来源
  fields:
    log_source: log_management   
  multiline:  
    pattern: '^\|'  
    negate: true 
    match: after
#-------------------------- reids output ------------------------------
output.redis:
  hosts: ["127.0.0.1:6379"]
  key: logs
  db: 4
  # 超时
  timeout: 5s
  # 最大重试次数
  max_retries: 3

logstash-sample.conf

input {
  redis {
    host => "127.0.0.1"
    port => "6379"
    data_type => "list"
    key => "logs"
	db => 4
    type => "redis-input"
  }
}
filter {
    grok {
		# 正则文件存放目录
		patterns_dir => "./patterns"
		match => {
		   "message" => "%{JAVA_LOG}"
		}	
		remove_field => ["message"]
    }
	if ([level]!= "INFO" and [level]!= "DEBUG" and [level]!= "ERROR" and [level]!= "WARN"){
		drop {}
	}	
	date {
		match => [ "logTime", "yyyy-MM-dd HH:mm:ss" ]
		target => "logTime"
		timezone =>"+00:00"
	}
}
output {
	if [fields][log_source] == 'project_management' {
	  elasticsearch {
		action => "index"  
		hosts  => "127.0.0.1:9201"  
		index  => "log_project"  
		template => "../config/project.json"
		template_name => "log_project"
		template_overwrite => true		
	  }
    }
	if [fields][log_source] == 'log_management' {
	  elasticsearch {
		action => "index"  
		hosts  => "127.0.0.1:9201"  
		index  => "log_log"    
		template => "../config/log.json"
		template_name => "log_log"
		template_overwrite => true
	  }
    }
}

Logstash 自定义模板

log.json

{
    "template":"log_log",
    "settings":{
        "index.refresh_interval":"1s"
    },
    "mappings":{
        "log":{
            "properties":{
                "level":{
                    "type":"keyword"
                },
                "line":{
                    "type":"keyword"
                },
                "logMessage":{
                    "type":"text"
                },
				"thread":{
                    "type":"text"
                },
				"logClass":{
                    "type":"text"
                },
				"logTime":{
                    "type":"date"
                }
			
            }
        }
    }
}

logstash 配置过滤器将日志信息截断

1. 在bin目录下创建patterns文件夹（文件名称自定义）
2. 创建配置文件 patterns.conf 内容如下：

MYSELFTIMESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND})
JAVACLASS (?:[a-zA-Z$_][a-zA-Z$_0-9]*\.)*[a-zA-Z$_][a-zA-Z$_0-9]*
MATCH_ANY [\s\S]*
JAVA_LOG \|\- %{MYSELFTIMESTAMP:timestamp}(\s*)%{LOGLEVEL:level}(\s*)\[%{MATCH_ANY:thread}\](\s*)%{JAVACLASS:class}([\s\S]*)\: %{NUMBER:line} %{MATCH_ANY:logmessage}