logstash-redis-logstash-elasticsearch 实现日志分类

相比logstash使用java程序，会占用较大的内存，几百M，还要安装jdk环境 ，filebeat更轻量级，，内存十多M，使用go写的天生支持多线程，专门收集日志文件数据，常用于容器日志收集

web1   tomcat + logstash     =>     redis （主从，vip） => 负载均衡 logstash => 负载均衡es（haproxy）=>  es Cluster=> kibana

第一步：web1  把日志写入redis

[root@node1 conf.d]# cat tom.conf 
input {
 file {
 path => "/usr/local/tomcat/logs/tomcat_access_log.*.log"
 start_position => "end"
 type => "tomct-access-log"
 stat_interval => 3
 codec => "json"
 }

 file {
 path => "/usr/local/tomcat/logs/catalina.out"
 start_position => "end"
 type => "tomct-catalina-log"
 stat_interval => 3
 }
}

output {
if [type] == "tomct-access-log" {
 redis {
 data_type => "list"
 key => "tomcat-access-log-1512"
 host => "10.1.234.110"
 port => "6379"
 db => "0"
 password => "1ZdqlgWBfw"
 }}

 if [type] == "tomct-catalina-log" {
 redis {
 data_type => "list"
 key => "tomcat-catalina-log-1512"
 host => "10.1.234.110"
 port => "6379"
 db => "1"
 password => "1ZdqlgWBfw"
 }}
}

 

第二步：logstash读取reids 的日志写入到es 

[root@node3 conf.d]# cat redis-to-es.conf 
input {
redis {
 data_type => "list"
 key => "tomcat-access-log-1512"
 host => "10.1.234.110"
 port => "6379"
 db => "0"
 password => "1ZdqlgWBfw"
 threads => 2
}
redis {
 data_type => "list"
 key => "tomcat-catalina-log-1512"
 host => "10.1.234.110"
 port => "6379"
 db => "1"
 password => "1ZdqlgWBfw"
 }
}

output {
 if [type] == "tomct-access-log" {
 elasticsearch {
 hosts => ["10.1.234.120:9200"]
 index => "logstash-tomcat-1516-access-%{+YYYY.MM.dd}"
 }}

 if [type] == "tomct-catalina-log" {
 elasticsearch {
 hosts => ["10.1.234.120:9200"]
 index => "logstash-tomcat-1516-catalina-%{+YYYY.MM.dd}"
 }}
}

测试

for i in {1..100};do curl 10.1.234.110:8080;done

 

 

 存在问题，如果写入es的 logstash 如果挂了，redis的压力就会突增，redis当中堆积大量数据，redis服务器内存被大量使用，可能导致redis挂掉，

监控redis 键的长度

#!/usr/bin/env python
#coding:utf-8
#Author Zhang jie
import redis
def redis_conn():
 
pool=redis.ConnectionPool(host="192.168.15.12",port=6379,db=0,password=12345
6)
 conn = redis.Redis(connection_pool=pool)
 data = conn.llen('tomcat-accesslog-1512')
 print(data)
redis_conn()