ELK使用redis做中间键，减少整个elk的压力

1、安装redis，安装部署过程这里不介绍（这里redis的ip为192.168.0.197，端口为6379）

2、配置logstash的收集数并导入redis的配置文件和从redis中获取数据导入elasticsearch的两个配置文件

1、配置导入redis数据的配置文件，并启动logstash服务

input {
      file{
        path => "/var/log/messages"             #指定要收集的日志文件
  type => "system"  #指定类型为system，可以自定义，type值和output{ } 中的type对应即可
        start_position => "beginning"       #从开始处收集
        }
      file{
        path => "/home/otc/otc-web/logs/gxzx-otc-web.log"
        type => "otc"
        start_position => "beginning"
         }
       file{ 
        path => "/home/deploy/financial-management/logs/gxzx-fin-web.log"
        type => "financial"
        start_position => "beginning"
          }
       file{
        path => "/home/deploy/activity_service/logs/gxzx-act-web.log"
        type => "act"
        start_position => "beginning"
       }     
       file{
        path => "/home/deploy/mining/logs/gxzx-min-web.log"
        type => "mining"
        start_position => "beginning"
}     
       }
output {
        if [type] == "system" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_system'
             }
        }
        if [type] == "otc" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_otc'
             }
        }
        if [type] == "financial" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_financial'
             }
        }
        if [type] == "act" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_act'
             }
        }
        if [type] == "mining" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_mining'
             }
        }
}

启动服务：

在源码安装的logstash的bin目录下执行，后面还要加一路径，默认是当前执行这的家目录下
 ./logstash -f input_redis.conf &

2、配置从redis导出数据的配置文件

input {
        beats {
            port => 5045
        }
        if [type] == "system" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_system'
        }
        }
        if [type] == "otc" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_otc'
             }
        }
        if [type] == "financial" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_financial'
             }
        }
        if [type] == "act" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_act'
             }
        }
        if [type] == "mining" {
             redis {
                 host => "192.168.0.197"
                 password => '901Bcpct'
                 port => "6379"
                 db => "3"
                 data_type => "list"
                 key => 'logs_mining'
             }
        }
}


output {
        if [type] == "system" {           #如果type为system，
           elasticsearch {                  #就输出到Elasticsearch服务器
             hosts => ["192.168.0.117:9200"]               #Elasticsearch监听地址及端口
             index => "system-%{+YYYY.MM.dd}"           #指定索引格式
                }
             }
        if [type] == "otc" {
  elasticsearch {
             hosts => ["192.168.0.117:9200"]
             index => "nginx_otc-%{+YYYY.MM.dd}"
               }
             } 
        if [type] == "financial" {
          elasticsearch {
             hosts => ["192.168.0.117:9200"]
             index => "nginx_financial-%{+YYYY.MM.dd}"
             }
          }  
        if [type] == "act" {
          elasticsearch {
             hosts => ["192.168.0.117:9200"]
             index => "act_log-%{+YYYY.MM.dd}"
             }
          }  
        if [type] == "mining" {
          elasticsearch {
             hosts => ["192.168.0.117:9200"]
             index => "mining_log-%{+YYYY.MM.dd}"
             }
          }  
        } 

同样启动服务

./logstash -f output_redis.conf --path.data=/home/elk/ &
此时我将路径改成了另一个路径了

此时我们就可以在redis上看到我们刚刚加的key和值

 此时我们的redis就加入到我们的elk当中