filebeat收集日志传输到Redis集群,logstash从Redis集群中拉取数据

前提：已配置好Redis集群，并设置的有统一的访问密码

架构是filebeat-->redis集群-->logstash->elasticsearch,需要修改filebeat的输出和logstash的输入值
filebeat地址：192.168.80.108
redis集群地址：192.168.80.107 ，采用的是伪集群的方式

1 filebeat配置

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/openresty/nginx/logs/host.access.log
  fields:
    log_source: messages

- type: log
  enabled: true
  paths:
    - /usr/local/openresty/nginx/logs/error.log
  fields:
    log_source: secure

output.redis:
  # Redis集群地址列表
  hosts: ["192.168.80.107:7001","192.168.80.107:7002","192.168.80.107:7003","192.168.80.107:7004","192.168.80.107:7005","192.168.80.107:7006","192.168.80.107:7007","192.168.80.107:7008"]
  # Redis集群key
  key: messages_secure
  password: foobar2000
  # 集群模式下只能用第0数据库，填写其他的会报错
  db: 0

2 redis端查看数据

登录：

# -h是地址，-p是端口，-c表示集群，-a是密码
/elk/redis/redis-4.0.1/src/redis-cli -h 192.168.80.107 -c -p 7001 -a foobar2000

查看：

redis 127.0.0.1:7000[0]> keys *    # 出现这个key了  说明fielebeat的数据已经传输到redis集群中了
1) "messages_secure"     
redis 127.0.0.1:7000[0]> llen emessages_secure   ##查看list长度
(integer) 2002
redis 127.0.0.1:7000[0]> lindex messages_secure 0  #查看相关数据

或者使用redis客户端RedisDesktopManager使用

发现一个问题，Redis集群中出现俩messages_secure，且存储的数据一模一样，这个问题还有待继续研究..

3 logstash配置

input { 
  redis {
    host => "192.168.80.107"
    port => 7001
    password => foobar2000
    data_type => "list"
    key => "messages_secure"
    db => 0
  }
  redis {
    host => "192.168.80.107"
    port => 7002
    password => foobar2000
    data_type => "list"
    key => "messages_secure"
    db => 0
  }
  redis {
    host => "192.168.80.107"
    port => 7003
    password => foobar2000
    data_type => "list"
    key => "messages_secure"
    db => 0
  }
  redis {
    host => "192.168.80.107"
    port => 7004
    password => foobar2000
    data_type => "list"
    key => "messages_secure"
    db => 0
  }
  redis {
    host => "192.168.80.107"
    port => 7005
    password => foobar2000
    data_type => "list"
    key => "messages_secure"
    db => 0
  }
  redis {
    host => "192.168.80.107"
    port => 7006
    password => foobar2000
    data_type => "list"
    key => "messages_secure"
    db => 0
  }
  redis {
    host => "192.168.80.107"
    port => 7007
    password => foobar2000
    data_type => "list"
    key => "messages_secure"
    db => 0
  }
  redis {
    host => "192.168.80.107"
    port => 7008
    password => foobar2000
    data_type => "list"
    key => "messages_secure"
    db => 0
  }
  redis {
    batch_count => 1
    host => "192.168.80.107"
    port => 7001
    password => foobar2000
    data_type => "list"
    key => "messages_secure"
    db => 0
  }
}

# 输出到elasticsearch中，根据不同的日志来源创建不同的索引
output {

  if [fields][log_source] == 'messages' {
    elasticsearch {
      hosts => ["http://192.168.80.104:9200", "http://192.168.80.105:9200","http://192.168.80.106:9200"]
      index => "messages-%{+YYYY.MM.dd}"
      user => "elastic"
      password => "elkstack123456"
    }
  }

  if [fields][log_source] == "secure" {
    elasticsearch {
      hosts => ["http://192.168.80.104:9200", "