elk系列之将日志存入到redis，在存入到es

说明：此篇在elk系列之nginx、apache的日志到kibana展示（https://blog.csdn.net/weixin_44320761/article/details/121336490）的基础上操作。通过logstash将日志存入到redis，在存入到es,在kibana展示。

1、redis安装

（1）下载安装包
wget http://download.redis.io/releases/redis-5.0.9.tar.gz
（2）解压
tar xvf redis-5.0.9.tar.gz
（3）编译安装
cd redis-5.0.9
make
make install
（4）修改配置
#bind 127.0.0.1
daemonize yes
requirepass foobared

2、启动redis

redis-server redis.conf

3、测试连接

redis-cli -a foobared
如下表示正常

4、配置logstash

redis-logstash.conf
input {
            stdin {}
}

output {

        redis {
                host => "10.1.0.103"
                port => "6379"
                password => 'foobared'
                db => '0'
                data_type => "list"
                key => 'test-elk'
        }
}

5、重启logstash

logstash -f redis-logstash.conf

6、修改配置文件

vim logstash.conf
input {
    file {
        path => "/var/log/httpd/access_log"
        type => "http"
        start_position => "beginning"
    }   

    file {
        path => "/usr/local/nginx/logs/elk.access.log"
        type => "nginx"
        start_position => "beginning"
    }   
}

output {

    if [type] == "http" {
        redis {
            host => "10.1.0.103"
            password => 'foobared'
            port => "6379"
            db => "2" 
            data_type => "list"
            key => 'test_http'
        }
    }

    if [type] == "nginx" {
        redis {
            host => "10.1.0.103"
            password => 'foobared'
            port => "6379"
            db => "2" 
            data_type => "list"
            key => 'test_nginx'
        }
    }
}

7、启动logstask

nohup logstash -f logstash.conf >/dev/null 2>&1 &

8、查看redis

127.0.0.1:6379> select 2
OK
127.0.0.1:6379[2]> KEYS *
1) "test_http"
2) "test_nginx"

8、在另一台机器（10.1.0.102）安装logstash

安装可参考elk系列之elk单机部署（https://blog.csdn.net/weixin_44320761/article/details/121333670）

9、配置

vim logstash.conf
input {
    redis {
        type => "http"
        host => "10.1.0.103"
        password => 'foobared'
        port => "6379"
        db => "2"
        data_type => "list"
        key => 'test_http' 
        batch_count => 1
     }

    redis {
        type => "nginx"
        host => "10.1.0.103"
        password => 'foobared'
        port => "6379"
        db => "2"
        data_type => "list"
        key => 'test_nginx'
        batch_count => 1
     }
}
    
output {
    if [type] == "http" {
        elasticsearch {
            hosts => ["10.1.0.103:9200"]
            index => "redis-http-%{+YYYY.MM.dd}"
        }   
    }   

    if [type] == "nginx" {
        elasticsearch {
            hosts => ["10.1.0.103:9200"]
            index => "redis-nginx-%{+YYYY.MM.dd}"
        }   
    } 
} 

10、启动logstash

nohup logstash -f logstash.conf >/dev/null 2>&1 &

11、查看

1图为elasticsearch-head查看的结果
2图为添加index时的结果
3图为在页面查看到的日志