Redis5集群搭建

前言

随着微服务架构的不断通用化，作为微服务缓存的redis也必须进行响应的分布式部署，共享缓存。

环境

1. vmwarestation 14.*
2. centos7.0
3. redis5.04

过程

1. 下载

    wget http://download.redis.io/releases/redis-2.8.17.tar.gz
    或者
    官网下载
   

2. 编译、安装（src）目录

    make && make install
   

   常见错误：

    如果报错提示缺少gcc，则安装gcc ：  yum install -y gcc
    如果报错显示：jemalloc/jemalloc.h: No such file or directory，则在make时添加参数
    make MALLOC=libc
    之后再make install
   

3. 创建配置文件
   创建cluster-test文件夹，并创建几个节点文件夹（我创建了6个）

    [root@localhost soft]# cd cluster-test/
    [root@localhost cluster-test]# ll
    total 0
    drwxr-xr-x. 2 root root 69 Apr 24 19:43 7000
    drwxr-xr-x. 2 root root 69 Apr 24 19:48 7001
    drwxr-xr-x. 2 root root 69 Apr 24 19:48 7002
    drwxr-xr-x. 2 root root 69 Apr 24 19:48 7003
    drwxr-xr-x. 2 root root 69 Apr 24 19:48 7004
    drwxr-xr-x. 2 root root 69 Apr 24 19:48 7005
    [root@localhost cluster-test]# 
   

   创建配置文件：redis.conf,并增加如下配置

    [root@localhost 7000]# cat redis.conf 
    bind 192.168.254.131#绑定ip
    port 7000#端口
    dir /opt/soft/cluster-test/7000#文件地址
    daemonize yes#后台运行
    cluster-enabled yes#开启集群
    cluster-config-file nodes-7000.conf#集群配置文件
    cluster-node-timeout 15000#集群节点失效时长
    appendonly yes#持久化方式
    [root@localhost 7000]# 
   

   拷贝配置文件到相应的文件夹，并做相应的修改。

4. 启动节点redis

    [root@localhost 7000]# redis-server /opt/soft/cluster-test/7000/redis.conf 
    [root@localhost 7000]# redis-server /opt/soft/cluster-test/7001/redis.conf 
    [root@localhost 7000]# redis-server /opt/soft/cluster-test/7002/redis.conf 
    [root@localhost 7000]# redis-server /opt/soft/cluster-test/7003/redis.conf 
    [root@localhost 7000]# redis-server /opt/soft/cluster-test/7004/redis.conf 
    [root@localhost 7000]# redis-server /opt/soft/cluster-test/7005/redis.conf 
   

   查看是否启动成功：ps -ef | grep redis-server

    [root@localhost 7000]# ps -ef | grep redis-server
    root       2128      1  0 19:43 ?        00:00:00 redis-server 192.168.254.131:7000 [cluster]
    root       2141      1  0 19:48 ?        00:00:00 redis-server 192.168.254.131:7001 [cluster]
    root       2148      1  0 19:48 ?        00:00:00 redis-server 192.168.254.131:7002 [cluster]
    root       2153      1  0 19:48 ?        00:00:00 redis-server 192.168.254.131:7003 [cluster]
    root       2160      1  0 19:48 ?        00:00:00 redis-server 192.168.254.131:7004 [cluster]
    root       2165      1  0 19:48 ?        00:00:00 redis-server 192.168.254.131:7005 [cluster]
    root       2170   2020  0 19:48 pts/0    00:00:00 grep --color=auto redis-server
   

5. 创建集群
   旧版本的集群创建方式：

    [root@localhost src]# ./redis-trib.rb create --replicas 1 192.168.254.131:7000 192.168.254.131:7001 192.168.254.131:7002 192.168.254.131:7003 192.168.254.131:7004 192.168.254.131:7005
    WARNING: redis-trib.rb is not longer available!
    You should use redis-cli instead.
    
    All commands and features belonging to redis-trib.rb have been moved
    to redis-cli.
    In order to use them you should call redis-cli with the --cluster
    option followed by the subcommand name, arguments and options.
    
    Use the following syntax:
    redis-cli --cluster SUBCOMMAND [ARGUMENTS] [OPTIONS]
    
    Example:
    redis-cli --cluster create 192.168.254.131:7000 192.168.254.131:7001 192.168.254.131:7002 192.168.254.131:7003 192.168.254.131:7004 192.168.254.131:7005 --cluster-replicas 1
    
    To get help about all subcommands, type:
    redis-cli --cluster help
   

   可以看到新版本下创建集群已经改为了使用 redis-cli --cluster的方式。

    		[root@localhost 7002]# redis-cli --cluster create 192.168.254.131:7000 192.168.254.131:7001 192.168.254.131:7002 192.168.254.131:7003 192.168.254.131:7004 192.168.254.131:7005 --cluster-replicas 1
    >>> Performing hash slots allocation on 6 nodes...
    Master[0] -> Slots 0 - 5460
    Master[1] -> Slots 5461 - 10922
    Master[2] -> Slots 10923 - 16383
    Adding replica 192.168.254.131:7004 to 192.168.254.131:7000
    Adding replica 192.168.254.131:7005 to 192.168.254.131:7001
    Adding replica 192.168.254.131:7003 to 192.168.254.131:7002
    >>> Trying to optimize slaves allocation for anti-affinity
    [WARNING] Some slaves are in the same host as their master
    M: 7d68d4f7653b3358b9d2e10e42f5a105a5d67c46 192.168.254.131:7000
       slots:[0-5460] (5461 slots) master
    M: cfe0ad38eb291229b250f27dddfeb0dd0c2581a3 192.168.254.131:7001
       slots:[5461-10922] (5462 slots) master
    M: 9a0f7b1cf8283525b3ad1e099021ed0639dc81a6 192.168.254.131:7002
       slots:[10923-16383] (5461 slots) master
    S: 51666c396b86df64f5d206ca1e63a47251bebf44 192.168.254.131:7003
       replicates 7d68d4f7653b3358b9d2e10e42f5a105a5d67c46
    S: 4b8be39dd071399da1749b47fbd8a2a539f795cc 192.168.254.131:7004
       replicates cfe0ad38eb291229b250f27dddfeb0dd0c2581a3
    S: 5f865575b54fc9ee89b63c412833f023c4776002 192.168.254.131:7005
       replicates 9a0f7b1cf8283525b3ad1e099021ed0639dc81a6
    Can I set the above configuration? (type 'yes' to accept): yes
    >>> Nodes configuration updated
    >>> Assign a different config epoch to each node
    >>> Sending CLUSTER MEET messages to join the cluster
    Waiting for the cluster to join
    .....
    >>> Performing Cluster Check (using node 192.168.254.131:7000)
    M: 7d68d4f7653b3358b9d2e10e42f5a105a5d67c46 192.168.254.131:7000
       slots:[0-5460] (5461 slots) master
       1 additional replica(s)
    S: 51666c396b86df64f5d206ca1e63a47251bebf44 192.168.254.131:7003
       slots: (0 slots) slave
       replicates 7d68d4f7653b3358b9d2e10e42f5a105a5d67c46
    M: cfe0ad38eb291229b250f27dddfeb0dd0c2581a3 192.168.254.131:7001
       slots:[5461-10922] (5462 slots) master
       1 additional replica(s)
    M: 9a0f7b1cf8283525b3ad1e099021ed0639dc81a6 192.168.254.131:7002
       slots:[10923-16383] (5461 slots) master
       1 additional replica(s)
    S: 5f865575b54fc9ee89b63c412833f023c4776002 192.168.254.131:7005
       slots: (0 slots) slave
       replicates 9a0f7b1cf8283525b3ad1e099021ed0639dc81a6
    S: 4b8be39dd071399da1749b47fbd8a2a539f795cc 192.168.254.131:7004
       slots: (0 slots) slave
       replicates cfe0ad38eb291229b250f27dddfeb0dd0c2581a3
    [OK] All nodes agree about slots configuration.
    >>> Check for open slots...
    >>> Check slots coverage...
    [OK] All 16384 slots covered.
   

6. 连接测试

    [root@localhost cluster-test]# redis-cli -c -h 192.168.254.131 -p 7000
    192.168.254.131:7000> info
    # Server
    redis_version:5.0.4    -----------------------版本
    redis_git_sha1:00000000   ----------加密
    redis_git_dirty:0   ------不知道
    redis_build_id:b7228e50c523d596 --------build-id
    redis_mode:cluster     ----------模式：集群
    os:Linux 3.10.0-862.14.4.el7.x86_64 x86_64   -----------------系统内核
    arch_bits:64
    multiplexing_api:epoll
    atomicvar_api:atomic-builtin
    gcc_version:4.8.5   -----gcc版本
    process_id:2128     ----进程id
    run_id:3cb0cdaca501e48afbe9433e680935d838597856
    tcp_port:7000  ---端口
    uptime_in_seconds:2575   ----运行时长
    uptime_in_days:0
    hz:10
    configured_hz:10
    lru_clock:12604937
    executable:/opt/soft/cluster-test/redis-server  ----------执行位置
    config_file:/opt/soft/cluster-test/7000/redis.conf   -----配置文件
    
    # Clients    -----------------客户端信息
    connected_clients:1    -----连接的客户端
    client_recent_max_input_buffer:2
    client_recent_max_output_buffer:0
    blocked_clients:0
    
    # Memory    -----内存信息
    used_memory:1443312
    used_memory_human:1.38M
    used_memory_rss:3678208
    used_memory_rss_human:3.51M
    used_memory_peak:1443312
    used_memory_peak_human:1.38M
    used_memory_peak_perc:100.00%
    used_memory_overhead:1441726
    used_memory_startup:1392032
    used_memory_dataset:1586
    used_memory_dataset_perc:3.09%
    allocator_allocated:1409472
    allocator_active:3640320
    allocator_resident:3640320
    total_system_memory:1910018048
    total_system_memory_human:1.78G
    used_memory_lua:37888
    used_memory_lua_human:37.00K
    used_memory_scripts:0
    used_memory_scripts_human:0B
    number_of_cached_scripts:0
    maxmemory:0
    maxmemory_human:0B
    maxmemory_policy:noeviction
    allocator_frag_ratio:2.58
    allocator_frag_bytes:2230848
    allocator_rss_ratio:1.00
    allocator_rss_bytes:0
    rss_overhead_ratio:1.01
    rss_overhead_bytes:37888
    mem_fragmentation_ratio:2.61
    mem_fragmentation_bytes:2268736
    mem_not_counted_for_evict:0
    mem_replication_backlog:0
    mem_clients_slaves:0
    mem_clients_normal:49694
    mem_aof_buffer:0
    mem_allocator:libc
    active_defrag_running:0
    lazyfree_pending_objects:0
    
    # Persistence    -----持久化信息
    loading:0
    rdb_changes_since_last_save:0
    rdb_bgsave_in_progress:0
    rdb_last_save_time:1556106234
    rdb_last_bgsave_status:ok
    rdb_last_bgsave_time_sec:-1
    rdb_current_bgsave_time_sec:-1
    rdb_last_cow_size:0
    aof_enabled:1
    aof_rewrite_in_progress:0
    aof_rewrite_scheduled:0
    aof_last_rewrite_time_sec:-1
    aof_current_rewrite_time_sec:-1
    aof_last_bgrewrite_status:ok
    aof_last_write_status:ok
    aof_last_cow_size:0
    aof_current_size:0
    aof_base_size:0
    aof_pending_rewrite:0
    aof_buffer_length:0
    aof_rewrite_buffer_length:0
    aof_pending_bio_fsync:0
    aof_delayed_fsync:0
    
    # Stats      ---------状态
    total_connections_received:2
    total_commands_processed:5
    instantaneous_ops_per_sec:0
    total_net_input_bytes:114
    total_net_output_bytes:18698
    instantaneous_input_kbps:0.00
    instantaneous_output_kbps:0.00
    rejected_connections:0
    sync_full:0
    sync_partial_ok:0
    sync_partial_err:0
    expired_keys:0
    expired_stale_perc:0.00
    expired_time_cap_reached_count:0
    evicted_keys:0
    keyspace_hits:0
    keyspace_misses:0
    pubsub_channels:0
    pubsub_patterns:0
    latest_fork_usec:0
    migrate_cached_sockets:0
    slave_expires_tracked_keys:0
    active_defrag_hits:0
    active_defrag_misses:0
    active_defrag_key_hits:0
    active_defrag_key_misses:0
    
    # Replication        ----------------集群信息
    role:master               ------------------master
    connected_slaves:0
    master_replid:fb37775be67657aaf26eeac00e736cefa0669f67
    master_replid2:0000000000000000000000000000000000000000
    master_repl_offset:0
    second_repl_offset:-1
    repl_backlog_active:0
    repl_backlog_size:1048576
    repl_backlog_first_byte_offset:0
    repl_backlog_histlen:0
    
    # CPU
    used_cpu_sys:1.578260
    used_cpu_user:1.303976
    used_cpu_sys_children:0.000000
    used_cpu_user_children:0.000000
    
    # Cluster
    cluster_enabled:1
    
    # Keyspace
    192.168.254.131:7000> exit
   

查看集群状态cluster info 或者 cluster nodes

	[root@localhost 7002]# redis-cli --cluster create 192.168.254.131:7000 192.168.254.131:7001 192.168.254.131:7002 192.168.254.131:7003 192.168.254.131:7004 192.168.254.131:7005 --cluster-replicas 1
	>>> Performing hash slots allocation on 6 nodes...
	Master[0] -> Slots 0 - 5460
	Master[1] -> Slots 5461 - 10922
	Master[2] -> Slots 10923 - 16383
	Adding replica 192.168.254.131:7004 to 192.168.254.131:7000
	Adding replica 192.168.254.131:7005 to 192.168.254.131:7001
	Adding replica 192.168.254.131:7003 to 192.168.254.131:7002
	>>> Trying to optimize slaves allocation for anti-affinity
	[WARNING] Some slaves are in the same host as their master
	M: 7d68d4f7653b3358b9d2e10e42f5a105a5d67c46 192.168.254.131:7000
	   slots:[0-5460] (5461 slots) master
	M: cfe0ad38eb291229b250f27dddfeb0dd0c2581a3 192.168.254.131:7001
	   slots:[5461-10922] (5462 slots) master
	M: 9a0f7b1cf8283525b3ad1e099021ed0639dc81a6 192.168.254.131:7002
	   slots:[10923-16383] (5461 slots) master
	S: 51666c396b86df64f5d206ca1e63a47251bebf44 192.168.254.131:7003
	   replicates 7d68d4f7653b3358b9d2e10e42f5a105a5d67c46
	S: 4b8be39dd071399da1749b47fbd8a2a539f795cc 192.168.254.131:7004
	   replicates cfe0ad38eb291229b250f27dddfeb0dd0c2581a3
	S: 5f865575b54fc9ee89b63c412833f023c4776002 192.168.254.131:7005
	   replicates 9a0f7b1cf8283525b3ad1e099021ed0639dc81a6
	Can I set the above configuration? (type 'yes' to accept): yes
	>>> Nodes configuration updated
	>>> Assign a different config epoch to each node
	>>> Sending CLUSTER MEET messages to join the cluster
	Waiting for the cluster to join
	.....
	>>> Performing Cluster Check (using node 192.168.254.131:7000)
	M: 7d68d4f7653b3358b9d2e10e42f5a105a5d67c46 192.168.254.131:7000
	   slots:[0-5460] (5461 slots) master
	   1 additional replica(s)
	S: 51666c396b86df64f5d206ca1e63a47251bebf44 192.168.254.131:7003
	   slots: (0 slots) slave
	   replicates 7d68d4f7653b3358b9d2e10e42f5a105a5d67c46
	M: cfe0ad38eb291229b250f27dddfeb0dd0c2581a3 192.168.254.131:7001
	   slots:[5461-10922] (5462 slots) master
	   1 additional replica(s)
	M: 9a0f7b1cf8283525b3ad1e099021ed0639dc81a6 192.168.254.131:7002
	   slots:[10923-16383] (5461 slots) master
	   1 additional replica(s)
	S: 5f865575b54fc9ee89b63c412833f023c4776002 192.168.254.131:7005
	   slots: (0 slots) slave
	   replicates 9a0f7b1cf8283525b3ad1e099021ed0639dc81a6
	S: 4b8be39dd071399da1749b47fbd8a2a539f795cc 192.168.254.131:7004
	   slots: (0 slots) slave
	   replicates cfe0ad38eb291229b250f27dddfeb0dd0c2581a3
	[OK] All nodes agree about slots configuration.
	>>> Check for open slots...
	>>> Check slots coverage...
	[OK] All 16384 slots covered.

8. 测试集群数据
   xshell中复制一个ssh连接，一个链接到7000端口的redis，一个链接到7004的redis（目前7000为master，并且它的slave是7004）

   7000端口的redis

    192.168.254.131:7000> set user jatham
    -> Redirected to slot [5474] located at 192.168.254.131:7001
    OK
    192.168.254.131:7001> get user
    "jatham"
    192.168.254.131:7001>
   

   7004端口的redis

    [root@localhost 7000]# redis-cli -c -h 192.168.254.131 -p 7004
    192.168.254.131:7004> get user
    -> Redirected to slot [5474] located at 192.168.254.131:7001 ---这里它做了一个重定向到master槽的操作
    "jatham"
    192.168.254.131:7001> 
   

   可以看到7000的slave中也查看到了7000中设置的值。那其他的master和slave节点呢？是否也能正常获取到master7000中设置的值？答案是肯定的。

    [root@localhost 7000]# redis-cli -c -h 192.168.254.131 -p 7001
    192.168.254.131:7001> get user
    "jatham"
    192.168.254.131:7001> 
   

还有一种情况，在slave节点设置值，其对应的master节点能否正常获取到值呢？测试下：7005slave节点下设置user的值

[root@localhost 7002]# redis-cli -c -h 192.168.254.131 -p 7005
192.168.254.131:7005> set user other-jatham
-> Redirected to slot [5474] located at 192.168.254.131:7001
OK
192.168.254.131:7001> 

###7001master节点

192.168.254.131:7001> get user
"other-jatham"
192.168.254.131:7001> 

###7000master节点
[root@localhost 7000]# redis-cli -c -h 192.168.254.131 -p 7000
192.168.254.131:7000> get user
-> Redirected to slot [5474] located at 192.168.254.131:7001
"other-jatham"
192.168.254.131:7001> -------------这里竟然到了7001

发现我在7000节点访问7005设置的值，结果转向了7005的master节点7001.而且无论在哪里获取设置的值，都会重定向到[5471]slot。

注意：设置值的时候有一个这样的提示

Redirected to slot [5474] located at 192.168.254.131:7001

说明数据存储到了这个槽内。

9. 原理
   做了这些操作，是不是对其中的一些实现很疑惑，反正我是挺疑惑的，贴一个链接

官方文档：官方教学
原理：redis分槽原理
其他版本配置方式：redis个版本配置方式