一、环境准备
三台虚拟机，每台主机上运行两个redis实例（主从架构，监听不同的端口，6379和6380），总共6个redis实例:
192.168.239.130：6379/6380
192.168.239.128：6379/6380
192.168.239.132：6379/6380
三台主机上分别部署redis服务（具体部署方式参见：redis部署）
可以先通过编译安装方式部署好一个实例后，在把redis的安装目录及相关配置文件打包复制到其余两台主机上，同时还要复制自己编写的service文件到/usr/lib/systemd/system/目录下用于通过systemd来管理redis服务。
在部署第二个实例时，同样可以把第一个实例的安装目录及相关配置文件复制一份出来，然后再修改redis.conf配置文件（主要修改监听端口，日志文件路径，pid文件路径，数据目录存放路径），同样也要修改一下service文件（主要改动启动时的指定的redis.conf的配置文件路径）
每台主机redis服务的目录结构如下
##两个redis的service服务，分别用于管理两个实例
[root@centos7 ~]# ls /usr/lib/systemd/system/redis
redis-6380.service redis.service
[root@centos7 ~]# ls /apps
redis redis_6380
##实例1
[root@centos7 ~]# ls /apps/redis
bin data etc logs run
##实例2
[root@centos7 ~]# ls /apps/redis_6380/
bin data etc logs run
启动服务，然后查看三台主机监听的端口
[root@centos7 ~]# ss -ntl | grep -e 6379 -e 6380
LISTEN 0 511 *:6379 *:*
LISTEN 0 511 *:6380 *:*
LISTEN 0 511 *:16379 *:*
LISTEN 0 511 *:16380 *:*
##16379和16380是用于集群通信的端口，通过修改redis.conf配置文件 clusterz-enabled yes实现
二、搭建集群
前提条件：
1、每个redis node都采取相同的密码，及相同的版本
2、每个节点都开启集群配置的相关参数
cluster-enabled yes
cluster-config-file nodes-xxxx.conf
3、所有redis服务器都没有任何数据，可以flushall清空
创建集群
我们使用的redis版本是4.0.14，在redis3和4版本中，创建redis集群需要用到redis-trib.rb这个集群管理工具，该工具在Redis源码包的src目录下。
此工具是依赖ruby环境的，但是系统自带的版本过低因此还得在每台主机上编译安装高版本的ruby
##三台主机上都要编译安装
[root@centos7 ~]# cd /usr/local/src
[root@centos7 src]# wget https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.5.tar.gz
[root@centos7 src]# tar xf ruby-2.5.5.tar.gz
[root@centos7 src]# cd ruby-2.5.5
[root@centos7 ruby-2.5.5]# ./configure
[root@centos7 ruby-2.5.5]# make -j 2 && make install
##安装redis模块
[root@centos7 ruby-2.5.5]#gem install redis -v 4.1.0
报错
通过gem install redis会报如下错误
ERROR: Loading command: install (LoadError)
cannot load such file -- zlib
ERROR: While executing gem ... (NoMethodError)
undefined method `invoke_with_build_args' for nil:NilClass
解决方法：
这是缺少zlib依赖，需要安装zlib库
yum install zlib-devel
进入到ruby安装目录:
[root@centos_7 ]# cd /usr/local/src/ruby-2.5.5/ext/zlib
[root@centos_7 zlib]# ruby ./extconf.rb
[root@centos_7 zlib]# make
#在编译时又报的错，需要zlib.o模块，但是路径不对，需要修改一下
报错内容：make: *** No rule to make target `/include/ruby.h', needed by `zlib.o'. Stop.
解决方法：
[root@centos_7 zlib]#vim Makefile
#zlib.o: $(top_srcdir)/include/ruby.h #找到此行修改为下面一行的代码，保存退出
zlib.o: ../../include/ruby.h
再次编译：
[root@centos_7 zlib]# make
compiling zlib.c
linking shared-object zlib.so #成功！！！
[root@centos_7 zlib]# make install
/usr/bin/install -c -m 0755 zlib.so /usr/local/lib/ruby/site_ruby/2.5.0/x86_64-linux
另一个错误：
gem install redis 安装时提示关于 openssl 的信息的报错
安装openssl库
yum install openssl-devel
集成openssl库到ruby
[root@centos_7 ]# cd /usr/local/src/ruby-2.5.5/ext/openssl
[root@centos_7 zlib]# ruby ./extconf.rb
[root@centos_7 zlib]#make
#在编译时又报的错
报错内容：make: *** No rule to make target `/include/ruby.h', needed by `ossl.o'. Stop.
解决方法：
[root@centos_7 zlib]#vim Makefile
将所有的$(top_srcdir)include/ruby.h改为../../include/ruby.h
进入：模式，输入%s/$(top_srcdir)/..\/../g，然后回车就全部改好了
再次编译：
[root@centos_7 zlib]# make
[root@centos_7 zlib]# make install
然后退出到ruby安装目录继续使用gem install redis
成功之后，再次运行gem install redis
gem install redis
报错解决之后可以运行一下redis-trib命令验证一下是否能够执行
redis-trib.rb
修改redis登录密码
##每台主机都得修改
vim /usr/local/lib/ruby/gems/2.5.0/gems/redis-4.1.0/lib/redis/client.rb
......
class Client
DEFAULTS = {
.......
:password => 123456,
.......
......
使用redis-trib.rb工具创建集群，无需手动指定master和slave，redis-trib.rb工具会自动配置
[root@centos7 ~]# redis-trib.rb create --replicas 1 192.168.239.130:6379 192.168.239.130:6380 192.168.239.128:6379 192.168.239.128:6380 192.168.239.132:6379 192.168.239.132:6380
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.239.130:6379
192.168.239.128:6379
192.168.239.132:6379
Adding replica 192.168.239.128:6380 to 192.168.239.130:6379
Adding replica 192.168.239.132:6380 to 192.168.239.128:6379
Adding replica 192.168.239.130:6380 to 192.168.239.132:6379
M: 860c2ab2d1edd19c747060e42b59331571434082 192.168.239.130:6379
slots:0-5460 (5461 slots) master
S: 20463a0f498b298a75f53d20d947fd25f28ffec1 192.168.239.130:6380
replicates f06306b80fc15a4dec5426aba88edb445236f2e2
M: 3ff70fb092cbd1576a20b756829cafb308d0908a 192.168.239.128:6379
slots:5461-10922 (5462 slots) master
S: 9d3d0dbf8c8469a9eeb595a0e52102380f41ec72 192.168.239.128:6380
replicates 860c2ab2d1edd19c747060e42b59331571434082
M: f06306b80fc15a4dec5426aba88edb445236f2e2 192.168.239.132:6379
slots:10923-16383 (5461 slots) master
S: 6d4558425c7cf902905e692e413e37f581daf71b 192.168.239.132:6380
replicates 3ff70fb092cbd1576a20b756829cafb308d0908a
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join....
>>> Performing Cluster Check (using node 192.168.239.130:6379)
M: 860c2ab2d1edd19c747060e42b59331571434082 192.168.239.130:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 9d3d0dbf8c8469a9eeb595a0e52102380f41ec72 192.168.239.128:6380
slots: (0 slots) slave
replicates 860c2ab2d1edd19c747060e42b59331571434082
S: 6d4558425c7cf902905e692e413e37f581daf71b 192.168.239.132:6380
slots: (0 slots) slave
replicates 3ff70fb092cbd1576a20b756829cafb308d0908a
M: f06306b80fc15a4dec5426aba88edb445236f2e2 192.168.239.132:6379
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: 3ff70fb092cbd1576a20b756829cafb308d0908a 192.168.239.128:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 20463a0f498b298a75f53d20d947fd25f28ffec1 192.168.239.130:6380
slots: (0 slots) slave
replicates f06306b80fc15a4dec5426aba88edb445236f2e2
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
出现长时间处于Waiting for the cluster to join…状态
1、配置文件redis.conf 中的bind 设置，IP要是本机地址
2、确保所有使用的端口之间互通（6379/16379/6380、16380），可用telnet ip port 测试
3、登录到每个客户端，执行 flushall、 cluster reset
虽然集群已经运行，但是由于未设置masterauth认证，所以主从还未真正建立起来
master节点显示连接的slave为0
slave节点也显示与master的连接状态为down状态
需要分别在每台salve节点控制台上设置masterauth密码才能实现主从连接（最好是在客户端设置密码后并写入配置文件中，这样能实现持久保存）
##每个slave节点分别设置masterauth
[root@centos7 ansible]# redis-cli -h 192.168.239.130 -p 6380 -a 123456
Warning: Using a password with '-a' option on the command line interface may not be safe.
192.168.239.130:6380> config set masterauth 123456
OK
[root@centos7 ansible]# redis-cli -h 192.168.239.128 -p 6380 -a 123456
Warning: Using a password with '-a' option on the command line interface may not be safe.
192.168.239.128:6380> config set masterauth 123456
OK
[root@centos7 ansible]# redis-cli -h 192.168.239.132 -p 6380 -a 123456
Warning: Using a password with '-a' option on the command line interface may not be safe.
192.168.239.132:6380> config set masterauth 123456
OK
确认slave状态为up
[root@centos7 ansible]# ansible redis -m shell -a 'redis-cli -p 6380 -a 123456 info replication'
192.168.239.128 | CHANGED | rc=0 >>
# Replication
role:slave
master_host:192.168.239.130
master_port:6379
master_link_status:up
master_last_io_seconds_ago:2
master_sync_in_progress:0
slave_repl_offset:588
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:945f4352b9be9975a3a5849898613078bebdcaf7
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:588
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:588Warning: Using a password with '-a' option on the command line interface may not be safe.
192.168.239.132 | CHANGED | rc=0 >>
# Replication
role:slave
master_host:192.168.239.128
master_port:6379
master_link_status:up
master_last_io_seconds_ago:6
master_sync_in_progress:0
slave_repl_offset:560
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:3c3b7f8ee3f09aca7a78218802f80aba63fc6194
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:560
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:560Warning: Using a password with '-a' option on the command line interface may not be safe.
192.168.239.130 | CHANGED | rc=0 >>
# Replication
role:slave
master_host:192.168.239.132
master_port:6379
master_link_status:up
master_last_io_seconds_ago:9
master_sync_in_progress:0
slave_repl_offset:616
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:7721e20948f762ed3a49757c7275d2c4620ee34a
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:616
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:616Warning: Using a password with '-a' option on the command line interface may not be safe.
[root@centos7 ansible]# ansible redis -m shell -a 'redis-cli -p 6380 -a 123456 info replication' | grep up
master_link_status:up
master_link_status:up
master_link_status:up
验证集群状态
[root@centos7 ruby-2.5.5]# redis-cli -p 6379 -a 123456 cluster info
Warning: Using a password with '-a' option on the command line interface may not be safe.
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:7
cluster_my_epoch:5
cluster_stats_messages_ping_sent:60667
cluster_stats_messages_pong_sent:58304
cluster_stats_messages_meet_sent:5
cluster_stats_messages_fail_sent:5
cluster_stats_messages_sent:118981
cluster_stats_messages_ping_received:58300
cluster_stats_messages_pong_received:60667
cluster_stats_messages_meet_received:4
cluster_stats_messages_fail_received:1
cluster_stats_messages_received:118972
[root@centos7 ansible]# ansible redis -m shell -a 'redis-cli -p 6379 -a 123456 cluster info' | grep state
cluster_state:ok
cluster_state:ok
cluster_state:ok
验证集群，写入一个key测试
[root@centos7 ansible]# redis-cli -h 192.168.239.130 -p 6379 -a 123456
Warning: Using a password with '-a' option on the command line interface may not be safe.
##经过算法计算，现在设定的key得写入指定的node，槽位不在130这个node上，所以无法写入
192.168.239.130:6379> set tom jerry
(error) MOVED 8919 192.168.239.128:6379
##根据提示到128节点上进行写入，这时成功写入
[root@centos7 ansible]# redis-cli -h 192.168.239.128 -p 6379 -a 123456
Warning: Using a password with '-a' option on the command line interface may not be safe.
192.168.239.128:6379> set tom jerry
OK
192.168.239.128:6379> keys *
1) "tom"
集群状态验证
[root@centos7 ansible]# redis-trib.rb check 192.168.239.128:6379
>>> Performing Cluster Check (using node 192.168.239.128:6379)
M: 3ff70fb092cbd1576a20b756829cafb308d0908a 192.168.239.128:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 20463a0f498b298a75f53d20d947fd25f28ffec1 192.168.239.130:6380
slots: (0 slots) slave
replicates f06306b80fc15a4dec5426aba88edb445236f2e2
S: 6d4558425c7cf902905e692e413e37f581daf71b 192.168.239.132:6380
slots: (0 slots) slave
replicates 3ff70fb092cbd1576a20b756829cafb308d0908a
M: 860c2ab2d1edd19c747060e42b59331571434082 192.168.239.130:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 9d3d0dbf8c8469a9eeb595a0e52102380f41ec72 192.168.239.128:6380
slots: (0 slots) slave
replicates 860c2ab2d1edd19c747060e42b59331571434082
M: f06306b80fc15a4dec5426aba88edb445236f2e2 192.168.239.132:6379
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
######
[root@centos7 ansible]# redis-trib.rb info 192.168.239.128:6379
192.168.239.128:6379 (3ff70fb0...) -> 1 keys | 5462 slots | 1 slaves.
192.168.239.130:6379 (860c2ab2...) -> 0 keys | 5461 slots | 1 slaves.
192.168.239.132:6379 (f06306b8...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
1772
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
