##集群规划 要让Redis3.0
集群正常工作至少需要3 个Master
节点,要想实现高可用,每个Master
节点要配备至少1 个Slave
节点。根据以上特点和要求,进行如下的集群实施规划: 使用6 台服务器(物理机或虚拟机)部署3 个Master
+ 3 个Slave
;
IP | 服务端口(默认6379) | 集群端口(服务端口数+10000) |
---|---|---|
10.1.200.77 | 7111 | 17111 |
10.1.200.78 | 7112 | 17112 |
10.1.200.87 | 7113 | 17113 |
10.1.200.118 | 7114 | 17114 |
10.1.200.120 | 7115 | 17115 |
10.1.200.125 | 7116 | 17116 |
##安装 环境:CentOS 6.7
Redis版本:redis-3.0.7
安装目录:/usr/local/redis
安装步骤: 1、编译和安装所需的包:
yum install gcc tcl
2、创建安装目录:
mkdir /usr/local/redis
3、解压安装包:
tar -zxvf redis-3.0.7.tar.gz
cd /usr/local/redis-3.0.7
4、安装(使用PREFIX
指定安装目录):
make
make PREFIX=/usr/local/redis install
安装完成后,可以看到/usr/local/redis
目录下有一个bin
目录,bin
目录里就是redis
的命令脚本:
redis-benchmark
redis-check-aof
redis-check-dump
redis-cli
redis-server
##配置 ###将Redis添加到环境变量中 1、在每个节点如下操作:
vim /etc/profile
2、在最后添加以下内容:
export REDIS_HOME=/usr/local/redis
export PATH=$REDIS_HOME/bin:$PATH
3、使配置生效:
source /etc/profile
###设置防火墙端口 1、按规划:防火墙中打开相应的端口:
vim /etc/sysconfig/iptables
2、每个节点添加如下配置: 10.1.200.77
服务器:
-A INPUT -m state --state NEW -m tcp -p tcp --dport 7111 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 17111 -j ACCEPT
10.1.200.78
服务器:
-A INPUT -m state --state NEW -m tcp -p tcp --dport 7112 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 17112 -j ACCEPT
10.1.200.87
服务器:
-A INPUT -m state --state NEW -m tcp -p tcp --dport 7113 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 17113 -j ACCEPT
10.1.200.118
服务器:
-A INPUT -m state --state NEW -m tcp -p tcp --dport 7114 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 17114 -j ACCEPT
10.1.200.120
服务器:
-A INPUT -m state --state NEW -m tcp -p tcp --dport 7115 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 17115 -j ACCEPT
10.1.200.125
服务器:
-A INPUT -m state --state NEW -m tcp -p tcp --dport 7116 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 17116 -j ACCEPT
3、每个节点重启防火墙:
service iptables restart
###将Redis配置成服务 1、Redis
的启动脚本为:/usr/local/redis-3.0.7/utils/redis_init_script
在每个节点上将启动脚本复制到/etc/rc.d/init.d/
目录下,并命名为redis
:
cp /usr/local/redis-3.0.7/utils/redis_init_script /etc/rc.d/init.d/redis
2、编辑/etc/rc.d/init.d/redis
,修改相应配置,使之能注册成为服务:
vim /etc/rc.d/init.d/redis
#!/bin/sh
#
# Simple Redis init.d script conceived to work on Linux systems
# as it does use of the /proc filesystem.
REDISPORT=6379
EXEC=/usr/local/bin/redis-server
CLIEXEC=/usr/local/bin/redis-cli
PIDFILE=/var/run/redis_${REDISPORT}.pid
CONF="/etc/redis/${REDISPORT}.conf"
case "$1" in
start)
if [ -f $PIDFILE ]
then
echo "$PIDFILE exists, process is already running or crashed"
else
echo "Starting Redis server..."
$EXEC $CONF
fi
;;
stop)
if [ ! -f $PIDFILE ]
then
echo "$PIDFILE does not exist, process is not running"
else
PID=$(cat $PIDFILE)
echo "Stopping ..."
$CLIEXEC -p $REDISPORT shutdown
while [ -x /proc/${PID} ]
do
echo "Waiting for Redis to shutdown ..."
sleep 1
done
echo "Redis stopped"
fi
;;
*)
echo "Please use start or stop as first argument"
;;
esac
查看以上Redis
服务脚本,做如下几个修改: 1)在脚本的第一行后面添加一行内容如下:
#chkconfig: 2345 80 90
注:如果不添加上面的内容,在注册服务时会提示:
service redis does not support chkconfig
2)REDISPORT
端口修改各节点对应的端口。
注:端口名将与下面的配置文件名有关。
3)EXEC=/usr/local/bin/redis-server
改为 EXEC=/usr/local/redis/bin/redis-server
4)CLIEXEC=/usr/local/bin/redis-cli
改为CLIEXEC=/usr/local/redis/bin/redis-cli
5) 配置文件设置,对CONF
属性作如下调整: CONF="/etc/redis/${REDISPORT}.conf"
改为 CONF="/usr/local/redis/cluster/${REDISPORT}/redis-${REDISPORT}.conf"
6)更改Redis
开启的命令,以后台运行的方式执行:
$EXEC $CONF &
注:“&”作用是将服务转到后面运行
修改后的/etc/rc.d/init.d/redis
服务脚本内容为:
#!/bin/sh
#chkconfig: 2345 80 90
#
# Simple Redis init.d script conceived to work on Linux systems
# as it does use of the /proc filesystem.
REDISPORT=7111
EXEC=/usr/local/redis/bin/redis-server
CLIEXEC=/usr/local/redis/bin/redis-cli
PIDFILE=/var/run/redis_${REDISPORT}.pid
CONF="/usr/local/redis/cluster/${REDISPORT}/redis-${REDISPORT}.conf"
case "$1" in
start)
if [ -f $PIDFILE ]
then
echo "$PIDFILE exists, process is already running or crashed"
else
echo "Starting Redis server..."
$EXEC $CONF &
fi
;;
stop)
if [ ! -f $PIDFILE ]
then
echo "$PIDFILE does not exist, process is not running"
else
PID=$(cat $PIDFILE)
echo "Stopping ..."
$CLIEXEC -p $REDISPORT shutdown
while [ -x /proc/${PID} ]
do
echo "Waiting for Redis to shutdown ..."
sleep 1
done
echo "Redis stopped"
fi
;;
*)
echo "Please use start or stop as first argument"
;;
esac
3、以上配置操作完成后,便可将Redis
注册成为服务:
chkconfig --add redis
###修改Redis配置文件 1、在每个节点创建集群配置目录 ,并拷贝 redis.conf
配置文件到各节点配置目录 10.1.200.77
服务器:
mkdir -p /usr/local/redis/cluster/7111
cp /usr/local/redis-3.0.7/redis.conf /usr/local/redis/cluster/7111/redis-7111.conf
10.1.200.78
服务器:
mkdir -p /usr/local/redis/cluster/7112
cp /usr/local/redis-3.0.7/redis.conf /usr/local/redis/cluster/7112/redis-7112.conf
10.1.200.87
服务器:
mkdir -p /usr/local/redis/cluster/7113
cp /usr/local/redis-3.0.7/redis.conf /usr/local/redis/cluster/7113/redis-7113.conf
10.1.200.118
服务器:
mkdir -p /usr/local/redis/cluster/7114
cp /usr/local/redis-3.0.7/redis.conf /usr/local/redis/cluster/7114/redis-7114.conf
10.1.200.120
服务器:
mkdir -p /usr/local/redis/cluster/7115
cp /usr/local/redis-3.0.7/redis.conf /usr/local/redis/cluster/7115/redis-7115.conf
10.1.200.125
服务器:
mkdir -p /usr/local/redis/cluster/7116
cp /usr/local/redis-3.0.7/redis.conf /usr/local/redis/cluster/7116/redis-7116.conf
2、在6个节点的redis.conf
配置文件内容,注意修改下红色字体部分的内容即可,其他都相同:
配置选项 | 选项值 | 说明 |
---|---|---|
daemonize | yes | 是否作为守护进程运行 |
pidfile | /var/run/redis-7111 .pid | 如以后台进程运行,则需指定一个pid,默认为/var/run/redis.pid |
port | 7111 | 监听端口,默认为6379。注意:集群通讯端口值默认为此端口值+10000,如17111 |
databases | 1 | 用数据库数,默认值为16,默认数据库存储在DB 0号ID库中,无特殊需求,建议仅设置一个数据库 databases 1 |
cluster-enabled | yes | 打开redis集群 |
cluster-config-file | /usr/local/redis/cluster/7111 /nodes.conf | 集群配置文件(启动自动生成),不用人为干涉 |
cluster-node-timeout | 15000 | 节点互连超时时间。毫秒 |
cluster-migration-barrier | 1 | 数据迁移的副本临界数,这个参数表示的是,一个主节点在拥有多少个好的从节点的时候就要割让一个从节点出来给另一个没有任何从节点的主节点。 |
cluster-require-full-coverage | yes | 如果某一些key space没有被集群中任何节点覆盖,集群将停止接受写入。 |
appendonly | yes | 启用aof持久化方式。因为redis本身同步数据文件是按上面save条件来同步的,所以有的数据会在一段时间内只存在于内存中。默认值为no |
dir | /usr/local/redis/cluster/7111 | 节点数据持久化存放目录(建议配置) |
包含了最少选项的集群配置文件示例如下: | ||
port:7000 | ||
cluster-enabled:yes | ||
cluster-config-file:nodes.conf | ||
cluster-node-timeout:5000 | ||
appendonly:yes |
###启动每个节点 1、使用如下命令启动这6个Redis
节点实例: 10.1.200.77
服务器:
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/7111/redis-7111.conf
10.1.200.78
服务器:
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/7112/redis-7112.conf
10.1.200.87
服务器:
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/7113/redis-7113.conf
10.1.200.118
服务器:
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/7114/redis-7114.conf
10.1.200.120
服务器:
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/7115/redis-7115.conf
10.1.200.125
服务器:
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/7116/redis-7116.conf
2、启动之后用PS
命令查看实例启动情况:
ps -ef | grep redis
10.1.200.77
服务器:
[root@centos4 7111]# ps -ef | grep redis
root 14251 1 0 13:37 ? 00:00:00 /usr/local/redis/bin/redis-server *:7111 [cluster]
root 14259 13698 0 13:40 pts/0 00:00:00 grep redis
10.1.200.78
服务器:
[root@centos4 ~]# ps -ef | grep redis
root 12123 1 0 13:38 ? 00:00:00 /usr/local/redis/bin/redis-server *:7112 [cluster]
root 12144 11666 0 13:41 pts/0 00:00:00 grep redis
10.1.200.87
服务器:
[root@centos4 ~]# ps -ef | grep redis
root 12063 1 0 13:38 ? 00:00:00 /usr/local/redis/bin/redis-server *:7113 [cluster]
root 12079 11603 0 13:42 pts/0 00:00:00 grep redis
10.1.200.118
服务器:
[root@centos4 ~]# ps -ef | grep redis
root 9509 1 0 13:39 ? 00:00:00 /usr/local/redis/bin/redis-server *:7114 [cluster]
root 9541 8125 0 13:43 pts/0 00:00:00 grep redis
10.1.200.120
服务器:
[root@centos4 ~]# ps -ef | grep redis
root 9497 1 0 13:39 ? 00:00:00 /usr/local/redis/bin/redis-server *:7115 [cluster]
root 9529 8116 0 13:43 pts/0 00:00:00 grep redis
10.1.200.125
服务器:
[root@centos4 ~]# ps -ef | grep redis
root 9316 1 0 13:39 ? 00:00:00 /usr/local/redis/bin/redis-server *:7116 [cluster]
root 9352 7955 0 13:43 pts/0 00:00:00 grep redis
注意:启动完毕后,6个
Redis
实例尚未构成集群。
##创建集群 1、在每个节点上安装ruby
和rubygems
(注意:需要ruby
的版本在1.8.7
以上)
yum install ruby rubygems
2、检查ruby
版本:
[root@centos4 ~]# ruby -v
ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux]
3、替换rubygems.org
镜像:
[root@centos4 ~]# gem sources --add https://ruby.taobao.org/ --remove http://rubygems.org/
https://ruby.taobao.org/ added to sources
http://rubygems.org/ removed from sources
[root@centos4 ~]# gem sources -l
*** CURRENT SOURCES ***
https://ruby.taobao.org/
4、gem
安装redis ruby
接口:
[root@centos4 ~]# gem install redis
Successfully installed redis-3.3.1
1 gem installed
Installing ri documentation for redis-3.3.1...
Installing RDoc documentation for redis-3.3.1...
5、执行Redis
集群创建命令(只需要在其中一个节点上执行一次则可):
[root@centos4 bin]# cp /usr/local/redis-3.0.7/src/redis-trib.rb /usr/local/redis/bin/redis-trib
[root@centos4 bin]# redis-trib create --replicas 1 10.1.200.118:7114 10.1.200.120:7115 10.1.200.125:7116 10.1.200.77:7111 10.1.200.78:7112 10.1.200.87:7113
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
10.1.200.125:7116
10.1.200.120:7115
10.1.200.87:7113
Adding replica 10.1.200.118:7114 to 10.1.200.125:7116
Adding replica 10.1.200.78:7112 to 10.1.200.120:7115
Adding replica 10.1.200.77:7111 to 10.1.200.87:7113
S: 967f4154bc4ced3c0c76a74c280f2779f2f70fb2 10.1.200.118:7114
replicates 7226e609e47616e0d3cff2f80831ac6d811e0027
M: 778b0cd6414d92ad4aec93754445648fc57963ca 10.1.200.120:7115
slots:5461-10922 (5462 slots) master
M: 7226e609e47616e0d3cff2f80831ac6d811e0027 10.1.200.125:7116
slots:0-5460 (5461 slots) master
S: c812379709120ab74b20bd76dd2b9fcaf6c62209 10.1.200.77:7111
replicates bb79ff3fdf24a99df6ebb93d17f026bb81eb221b
S: e1ced4f32594b0d2b82b268059e2ce89abebfde5 10.1.200.78:7112
replicates 778b0cd6414d92ad4aec93754445648fc57963ca
M: bb79ff3fdf24a99df6ebb93d17f026bb81eb221b 10.1.200.87:7113
slots:10923-16383 (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
(输入 yes 并按下回车确认之后,集群就会将配置应用到各个节点,并连接起(join)各个节点,也就是让各个节点开始互相通讯)
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join....
>>> Performing Cluster Check (using node 10.1.200.118:7114)
M: 967f4154bc4ced3c0c76a74c280f2779f2f70fb2 10.1.200.118:7114
slots: (0 slots) master
replicates 7226e609e47616e0d3cff2f80831ac6d811e0027
M: 778b0cd6414d92ad4aec93754445648fc57963ca 10.1.200.120:7115
slots:5461-10922 (5462 slots) master
M: 7226e609e47616e0d3cff2f80831ac6d811e0027 10.1.200.125:7116
slots:0-5460 (5461 slots) master
M: c812379709120ab74b20bd76dd2b9fcaf6c62209 10.1.200.77:7111
slots: (0 slots) master
replicates bb79ff3fdf24a99df6ebb93d17f026bb81eb221b
M: e1ced4f32594b0d2b82b268059e2ce89abebfde5 10.1.200.78:7112
slots: (0 slots) master
replicates 778b0cd6414d92ad4aec93754445648fc57963ca
M: bb79ff3fdf24a99df6ebb93d17f026bb81eb221b 10.1.200.87:7113
slots:10923-16383 (5461 slots) master
(一切正常的情况下输出以下信息:)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
最后一行信息表示集群中的 16384
个槽都有至少一个主节点在处理, 集群运作正常。 6、集群创建过程说明: 1)给定redis-trib
程序的命令是create
,这表示我们希望创建一个新的集群。 2)这里的--replicas 1
表示每个主节点下有一个从节点。 3)之后跟着的其它参数则是实例的地址列表,程序使用这些地址所指示的实例来创建新集群。 总的来说,以上命令的意思就是让redis-trib
程序创建一个包含三个主节点和三个从节点的集群。 接着,redis-trib
会打印出一份预想中的配置给你看,如果你觉得没问题的话(注意核对主从关系是否是你想要的),就可以输入 yes
,redis-trib
就会将这份配置应用到集群当中。 7、集群创建遇到问题: 1)执行Redis
集群创建命令出现如下异常:
[root@centos4 ~]# redis-trib create --replicas 1 10.1.200.118:7114 10.1.200.120:7115 10.1.200.125:7116 10.1.200.77:7111 10.1.200.78:7112 10.1.200.87:7113
>>> Creating cluster
[ERR] Sorry, can't connect to node 10.1.200.118:7114
这个错误表示无法连接到10.1.200.118:7114
这个节点,查看10.1.200.118:7114
这个节点发现服务器重启之后没有启动这个节点实例,使用如下命令重新启动节点实例:
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/7114/redis-7114.conf
重新执行Redis
集群创建命令,发现其他节点也无法连接,依次重启其他节点。 2)执行Redis
集群创建命令出现如下异常:
[root@centos4 ~]# redis-trib create --replicas 1 10.1.200.118:7114 10.1.200.120:7115 10.1.200.125:7116 10.1.200.77:7111 10.1.200.78:7112 10.1.200.87:7113
>>> Creating cluster
[ERR] Node 10.1.200.78:7112 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
这个错误表示10.1.200.78:7112
这个节点不为空,
##集群简单测试 1、在10.1.200.118:7114节点操作:
[root@centos4 redis-3.0.7]# redis-cli -c -p 7114
127.0.0.1:7114> set dj daijiong
-> Redirected to slot [2562] located at 10.1.200.125:7116
OK
10.1.200.125:7116> get dj
"daijiong"
10.1.200.125:7116>
2、在10.1.200.77:7111节点操作:
[root@centos4 bin]# redis-cli -c -p 7111
127.0.0.1:7111> get dj
-> Redirected to slot [2562] located at 10.1.200.125:7116
"daijiong"
10.1.200.125:7116>
3、查看各个节点信息:
[root@centos4 bin]# redis-cli -p 7111 cluster nodes
967f4154bc4ced3c0c76a74c280f2779f2f70fb2 10.1.200.118:7114 slave 7226e609e47616e0d3cff2f80831ac6d811e0027 0 1469521364098 3 connected
778b0cd6414d92ad4aec93754445648fc57963ca 10.1.200.120:7115 master - 0 1469521362053 2 connected 5461-10922
bb79ff3fdf24a99df6ebb93d17f026bb81eb221b 10.1.200.87:7113 master - 0 1469521361028 6 connected 10923-16383
e1ced4f32594b0d2b82b268059e2ce89abebfde5 10.1.200.78:7112 slave 778b0cd6414d92ad4aec93754445648fc57963ca 0 1469521360006 5 connected
7226e609e47616e0d3cff2f80831ac6d811e0027 10.1.200.125:7116 master - 0 1469521363073 3 connected 0-5460
c812379709120ab74b20bd76dd2b9fcaf6c62209 10.1.200.77:7111 myself,slave bb79ff3fdf24a99df6ebb93d17f026bb81eb221b 0 0 4 connected