个人运维习惯,会专门创建一个app账号,用户部署应用程序。本案例应用程序都部署在 /data 目录下,将 /data 权限设置成app [root@bl-redis01 ~] # useradd app [root@bl-redis01 ~] # passwd app [root@bl-redis01 ~] # chown -R app.app /data 前提准备 1)安裝 GCC 编译工具 不然会有编译不过的问题 [root@bl-redis01 ~] # yum install -y gcc g++ make gcc-c++ kernel-devel automake autoconf libtool make wget tcl vim ruby rubygems unzip git 2)升级所有的包,防止出现版本过久不兼容问题 [root@bl-redis01 ~] # yum -y update 3)关闭防火墙 节点之前需要开放指定端口,为了方便,生产不要禁用 [root@bl-redis01 ~] # /etc/init.d/iptables stop [root@bl-redis01 ~] # setenforce 0 [root@bl-redis01 ~] # vim /etc/sysconfig/selinux ...... SELINUX=disabled ...... redis cluster集群部署 4)下载并编译安装redis [root@bl-redis01 ~] # su - app [app@bl-redis01 ~]$ mkdir /data/software/ [app@bl-redis01 software]$ wget http: //download .redis.io /releases/redis-4 .0.1. tar .gz [app@bl-redis01 software]$ tar -zvxf redis-4.0.1. tar .gz [app@bl-redis01 software]$ mv redis-4.0.1 /data/ [app@bl-redis01 software]$ cd /data/redis-4 .0.1/ [app@bl-redis01 redis-4.0.1]$ make -------------------------------------------------------------------------------------- 如果因为上次编译失败,有残留的文件,做法如下: [app@bl-redis01 redis-4.0.1]$ make distclean -------------------------------------------------------------------------------------- 5)创建节点 首先在172.16.51.175机器(redis01)上 /data/redis-4 .0.1目录下创建redis-cluster目录 [app@bl-redis01 redis-4.0.1]$ mkdir /data/redis-4 .0.1 /redis-cluster 接着在redis-cluster目录下,创建名为7000、7001、7002的目录 [app@bl-redis01 redis-cluster]$ mkdir 7000 [app@bl-redis01 redis-cluster]$ mkdir 7001 [app@bl-redis01 redis-cluster]$ mkdir 7002 分别修改这三个配置文件redis.conf [app@bl-redis01 redis-4.0.1]$ cd redis-cluster/ [app@bl-redis01 redis-cluster]$ ll total 12 drwxrwxr-x 2 app app 4096 Nov 16 17:38 7000 drwxrwxr-x 2 app app 4096 Nov 16 17:39 7001 drwxrwxr-x 2 app app 4096 Nov 16 17:39 7002 [app@bl-redis01 redis-cluster]$ cat 7000 /redis .conf port 7000 bind 172.16.51.175 daemonize yes pidfile /var/run/redis_7000 .pid cluster-enabled yes cluster-config- file nodes_7000.conf cluster-node-timeout 10100 appendonly yes [app@bl-redis01 redis-cluster]$ cat 7001 /redis .conf port 7001 bind 172.16.51.175 daemonize yes pidfile /var/run/redis_7001 .pid cluster-enabled yes cluster-config- file nodes_7001.conf cluster-node-timeout 10100 appendonly yes [app@bl-redis01 redis-cluster]$ cat 7002 /redis .conf port 7002 bind 172.16.51.175 daemonize yes pidfile /var/run/redis_7002 .pid cluster-enabled yes cluster-config- file nodes_7002.conf cluster-node-timeout 10100 appendonly yes ---------------------------------------------------------------------------------------------------- redis.conf的配置说明: #端口7000,7001,7002 port 7000 #默认ip为127.0.0.1,需要改为其他节点机器可访问的ip,否则创建集群时无法访问对应的端口,无法创建集群 bind 172.16.51.175 #redis后台运行 daemonize yes #pidfile文件对应7000,7001,7002 pidfile /var/run/redis_7000 .pid #开启集群,把注释#去掉 cluster-enabled yes #集群的配置,配置文件首次启动自动生成 7000,7001,7002 cluster-config- file nodes_7000.conf #请求超时,默认15秒,可自行设置 cluster-node-timeout 10100 #aof日志开启,有需要就开启,它会每次写操作都记录一条日志 appendonly yes ---------------------------------------------------------------------------------------------------- 接着在另外两台机器上(172.16.51.176,172.16.51.178)重复以上三步,只是把目录改为7003、7004、7005和7006、7007、7008,对应的配置文件也按照这个规则修改即可(即修改redis.conf文件中的端口就行了) 6)启动集群(依次启动7000-7008端口) #第一个节点机器上执行 3个节点 [app@bl-redis01 redis-cluster]$ for ((i=0;i<=2;i++)); do /data/redis-4 .0.1 /src/redis-server /data/redis-4 .0.1 /redis-cluster/700 $i /redis .conf; done #第二个节点机器上执行 3个节点 [app@bl-redis01 redis-cluster]$ for ((i=3;i<=5;i++)); do /data/redis-4 .0.1 /src/redis-server /data/redis-4 .0.1 /redis-cluster/700 $i /redis .conf; done #第三个节点机器上执行 3个节点 [app@bl-redis01 redis-cluster]$ for ((i=6;i<=8;i++)); do /data/redis-4 .0.1 /src/redis-server /data/redis-4 .0.1 /redis-cluster/700 $i /redis .conf; done 7)检查服务 检查各 Redis 各个节点启动情况 [app@bl-redis01 redis-cluster]$ ps -ef | grep redis app 2564 2405 0 20:13 pts /0 00:00:00 grep redis app 15197 1 0 17:57 ? 00:00:05 /data/redis-4 .0.1 /src/redis-server 172.16.51.175:7000 [cluster] app 15199 1 0 17:57 ? 00:00:05 /data/redis-4 .0.1 /src/redis-server 172.16.51.175:7001 [cluster] app 15201 1 0 17:57 ? 00:00:05 /data/redis-4 .0.1 /src/redis-server 172.16.51.175:7002 [cluster] [app@bl-redis01 redis-cluster]$ ps -ef | grep redis app 2566 2405 0 20:13 pts /0 00:00:00 grep redis app 15197 1 0 17:57 ? 00:00:05 /data/redis-4 .0.1 /src/redis-server 172.16.51.175:7000 [cluster] app 15199 1 0 17:57 ? 00:00:05 /data/redis-4 .0.1 /src/redis-server 172.16.51.175:7001 [cluster] app 15201 1 0 17:57 ? 00:00:05 /data/redis-4 .0.1 /src/redis-server 172.16.51.175:7002 [cluster] 8)安装 Ruby(需要切换到root账号下进行安装,app账号下权限不够) [root@bl-redis01 ~] # yum -y install ruby ruby-devel rubygems rpm-build [root@bl-redis01 ~] # gem install redis ----------------------------------------------------------------------------------------------------- 注意:在centos6.x下执行上面的 "gem install redis" 操作可能会报错,坑很多! 默认yum安装的ruby版本是1.8.7,版本太低,需要升级到ruby2.2以上,否则执行上面安装会报错! 首先安装rvm(或者直接下载证书:https: //pan .baidu.com /s/1slTyJ7n 密钥:7uan 下载并解压后直接执行 "curl -L get.rvm.io | bash -s stable" 即可) [root@bl-redis01 ~] # curl -L get.rvm.io | bash -s stable //可能会报错,需要安装提示进行下面一步操作 [root@bl-redis01 ~] # curl -sSL https://rvm.io/mpapis.asc | gpg2 --import - //然后再接着执行:curl -L get.rvm.io | bash -s stable [root@bl-redis01 ~] # find / -name rvm.sh /etc/profile .d /rvm .sh [root@bl-redis01 ~] # source /etc/profile.d/rvm.sh [root@bl-redis01 ~] # rvm requirements 然后升级ruby到2.3 [root@bl-redis01 ~] # rvm install ruby 2.3.1 [root@bl-redis01 ~] # ruby -v ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux] 列出所有ruby版本 [root@bl-redis01 ~] # rvm list 设置默认的版本 [root@bl-redis01 ~] # rvm --default use 2.3.1 更新下载源 [root@bl-redis01 ~] # gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org https: //gems .ruby-china.org/ added to sources source https: //rubygems .org not present in cache [root@bl-redis01 ~] # gem sources *** CURRENT SOURCES *** https: //rubygems .org/ https: //gems .ruby-china.org/ 最后就能顺利安装了 [root@bl-redis01 src] # gem install redis Successfully installed redis-4.0.1 Parsing documentation for redis-4.0.1 Done installing documentation for redis after 1 seconds 1 gem installed ----------------------------------------------------------------------------------------------------- 9)创建集群 千万注意:在任意一台上运行即可,不要在每台机器上都运行,一台就够了!!!! Redis 官方提供了 redis-trib.rb 这个工具,就在解压目录的 src 目录中 [root@bl-redis01 ~] # su - app [app@bl-redis01 ~]$ /data/redis-4 .0.1 /src/redis-trib .rb create --replicas 1 172.16.51.175:7000 172.16.51.175:7001 172.16.51.175:7002 172.16.51.176:7003 172.16.51.176:7004 172.16.51.176:7005 172.16.51.178:7006 172.16.51.178:7007 172.16.51.178:7008 出现下面信息,从下面信息可以看出,本案例三台服务器启动9个实例,配置成4主5从,其中有一个是一主两从,其他3个都是一主一从。 >>> Creating cluster >>> Performing hash slots allocation on 9 nodes... Using 4 masters: 172.16.51.175:7000 172.16.51.176:7003 172.16.51.178:7006 172.16.51.175:7001 Adding replica 172.16.51.176:7004 to 172.16.51.175:7000 Adding replica 172.16.51.178:7007 to 172.16.51.176:7003 Adding replica 172.16.51.175:7002 to 172.16.51.178:7006 Adding replica 172.16.51.176:7005 to 172.16.51.175:7001 Adding replica 172.16.51.178:7008 to 172.16.51.175:7000 M: 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf 172.16.51.175:7000 slots:0-4095 (4096 slots) master M: 44c81c15b01d992cb9ede4ad35477ec853d70723 172.16.51.175:7001 slots:12288-16383 (4096 slots) master S: 38f03c27af39723e1828eb62d1775c4b6e2c3638 172.16.51.175:7002 replicates f1abb62a8c9b448ea14db421bdfe3f1d8075189c M: 987965baf505a9aa43e50e46c76189c51a8f17ec 172.16.51.176:7003 slots:4096-8191 (4096 slots) master S: 6555292fed9c5d52fcf5b983c441aff6f96923d5 172.16.51.176:7004 replicates 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf S: 2b5ba254a0405d4efde4c459867b15176f79244a 172.16.51.176:7005 replicates 44c81c15b01d992cb9ede4ad35477ec853d70723 M: f1abb62a8c9b448ea14db421bdfe3f1d8075189c 172.16.51.178:7006 slots:8192-12287 (4096 slots) master S: eb4067373d36d8a8df07951f92794e67a6aac022 172.16.51.178:7007 replicates 987965baf505a9aa43e50e46c76189c51a8f17ec S: 2919e041dd3d1daf176d6800dcd262f4e727f366 172.16.51.178:7008 replicates 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf Can I set the above configuration? ( type 'yes' to accept): yes 输入 yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join ......... >>> Performing Cluster Check (using node 172.16.51.175:7000) M: 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf 172.16.51.175:7000 slots:0-4095 (4096 slots) master 2 additional replica(s) S: 6555292fed9c5d52fcf5b983c441aff6f96923d5 172.16.51.176:7004 slots: (0 slots) slave replicates 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf M: 44c81c15b01d992cb9ede4ad35477ec853d70723 172.16.51.175:7001 slots:12288-16383 (4096 slots) master 1 additional replica(s) S: 2919e041dd3d1daf176d6800dcd262f4e727f366 172.16.51.178:7008 slots: (0 slots) slave replicates 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf M: f1abb62a8c9b448ea14db421bdfe3f1d8075189c 172.16.51.178:7006 slots:8192-12287 (4096 slots) master 1 additional replica(s) S: eb4067373d36d8a8df07951f92794e67a6aac022 172.16.51.178:7007 slots: (0 slots) slave replicates 987965baf505a9aa43e50e46c76189c51a8f17ec S: 38f03c27af39723e1828eb62d1775c4b6e2c3638 172.16.51.175:7002 slots: (0 slots) slave replicates f1abb62a8c9b448ea14db421bdfe3f1d8075189c S: 2b5ba254a0405d4efde4c459867b15176f79244a 172.16.51.176:7005 slots: (0 slots) slave replicates 44c81c15b01d992cb9ede4ad35477ec853d70723 M: 987965baf505a9aa43e50e46c76189c51a8f17ec 172.16.51.176:7003 slots:4096-8191 (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. 10)关闭集群 推荐做法: [app@bl-redis01 ~]$ pkill redis [app@bl-redis02 ~]$ pkill redis [app@bl-redis03 ~]$ pkill redis 或者循环节点逐个关闭 [app@bl-redis01 ~]$ for ((i=0;i<=2;i++)); do /opt/redis-4 .0.1 /src/redis-cli -c -h 172.16.51.175 -p 700$i shutdown ; done [app@bl-redis02 ~]$ for ((i=3;i<=5;i++)); do /opt/redis-4 .0.1 /src/redis-cli -c -h 172.16.51.176 -p 700$i shutdown ; done [app@bl-redis03 ~]$ for ((i=6;i<=8;i++)); do /opt/redis-4 .0.1 /src/redis-cli -c -h 172.16.51.178 -p 700$i shutdown ; done 11)集群验证 连接集群测试 参数-C可连接到集群,因为redis.conf将bind改为了ip地址,所以-h参数不可以省略,-p参数为端口号 可以先在172.16.51.175机器redis 7000 的节点 set 一个key [app@bl-redis01 ~]$ /data/redis-4 .0.1 /src/redis-cli -h 172.16.51.175 -c -p 7000 172.16.51.175:7000> set name www.ymq.io -> Redirected to slot [5798] located at 172.16.51.176:7003 OK 172.16.51.176:7003> get name "www.ymq.io" 172.16.51.176:7003> 由上面信息可发现redis set name 之后重定向到172.16.51.176机器 redis 7003 这个节点 然后在172.16.51.178机器redis 7008 的节点get一个key [app@bl-redis03 ~]$ /data/redis-4 .0.1 /src/redis-cli -h 172.16.51.178 -c -p 7008 172.16.51.178:7008> get name -> Redirected to slot [5798] located at 172.16.51.176:7003 "www.ymq.io" 172.16.51.176:7003> 发现redis get name 重定向到172.16.51.176机器 redis 7003 这个节点. 如果看到这样的现象,说明redis cluster集群已经是可用的了!!!!!! 12)检查集群状态(通过下面的命令,可以看到本案例实现的是4主5从,4个主节点会默认分配到三个机器上,每个机器上都要有master;另:创建集群的时候可以指定master和slave。这里我是默认创建的) [app@bl-redis01 ~]$ /data/redis-4 .0.1 /src/redis-cli -h 172.16.51.175 -c -p 7000 172.16.51.175:7000> [app@bl-redis01 ~]$ /data/redis-4 .0.1 /src/redis-trib .rb check 172.16.51.175:7000 >>> Performing Cluster Check (using node 172.16.51.175:7000) M: 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 172.16.51.175:7000 slots:0-4095 (4096 slots) master 2 additional replica(s) M: c64b0839e0199f73c5c192cc8c90f12c999f79b2 172.16.51.175:7001 slots:12288-16383 (4096 slots) master 1 additional replica(s) S: 81347f01cf38d8f0faef1ad02676ebb4cffbec9e 172.16.51.176:7005 slots: (0 slots) slave replicates c64b0839e0199f73c5c192cc8c90f12c999f79b2 M: da5dde3f2f02c232784bf3163f5f584b8cf046f2 172.16.51.178:7006 slots:8192-12287 (4096 slots) master 1 additional replica(s) M: b217ab2a6c05497af3b2a859c1bb6b3fae5e0d92 172.16.51.176:7003 slots:4096-8191 (4096 slots) master 1 additional replica(s) S: 0420c49fbc9f1fe16066d189265cca2f5e71c86e 172.16.51.178:7007 slots: (0 slots) slave replicates b217ab2a6c05497af3b2a859c1bb6b3fae5e0d92 S: 5ad89453fb36e50ecc4560de6b4acce1dbbb78b3 172.16.51.176:7004 slots: (0 slots) slave replicates 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 S: bbd1f279b99b95cf00ecbfab22b6b8dd5eb05989 172.16.51.178:7008 slots: (0 slots) slave replicates 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 S: e95407b83bfeb30e3cc537161eadc372d6aa1fa2 172.16.51.175:7002 slots: (0 slots) slave replicates da5dde3f2f02c232784bf3163f5f584b8cf046f2 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. 13)列出集群节点 列出集群当前已知的所有节点(node),以及这些节点的相关信息 [app@bl-redis01 ~]$ /data/redis-4 .0.1 /src/redis-cli -h 172.16.51.175 -c -p 7000 172.16.51.175:7000> cluster nodes 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 172.16.51.175:7000@17000 myself,master - 0 1510836027000 1 connected 0-4095 c64b0839e0199f73c5c192cc8c90f12c999f79b2 172.16.51.175:7001@17001 master - 0 1510836030068 2 connected 12288-16383 81347f01cf38d8f0faef1ad02676ebb4cffbec9e 172.16.51.176:7005@17005 slave c64b0839e0199f73c5c192cc8c90f12c999f79b2 0 1510836031000 6 connected da5dde3f2f02c232784bf3163f5f584b8cf046f2 172.16.51.178:7006@17006 master - 0 1510836031000 7 connected 8192-12287 b217ab2a6c05497af3b2a859c1bb6b3fae5e0d92 172.16.51.176:7003@17003 master - 0 1510836030000 4 connected 4096-8191 0420c49fbc9f1fe16066d189265cca2f5e71c86e 172.16.51.178:7007@17007 slave b217ab2a6c05497af3b2a859c1bb6b3fae5e0d92 0 1510836029067 8 connected 5ad89453fb36e50ecc4560de6b4acce1dbbb78b3 172.16.51.176:7004@17004 slave 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 0 1510836032672 5 connected bbd1f279b99b95cf00ecbfab22b6b8dd5eb05989 172.16.51.178:7008@17008 slave 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 0 1510836031000 9 connected e95407b83bfeb30e3cc537161eadc372d6aa1fa2 172.16.51.175:7002@17002 slave da5dde3f2f02c232784bf3163f5f584b8cf046f2 0 1510836031672 7 connected 14)打印集群信息 [app@bl-redis01 ~]$ /data/redis-4 .0.1 /src/redis-cli -h 172.16.51.175 -c -p 7000 172.16.51.175:7000> cluster info cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:9 cluster_size:4 cluster_current_epoch:9 cluster_my_epoch:1 cluster_stats_messages_ping_sent:8627 cluster_stats_messages_pong_sent:8581 cluster_stats_messages_sent:17208 cluster_stats_messages_ping_received:8573 cluster_stats_messages_pong_received:8626 cluster_stats_messages_meet_received:8 cluster_stats_messages_received:17207 ------------------------------------------------------------------------------------------------ [root@bl-redis01 src] # pwd /data/redis-4 .0.1 /src [root@bl-redis01 src] # ./redis-trib.rb help Usage: redis-trib < command > <options> <arguments ...> create host1:port1 ... hostN:portN --replicas <arg> check host:port info host:port fix host:port --timeout <arg> reshard host:port --from <arg> --to <arg> --slots <arg> -- yes --timeout <arg> --pipeline <arg> rebalance host:port --weight <arg> --auto-weights --use-empty-masters --timeout <arg> --simulate --pipeline <arg> --threshold <arg> add-node new_host:new_port existing_host:existing_port --slave --master- id <arg> del-node host:port node_id set -timeout host:port milliseconds call host:port command arg arg .. arg import host:port --from <arg> --copy --replace help (show this help) For check, fix, reshard, del-node, set -timeout you can specify the host and port of any working node in the cluster. 上面已经多次出现了slot这个词,略为解释一下: redis-cluster把整个集群的存储空间划分为16384个slot(译为:插槽?),当9个实例分为3主6从时,相当于整个cluster中有3组HA的节点, 3个master会平均分摊所有slot,每次向cluster中的key做操作时(比如:读取/写入缓存),redis会对key值做CRC32算法处理,得到一个数值, 然后再对16384取模,通过余数判断该缓存项应该落在哪个slot上,确定了slot,也就确定了保存在哪个master节点上,当cluster扩容或删除 节点时,只需要将slot重新分配即可(即:把部分slot从一些节点移动到其它节点)。 --------------------------------------------------------------------------------------------------------- 在代码里连接以上redis cluster集群节点配置如下: spring.redis.cluster.nodes = 172.16.51.175:7000,172.16.51.175:7001,172.16.51.175:7002,172.16.51.176:7003,172.16.51.176:7004,172.16.51.176:7005,172.16.51.178:7006,172.16.51.178:7007,172.16.51.178:7008 |