周氏一族，整理技术文档，给下一代留点教程......
Redis 3.0 cluster 基础篇，请看下面链接
http://zhoushouby.blog.51cto.com/9150272/1560346
本篇，主要是讲述，如果安装 Redis 3.0 cluster，安装步骤跟redis主备大同小异，只是需要注意两个方面
1、开启 cluster enable ，以及设置相关cluster timeout
2、Redis cluster 是用自带的ruby工具(redis-trib.rb)构建集群[在src目录里面]
准备资料：
redis-3.0.0-beta8.tar.gz 官网下载地址 http://redis.io/download/
tcl8.6.1-src.tar.gz 官网下载地址 http://sourceforge.jp/projects/sfnet_tcl/releases/
rubygems-2.4.2.zip 官网下载地址 http://rubygems.org/pages/download/
redis-3.0.0.gem 官网下载地址 http://rubygems.org/gems/redis/versions/3.1.0
系统采用的是 centos 6.3 64bit
[root@localhost src]# file /sbin/init
/sbin/init: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
实验环境模拟：
192.168.1.222 三个实例
192.168.1.223 三个实例
注意，由于官网，必须至少6台服务器，所以鄙人笔记本过卡，直接用两台vm来虚拟，我用不同端口号来区分不同的redis实例即可。因为启动redis时只需要指定“该redis实例的配置文件”即可。在现实当中，如果有六台物理机，那样会更加简单部署，不会像我这么复杂，不过也不复杂，就是多copy一下配置文件而已。
实验开始：
1、安装一下开发工具
| yum -y install gcc make |
2、安装redis依赖包tcl，这个不懂的请看 redis 主备 那篇文章
http://zhoushouby.blog.51cto.com/9150272/1547084
tar xzvf tcl8.6.1-src.tar.gz cd tcl8.6.1/ cd unix && ./configure --prefix=/usr \ --without-tzdata \ --mandir=/usr/share/man \ $([ $(uname -m) = x86_64 ] && echo --enable-64bit) && make && sed -e "s@^\(TCL_SRC_DIR='\).*@\1/usr/include'@" \ -e "/TCL_B/s@='\(-L\)\?.*unix@='\1/usr/lib@" \ -i tclConfig.sh # 测试make是否成功，最好不要，太烦了，要等N久，半个小时有多 make test # root用户登录，执行下面命令 make install && make install-private-headers && ln -v -sf tclsh8.6 /usr/bin/tclsh && chmod -v 755 /usr/lib/libtcl8.6.so |
不要以为3.0就不用这个tcl了，他扮演着很重要的角色，如果没有tcl，将会报错如下:
[root@localhost redis-3.0.0-beta8]# make test cd src && make test make[1]: Entering directory `/root/redis-3.0.0-beta8/src' You need tcl 8.5 or newer in order to run the Redis test make[1]: *** [test] Error 1 make[1]: Leaving directory `/root/redis-3.0.0-beta8/src' make: *** [test] Error 2 [root@localhost redis-3.0.0-beta8]# |
3、需要先配置下面的内核参数，否则Redis脚本在重启或停止redis时，将会报错，并且不能自动在停止服务前同步数据到磁盘上
echo 1 > /proc/sys/vm/overcommit_memory echo vm.overcommit_memory=1 >> /etc/sysctl.conf sysctl vm.overcommit_memory=1 |
4、开始安装redis 3.0 cluster
cd /root/ tar xzvf redis-3.0.0-beta8.tar.gz cd redis-3.0.0-beta8 make make test make install vi /etc/redis.conf |
5、编辑配置文件，修改下面几个参数
port 6379 appendonly yes appendfilename "appendonly-6379.aof" cluster-enabled yes cluster-config-file /opt/nodes-6379.conf cluster-node-timeout 5000 |
为什么呢？
因为，我们要在一台vm里面，虚拟三个实例
port 端口不一样，这个是肯定要的吧，不然服务都在同一个端口运行，就只能说是一个实例而已
appendonly 这个是说在后台运行，不要我们一关闭ssh，redis就挂了
appendfilename 这个默认就是appendonly.aof，如果我们不在后面加一个后缀区分，那么所有服务，他们的快照都会存放到在同一个文件 appendonly.aof，那边就会数据紊乱，A，B,C,三者数据都一样，所以，我们要区分出来，让他们一个实例，一个文件。这里还有一个主意的地方，那就是 它，不支持路径指定，你不能在 appendonly.aof 加一个 例如/opt/appendonly.aof，否则会报错，启动不了，查看日志，报错如下
[root@localhost opt]# cat redis-6379.log *** FATAL CONFIG FILE ERROR *** Reading the configuration file, at line 177 >>> 'dbfilename /opt/dump-6379.rdb' dbfilename can't be a path, just a filename |
cluster-enabled 这个就不用多说了
cluster-config-file 这个是配置集群的时候，会自动生成的，同样道理，不能默认
cluster-node-timeout 失效时间，设置低一些，因为我是同一台物理机的vm
6、拷贝三个实例配置文件到opt目录下
cp redis.conf /opt/redis-6379.conf 实例A的配置文件 cp redis.conf /opt/redis-6380.conf 实例B的配置文件 cp redis.conf /opt/redis-6381.conf 实例C的配置文件 |
7、编辑三个实例配置文件，把端口，对应改成非重叠的，为了更好的区分，我就按照
192.168.1.222 “实例A 6379” “实例B 6380” “实例C 6381”
192.168.1.223 “实例D 6382” “实例E 6383” “实例F 6384”
port appendfilename cluster-config-file 注意根据不同实例，修改成不同的值
例如，实例C的配置文件主要是
port 6381 appendfilename "appendonly-6381.aof" cluster-config-file /opt/nodes-6381.conf |
再例如，实例E的配置文件主要变化是
port 6383 appendfilename "appendonly-6383.aof" cluster-config-file /opt/nodes-6383.conf |
废话就不多说了，越说越糊涂，自己慢慢琢磨一下
8、启动redis服务
redis-server /opt/redis-6379.conf > /opt/redis-6379.log 2>&1 & redis-server /opt/redis-6380.conf > /opt/redis-6380.log 2>&1 & redis-server /opt/redis-6381.conf > /opt/redis-6381.log 2>&1 & |
注意，为了方便查看错误信息，我把它日志，重定向到 对应的实例名字，存放在opt下面
9、一切如果正常的话，你随便找一个 redis-6379.log redis-6380.log redis-6379.log
例如，我找一个 6380的日志
[root@localhost opt]# cat redis-6380.log 6115:M 31 Aug 12:34:28.904 * Increased maximum number of open files to 10032 (it was originally set to 1024). 6115:M 31 Aug 12:34:28.906 * No cluster configuration found, I'm 92999f9840418a848f7b10c5bca0119e3b515fa4 _._ _.-``__ ''-._ _.-`` `. `_. ''-._ Redis 2.9.57 (00000000/0) 64 bit .-`` .-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in cluster mode |`-._`-...-` __...-.``-._|'` _.-'| Port: 6380 | `-._ `._ / _.-' | PID: 6115 `-._ `-._ `-./ _.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | http://redis.io `-._ `-._`-.__.-'_.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | `-._ `-._`-.__.-'_.-' _.-' `-._ `-.__.-' _.-' `-._ _.-' `-.__.-' 6115:M 31 Aug 12:34:28.924 # Server started, Redis version 2.9.57 6115:M 31 Aug 12:34:28.924 * The server is now ready to accept connections on port 6380 |
10、说明一切准备就绪，都能正常运作，就差集群了，为了更加放心，看一下是否三个实例都在正常运行
[root@localhost opt]# ps aux |grep redis root 6109 0.6 0.9 137408 9740 pts/1 Sl 12:33 0:52 redis-server *:6379 [cluster] root 6115 0.6 0.9 137408 9912 pts/2 Sl 12:34 0:51 redis-server *:6380 [cluster] root 6123 0.6 0.7 137408 7876 pts/2 Sl 12:35 0:50 redis-server *:6381 [cluster] root 6235 0.0 0.0 103240 840 pts/2 S+ 14:47 0:00 grep redis [root@localhost opt]# |
11、以上，所有步骤，在192.168.1.223，同样的做法，注意变化的就是 redis 端口的改变，以及对于配置文件的变化。
12、那么，开始集群吧，集群之前，先表示怀疑一下[请注意，只需要在其中一台服务器操作下面内容]
[root@localhost ~]# cd redis-3.0.0-beta8 [root@localhost redis-3.0.0-beta8]# cd src/ [root@localhost src]# ./redis-trib.rb create --replicas 1 192.168.1.222:6379 192.168.1.222:6780 192.168.1.222:6381 192.168.1.223:6382 192.168.1.223:6383 192.168.1.223:6384 /usr/bin/env: ruby: No such file or directory [root@localhost src]# |
你会发现，他马上报错，报错信息如下：/usr/bin/env: ruby: No such file or directory
就是说，没有ruby，
因为
redis-trib位于Redis源码的src文件夹中， 它是一个Ruby程序， 这个程序通过向实例发送特殊命令来完成创建新集 群， 检查集群， 或者对集群进行重新分片（reshared）等工作。
13、既然没有ruby，那我们就yum安装一个吧
yum -y install ruby ruby-rdoc |
14、好，装好了，继续来集群
[root@localhost src]# ./redis-trib.rb create --replicas 1 192.168.1.222:6379 192.168.1.222:6780 192.168.1.222:6381 192.168.1.223:6382 192.168.1.223:6383 192.168.1.223:6384 ./redis-trib.rb:24:in `require': no such file to load -- rubygems (LoadError) from ./redis-trib.rb:24 [root@localhost src]# |
你会发现，又有问题了，这是什么问题啊，不过看了一下，后面提示rubygems (LoadError) ，嘿嘿，应该就是gem没有上去，那么装一个咯
15、gem下载地址，上面有 http://rubygems.org/pages/download/
官网上面有很多种哦，有TGZ ZIP GEM GIT, 我这里下载的是zip的包
16、安装rubygems
unzip rubygems-2.4.2.zip cd rubygems-2.4.2 ruby setup.rb |
刷刷刷一大堆东西出来，不过，细心的朋友，你会发现，下面有一处非常重要的 代码
[root@localhost rubygems-2.4.2]# ruby setup.rb RubyGems 2.4.2 installed Installing ri documentation for rubygems-2.4.2 /usr/lib/ruby/1.8/rdoc/rdoc.rb:280: warning: conflicting chdir during another chdir block /usr/lib/ruby/1.8/rdoc/rdoc.rb:287: warning: conflicting chdir during another chdir block === 2.4.2 / 2014-10-01 This release was sponsored by Ruby Central. Bug fixes: * RubyGems now correctly matches wildcard no_proxy hosts. Issue #997 by voelzemo. * Added support for missing git_source method in the gem dependencies API. * Fixed handling of git gems with an alternate install directory. * Lockfiles will no longer be truncated upon resolution errors. * Fixed messaging for `gem owner -a`. Issue #1004 by Aaron Patterson, Ryan Davis. * Removed meaningless ensure. Pull request #1003 by gogotanaka. * Improved wording of --source option help. Pull request #989 by Jason Clark. * Empty build_info files are now ignored. Issue #903 by Adan Alvarado. * Gem::Installer ignores dependency checks when installing development dependencies. Issue #994 by Jens Willie. * `gem update` now continues after dependency errors. Issue #993 by aaronchi. * RubyGems no longer warns about semantic version dependencies for the 0.x range. Issue #987 by Jeff Felchner, pull request #1006 by Hsing-Hui Hsu. * Added minimal lock to allow multithread installation of gems. Issue #982 and pull request #1005 by Yorick Peterse * RubyGems now considers prerelease dependencies as it did in earlier versions when --prerelease is given. Issue #990 by Jeremy Tryba. * Updated capitalization in README. Issue #1010 by Ben Bodenmiller. * Fixed activating gems from a Gemfile for default gems. Issue #991 by khoan. * Fixed windows stub script generation for Cygwin. Issue #1000 by Brett DiFrischia. * Allow gem bindir and ruby.exe to live in separate diretories. Pull request #942 by Ian Flynn. * Fixed handling of gemspec in gem dependencies files to match Bundler behavior. Issue #1020 by Michal Papis. * Fixed `gem update` when updating to prereleases. Issue #1028 by Santiago Pastorino. * RubyGems now fails immediately when a git reference cannot be found instead of spewing git errors. Issue #1031 by Michal Papis === 2.4.1 / 2014-07-17 Bug fixes: * RubyGems can now be updated on Ruby implementations that do not support vendordir in RbConfig::CONFIG. Issue #974 by net1957. === 2.4.0 / 2014-07-16 Minor enhancements: * The contents command now supports a --show-install-dir option that shows only the directory the gem is installed in. Feature request #966 by Akinori MUSHA. * Added a --build-root option to the install command for packagers. Pull request #965 by Marcus R眉ckert. * Added vendor gem support to RubyGems. Package managers may now install gems in Gem.vendor_dir with the --vendor option to gem install. Issue #943 by Marcus R眉ckert. Bug fixes: * Kernel#gem now respects the prerelease flag when activating gems. Previously this behavior was undefined which could lead to bugs when a prerelease version was unintentionally activated. Bug #938 by Joe Ferris. * RubyGems now prefers gems from git over installed gems. This allows gems from git to override an installed gem with the same name and version. Bug #944 by Thomas Kriechbaumer. * Fixed handling of git gems in a lockfile with unversioned dependencies. Bug #940 by Michael Kaiser-Nyman. * The ruby directive in a gem dependencies file is ignored when installing. Bug #941 by Michael Kaiser-Nyman. * Added open to list of builtin commands (`gem open` now works). Reported by Espen Antonsen. * `gem open` now works with command-line editors. Pull request #962 by Tim Pope. * `gem install -g` now respects `--conservative`. Pull request #950 by Jeremy Evans. * RubyGems releases announcements now now include checksums. Bug #939 by Alexander E. Fischer. * RubyGems now expands ~ in $PATH when checking if installed executables will be runnable. Pull request #945 by Alex Talker. * Fixed `gem install -g --explain`. Issue #947 by Luis Lavena. Patch by Hsing-Hui Hsu. * RubyGems locks less during gem activation. Pull request #951 by Aaron Patterson and Justin Searls, #969 by Jeremy Tryba. * Kernel#gem is now thread-safe. Pull request #967 by Aaron Patterson. * RubyGems now handles spaces in directory names for some parts of extension building. Pull request #949 by Tristan Hill. * RubyGems no longer defines an empty Date class. Pull Request #948 by Benoit Daloze. * RubyGems respects --document options for `gem update` again. Bug 946 by jonforums. Patch by Hsing-Hui Hsu. * RubyGems generates documentation again with --ignore-dependencies. Bug #961 by Pulfer. * RubyGems can install extensions across partitions now. Pull request #970 by Michael Scherer. * `-s` is now short for `--source` which resolves an ambiguity with --no-suggestions. Pull request #955 by Alexander Kahn. * Added extra test for ~> for 0.0.X versions. Pull request #958 by Mark Lorenz. * Fixed typo in gem updated help. Pull request #952 by Per Modin. * Clarified that the gem description should not be excessively long. Part of bug #956 by Renier Morales. * Hid documentation of outdated test_files related methods in Specification. Guides issue #90 by Emil Soman. * RubyGems now falls back to the old index if the rubygems.org API fails during gem resolution. ------------------------------------------------------------------------------ RubyGems installed the following executables: /usr/bin/gem Ruby Interactive (ri) documentation was installed. ri is kind of like man pages for ruby libraries. You may access it like this: ri Classname ri Classname.class_method ri Classname#instance_method If you do not wish to install this documentation in the future, use the --no-document flag, or set it as the default in your ~/.gemrc file. See 'gem help env' for details. |
为了方便操作rubygem，我们把它拷贝到系统环境变量当中，方便调用
| cp bin/gem /usr/local/bin/ |
17、接下来，安装 redis 的api 接口，这就是为什么上面要装rubygem的道理了
| gem install -l redis-3.0.0.gem |
redis-3.0.0.gem 上面讲到，可以到 官网 http://rubygems.org/gems/redis/versions/3.1.0 下载
18、一切就绪，现在再一次，来执行集群看看还有没有报错的信息
[root@localhost src]# ./redis-trib.rb create --replicas 1 192.168.1.222:6379 192.168.1.222:6380 192.168.1.222:6381 192.168.1.223:6382 192.168.1.223:6383 192.168.1.223:6384 >>> Creating cluster Connecting to node 192.168.1.222:6379: OK Connecting to node 192.168.1.222:6380: OK Connecting to node 192.168.1.222:6381: OK Connecting to node 192.168.1.223:6382: OK Connecting to node 192.168.1.223:6383: OK Connecting to node 192.168.1.223:6384: OK >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 192.168.1.223:6382 192.168.1.222:6379 192.168.1.223:6383 Adding replica 192.168.1.222:6380 to 192.168.1.223:6382 Adding replica 192.168.1.223:6384 to 192.168.1.222:6379 Adding replica 192.168.1.222:6381 to 192.168.1.223:6383 M: faa2fab465017c9be2475d0cbd9ffb7b8e8267e6 192.168.1.222:6379 slots:5461-10922 (5462 slots) master S: 92999f9840418a848f7b10c5bca0119e3b515fa4 192.168.1.222:6380 replicates c13fbd7fadda1e46b133b8f87248414d9a7a0fcc S: 5d44e7e3bf2c19fcdcc1548feef42d5d9665da0b 192.168.1.222:6381 replicates 7d674df0ab68be0b4c6a461b0c1484b02eeb12c5 M: c13fbd7fadda1e46b133b8f87248414d9a7a0fcc 192.168.1.223:6382 slots:0-5460 (5461 slots) master M: 7d674df0ab68be0b4c6a461b0c1484b02eeb12c5 192.168.1.223:6383 slots:10923-16383 (5461 slots) master S: c4b565e9236effee3957c6f03b533dced729e39b 192.168.1.223:6384 replicates faa2fab465017c9be2475d0cbd9ffb7b8e8267e6 Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join... >>> Performing Cluster Check (using node 192.168.1.222:6379) M: faa2fab465017c9be2475d0cbd9ffb7b8e8267e6 192.168.1.222:6379 slots:5461-10922 (5462 slots) master M: 92999f9840418a848f7b10c5bca0119e3b515fa4 192.168.1.222:6380 slots: (0 slots) master replicates c13fbd7fadda1e46b133b8f87248414d9a7a0fcc M: 5d44e7e3bf2c19fcdcc1548feef42d5d9665da0b 192.168.1.222:6381 slots: (0 slots) master replicates 7d674df0ab68be0b4c6a461b0c1484b02eeb12c5 M: c13fbd7fadda1e46b133b8f87248414d9a7a0fcc 192.168.1.223:6382 slots:0-5460 (5461 slots) master M: 7d674df0ab68be0b4c6a461b0c1484b02eeb12c5 192.168.1.223:6383 slots:10923-16383 (5461 slots) master M: c4b565e9236effee3957c6f03b533dced729e39b 192.168.1.223:6384 slots: (0 slots) master replicates faa2fab465017c9be2475d0cbd9ffb7b8e8267e6 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@localhost src]# |
19、很明显，全部正常，一切OK，搞定。
20、如果不太相信，那么你大可以来测试一下
[root@localhost src]# ./redis-trib.rb check 192.168.1.223:6384 Connecting to node 192.168.1.223:6384: OK Connecting to node 192.168.1.222:6380: OK Connecting to node 192.168.1.222:6379: OK Connecting to node 192.168.1.222:6381: OK Connecting to node 192.168.1.223:6383: OK Connecting to node 192.168.1.223:6382: OK >>> Performing Cluster Check (using node 192.168.1.223:6384) S: c4b565e9236effee3957c6f03b533dced729e39b 192.168.1.223:6384 slots: (0 slots) slave replicates faa2fab465017c9be2475d0cbd9ffb7b8e8267e6 S: 92999f9840418a848f7b10c5bca0119e3b515fa4 192.168.1.222:6380 slots: (0 slots) slave replicates c13fbd7fadda1e46b133b8f87248414d9a7a0fcc M: faa2fab465017c9be2475d0cbd9ffb7b8e8267e6 192.168.1.222:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) M: 5d44e7e3bf2c19fcdcc1548feef42d5d9665da0b 192.168.1.222:6381 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 7d674df0ab68be0b4c6a461b0c1484b02eeb12c5 192.168.1.223:6383 slots: (0 slots) slave replicates 5d44e7e3bf2c19fcdcc1548feef42d5d9665da0b M: c13fbd7fadda1e46b133b8f87248414d9a7a0fcc 192.168.1.223:6382 slots:0-5460 (5461 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@localhost src]# |
注意哦，我是直接在192.168.1.222，测试到192.168.1.223当中的6384这个实例了
从上面，我们也不难看出
redis_master 是
192.168.1.222:6379
192.168.1.222:6381
192.168.1.223:6382
redis_salve 是
192.168.1.223:6384
192.168.1.222:6380
192.168.1.223:6383
21、实验结果
任何一个实例，都能写入，查询，不存在主备关系，redis会自动通过内部算法，选举，不需要你去关心
22、做一个小测试
在 192.168.1.222 ，连接 192.168.1.223 的实例D 6382 ，写入数据 zhou
[root@localhost src]# redis-cli -h 192.168.1.223 -p 6382 -c 192.168.1.223:6382> set zhou mynameiszhou -> Redirected to slot [11570] located at 192.168.1.223:6383 OK |
Redis会自己重定向到 6383 ，也就是 E 当中，换句话说，6382和6382是主备关系，其中6383是6382的 salver 服务器
好，那我们来 192.168.1.223 里面的 6484，也就是 F当中读取 zhou的值
[root@localhost opt]# redis-cli -h 192.168.1.223 -p 6384 -c 192.168.1.223:6384> get zhou -> Redirected to slot [11570] located at 192.168.1.223:6383 "mynameiszhou" |
很明显，读取到了，它 会自动映射到 6383 去读取
现在疑问来了，我就把6383干掉，看看是什么样的情况
[root@localhost opt]# ps aux |grep redis root 28922 0.2 0.9 137408 9752 pts/2 Sl 03:30 0:06 redis-server *:6382 [cluster] root 28928 0.2 0.9 137408 9596 pts/2 Sl 03:31 0:06 redis-server *:6383 [cluster] root 28932 0.2 0.7 137408 7884 pts/2 Sl 03:32 0:05 redis-server *:6384 [cluster] root 28973 0.0 0.0 103232 876 pts/2 S+ 04:18 0:00 grep redis [root@localhost opt]# kill -9 28928 [root@localhost opt]# ps aux |grep redis root 28922 0.2 0.9 137408 9752 pts/2 Sl 03:30 0:06 redis-server *:6382 [cluster] root 28932 0.2 0.7 137408 7884 pts/2 Sl 03:32 0:05 redis-server *:6384 [cluster] root 28975 0.0 0.0 103232 876 pts/2 S+ 04:19 0:00 grep redis [2]- Killed redis-server /opt/redis-6383.conf > /opt/redis-6383.log 2>&1 |
很明显，6383已经被干掉了
[root@localhost src]# redis-cli -h 192.168.1.223 -p 6383 -c Could not connect to Redis at 192.168.1.223:6383: Connection refused not connected> get zhou Could not connect to Redis at 192.168.1.223:6383: Connection refused not connected> [root@localhost src]# redis-cli -h 192.168.1.223 -p 6384 -c 192.168.1.223:6384> get zhou -> Redirected to slot [11570] located at 192.168.1.222:6381 "mynameiszhou" 192.168.1.222:6381> |
虽然，在6383，建立不了链接，但是不影响我数据的读取，在6381一样可以获取到zhou的key值
very good，perfect!!!
转载于:https://blog.51cto.com/zhoushouby/1560400
295
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
