codis集群搭建

一、搭建过程

官方文档：https://github.com/CodisLabs/codis/blob/release3.2/doc/tutorial_zh.md

（一）、软件及配置位置

1、上传bin.tar至/opt/app/codis下（tar -xvf bin.tar），将配置放在/opt/app/codis/ripsg/config下

2、优点：

a、支持动态水平扩展，对client完全透明，不影响服务的情况下可完redis-group的动态增减
b、基于proxy设计，可以增加proxy来提高client的访问效率（通过监控zk上的注册信息来实时获取当前可用的proxy列表，既可以保证高可用，也可以通过轮流请求所有的proxy实现负载均衡）。而redis-cluster想要同时执行多个请求来提速必须在client端自行实现异步逻辑
c、端口信息  
zk：2181客户端访问、2888主从通信（从连主：serverSocke.acceptor线程、每个连接一个learnHandler线程，有半数以上follower通知getEpochToPropose的notifyAll）、3888选主(默认大的连小的)、zab原子消息广播（恢复模式：选主、同步 广播模式：广播）
redis-group：6480
redis-proxy：19000（客户端访问的）、11080（codis-dashboard保证状态同步）
sentinel：24680
dashboard：18080
fe：9090
注：dashboard、proxy非安全退出时（kill -9），zk上的节点不会被删除，启动时会报错：节点已经存在
 

（二）、启动过程

注：启动dashboard、proxy、fe（先获取codis.json）、server、sentinel

1、启动dashboard
cd /opt/app/codis
nohup ./bin/codis-dashboard --ncpu=4 --config=ripsg/config/dashboard_18080.toml --log=dashboard.log --log-level=WARN &
2、启动proxy
cd /opt/app/codis
nohup ./bin/codis-proxy --ncpu=4 --config=ripsg/config/proxy_11080.toml --log=proxy.log --log-level=WARN &
3、启动fe
a、用admin命令获取codis.json
cd /opt/app/codis/bin
./codis-admin --dashboard-list --zookeeper=zk的ip:2181 | tee codis.json
b、启动fe
cd /opt/app/codis/bin
nohup ./codis-fe --ncpu=4 --log=fe.log --log-level=WARN --dashboard-list=codis.json --listen=10.221.7.57:9090 &
c、登录fe
http://p:9090
4、启动redis-server
cd /opt/app/codis
./bin/codis-server ripsg/config/group_1/redis_6480.conf
5、启动redis-sentinel
cd /opt/app/codis
./bin/codis-server ripsg/config/sentinel_26480.conf --sentinel  &

（二）、停掉过程

1、删除这个codis安装目录
2、kill掉codis相关进程

（四）、在dashboard上配置

1、添加proxy

2、添加group

3、分配slots

4、添加sentinel

 

二、常用指令

1、master客户端登录：./redis-cli -p 6480 -h ip -a 'Test@123456'

2、proxy客户端登录：./redis-cli -p 19000 -h ip -a 'Test@123456'

3、codis版本信息：./codis-server --version

 

三、配置说明

注：redis_6480.conf、sentinel_26480.conf无个性ip；proxy_11080.toml、dashboard_18080.toml有个性ip

（一）、redis-server配置：redis_6480.conf

daemonize yes
pidfile "/opt/app/codis/ripsg/rdb/codis_6480.pid"
port 6480
tcp-backlog 511
timeout 0
tcp-keepalive 60
loglevel warning
logfile "/opt/app/codis/ripsg/rdb/codis_6480.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump_6480.rdb"
dir "/opt/app/codis/ripsg/rdb"
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
protected-mode no //升级sentinel指定
maxclients 4064
requirepass "Test@123456" //集群加密指定
masterauth "Test@123456"

rename-command FLUSHALL ""  //高危漏洞修复,防止清库

# Generated by CONFIG REWRITE //启动后才会产生如下配置
#slaveof 10.12.12.12 6382 //启动后，且在dashboard配置后才生成

（二）、redis-sentinel配置：sentinel_26480.conf

protected-mode no
port 26480
dir "/tmp"

# Generated by CONFIG REWRITE  //启动后才会有如下配置
#sentinel myid 24e6227a009519ecddae196fde0331b1ccec76ba
#sentinel monitor ripsg-2 ip 6380 2

#maxclients 4064
#sentinel failover-timeout ripsg-2 300000
#sentinel auth-pass ripsg-2 Test@123456 //启动后，密码自动添加到配置文件中
#sentinel config-epoch ripsg-2 0
#sentinel leader-epoch ripsg-2 0
#sentinel known-slave ripsg-2 ip 6381
#sentinel known-sentinel ripsg-2 ip 26482 b1196911fa5af4b317ccb3f5a19455aee51f40de
#sentinel known-sentinel ripsg-2 ip 26481 5383d1460dc9fe3bb0939e8f6d9999bdac1fa9a0
#sentinel monitor ripsg-1 ip 6382 2
#sentinel failover-timeout ripsg-1 300000
#sentinel auth-pass ripsg-1 Test@123456 //启动后，密码自动添加到配置文件中
#sentinel config-epoch ripsg-1 16
#sentinel leader-epoch ripsg-1 16
#sentinel known-slave ripsg-1 ip 6379
#sentinel known-sentinel ripsg-1 ip 26482 b1196911fa5af4b317ccb3f5a19455aee51f40de
#sentinel known-sentinel ripsg-1 ip 26481 5383d1460dc9fe3bb0939e8f6d9999bdac1fa9a0
#sentinel current-epoch 16

（三）、codis-proxy配置：proxy_11080.toml

product_name = "ripsg"
product_auth = "Test@123456"
session_auth = "Test@123456"
admin_addr = "本地ip:11080"
proto_type = "tcp4"
proxy_addr = "本地ip:19000"
jodis_name = "zookeeper"  //升级sentinel指定
jodis_addr = "zk1_ip:2181,zk2_ip:2181,zk3_ip:2181"
jodis_timeout = "20s"
jodis_compatible = true   //升级sentinel指定
proxy_datacenter = "ripsgDataCenter"
proxy_max_clients = 1000
proxy_max_offheap_size = "1024mb"
proxy_heap_placeholder = "256mb"
backend_ping_period = "5s"
backend_recv_bufsize = "128kb"
backend_recv_timeout = "30s"
backend_send_bufsize = "128kb"
backend_send_timeout = "30s"
backend_max_pipeline = 20480
backend_primary_only = false
backend_primary_parallel = 1
backend_replica_parallel = 1
backend_keepalive_period = "75s"
backend_number_databases = 16
session_recv_bufsize = "128kb"
session_recv_timeout = "30m"
session_send_bufsize = "64kb"
session_send_timeout = "30s"
session_max_pipeline = 10000
session_keepalive_period = "75s"
session_break_on_failure = false
metrics_report_server = ""
metrics_report_period = "1s"
metrics_report_influxdb_server = "http://172.16.122.18:8086"
metrics_report_influxdb_period = "1s"
metrics_report_influxdb_username = ""
metrics_report_influxdb_password = ""
metrics_report_influxdb_database = "ripsgcodismetrics"
metrics_report_statsd_server = ""
metrics_report_statsd_period = "1s"
metrics_report_statsd_prefix = ""

（四）、codis-dashboard配置：dashboard_18080.toml

coordinator_name = "zookeeper"
coordinator_addr = "zk1_ip:2181,zk2_ip:2182,zk3_ip:2183"
product_name = "ripsg"
product_auth = "Test@123456"
admin_addr = "本地ip:18080"
# Set arguments for data migration (only accept 'sync' & 'semi-async').
migration_method = "semi-async" //升级sentinel指定
migration_parallel_slots = 100 //升级sentinel指定
migration_async_maxbulks = 200 //升级sentinel指定
migration_async_maxbytes = "32mb" //升级sentinel指定
migration_async_numkeys = 500 //升级sentinel指定
migration_timeout = "30s" //升级sentinel指定
# Set configs for redis sentinel. 
sentinel_quorum = 2  //升级sentinel指定
sentinel_parallel_syncs = 1 //升级sentinel指定
sentinel_down_after = "30s" //升级sentinel指定
sentinel_failover_timeout = "5m" //升级sentinel指定
sentinel_notification_script = "" //升级sentinel指定
sentinel_client_reconfig_script = "" //升级sentinel指定

 

四、codis与redisCluster的区别

1、redis-client的jar包

a、redisCluster：jedis.jar

b、codis：jodis.jar

2、在哪计算key属于哪个槽，槽属于哪个masterNode

a、redisCluster：在客户端计算

JedisCluster.set(key, value)--->JedisClusterCommands.run(key)--->JedisClusterCommands.runWithRetries(key, ..., asking)--->JedisSlotBasedConnectionHandler.getConnectionFromSlot(slot)--->JedisClusterCRC16.getSlot(key)【计算槽属于哪个masterNode】&& JedisClusterInfoCache.getSlotPool(slot)【根据slot找到masterNode，JedisClusterInfoCache中含有clusterStat的缓存信息】

b、codis：在codis-proxy中计算

codis-proxy通过zookeeper拿到整个codis的拓扑（包含redis-group（master-slave）、redis-proxy、sentinel、slots、dashboard）