codis3部署实践

1. 组件列表：

Codis Server：基于 redis-3.2.8 分支开发。增加了额外的数据结构，以支持 slot 有关的操作以及数据迁移指令。具体的修改可以参考文档 redis 的修改。

Codis Proxy：客户端连接的 Redis 代理服务, 实现了 Redis 协议。 除部分命令不支持以外(不支持的命令列表)，表现的和原生的 Redis 没有区别（就像 Twemproxy）。

对于同一个业务集群而言，可以同时部署多个 codis-proxy 实例；
不同 codis-proxy 之间由 codis-dashboard 保证状态同步。
Codis Dashboard：集群管理工具，支持 codis-proxy、codis-server 的添加、删除，以及据迁移等操作。在集群状态发生改变时，codis-dashboard 维护集群下所有 codis-proxy 的状态的一致性。

对于同一个业务集群而言，同一个时刻 codis-dashboard 只能有 0个或者1个；
所有对集群的修改都必须通过 codis-dashboard 完成。
Codis Admin：集群管理的命令行工具。

可用于控制 codis-proxy、codis-dashboard 状态以及访问外部存储。
Codis FE：集群管理界面。

多个集群实例共享可以共享同一个前端展示页面；
通过配置文件管理后端 codis-dashboard 列表，配置文件可自动更新。
Storage：为集群状态提供外部存储。

提供 Namespace 概念，不同集群的会按照不同 product name 进行组织；
目前仅提供了 Zookeeper、Etcd、Fs 三种实现，但是提供了抽象的 interface 可自行扩展。

2. 下载

https://github.com/CodisLabs/codis/releases

3. 安装

安装dashboard:

• 生成dashboard配置文件

./codis-dashboard --default-config | tee dashboard.toml

• 启动：

nohup ./codis-dashboard --ncpu=1 --config=dashboard.toml --log=logs/dashboard.log --log-level=WARN &

安装proxy

./codis-admin --remove-lock --product=codis-zijinph --zookeeper=10.1.30.101:2181

• 生成配置文件

./codis-proxy --default-config | tee proxy.toml

• 启动：

nohup ./codis-proxy --ncpu=2 --config=proxy.toml --log=logs/proxy.log --log-level=WARN &

./codis-admin --dashboard=127.0.0.1:18080 --remove-proxy --addr=127.0.0.1:11080 --force

./codis-admin --dashboard=10.1.28.110:18080 --create-proxy -x 10.1.28.110:11080

启动codis-server

• 启动server
  nohup ./codis-server redis-6379.conf &
  nohup ./codis-server redis-6380.conf &
  nohup ./codis-server redis-6479.conf &
  nohup ./codis-server redis-6480.conf &

• 启动哨兵，做高可用，不需要额外配置，只要配置下面几个信息即可

port 26379
dir "/data/codis/sentine26379"
protected-mode no

• 创建日志目录

mkdir -p /data/codis/sentine26379/
mkdir -p /data/codis/sentine26380/
mkdir -p /data/codis/sentine26381/

• 启动哨兵

nohup ./codis-server sentinel-26379.conf --sentinel &
nohup ./codis-server sentinel-26380.conf --sentinel &
nohup ./codis-server sentinel-26381.conf --sentinel &

启动fe

• 生成配置文件
  ./codis-admin --dashboard-list --zookeeper=10.1.30.101:2181,10.1.30.102:2181,10.1.30.103:2181,10.1.30.104:2181 | tee codis.json

• 启动fe
  nohup ./codis-fe --ncpu=1 --log=logs/fe.log --log-level=WARN --zookeeper=10.1.30.101:2181,10.1.30.102:2181,10.1.30.103:2181,10.1.30.104:2181 --listen=10.1.28.110:8080 &

4. 压测

• 单proxy
  ./redis-benchmark -h 10.1.28.112 -p 19000 -a “xxxxxx” -c 10 -n 10000
• 单master
  ./redis-benchmark -h 10.1.28.110 -p 6479 -a “xxxxxx” -c 10 -n 10000

5. 问题

• 编译文件内没有codis-server的配置文件，可以找一个redis-server复制一个server.conf；
• 节点故障后，sentinel的主备切换时间通过dashboard.toml配置文件的sentinel_down_after = “3s” 进行修改；
• proxy、server、sentinel 都可以从fe的页面进行添加；

6. 配置文件示例：

• [root@xx codis3.2.0-go1.8.1-linux]# more dashboard.toml

##################################################
#                                                #
#                  Codis-Dashboard               #
#                                                #
##################################################

# Set Coordinator, only accept "zookeeper" & "etcd" & "filesystem".
# Quick Start
coordinator_name = "zookeeper"
coordinator_addr = "10.1.30.101:2181,10.1.30.102:2181,10.1.30.103:2181,10.1.30.104:2181"

# Set Codis Product Name/Auth.
product_name = "codis-zijinph"
product_auth = "xxx"

# Set bind address for admin(rpc), tcp only.
admin_addr = "10.1.28.110:18080"

# Set arguments for data migration (only accept 'sync' & 'semi-async').
migration_method = "semi-async"
migration_parallel_slots = 100
migration_async_maxbulks = 200
migration_async_maxbytes = "32mb"
migration_async_numkeys = 500
migration_timeout = "30s"

# Set configs for redis sentinel.
sentinel_quorum = 2
sentinel_parallel_syncs = 1
sentinel_down_after = "3s"
sentinel_failover_timeout = "5m"
sentinel_notification_script = ""
sentinel_client_reconfig_script = ""

• [root@microserviceapp001 codis3.2.0-go1.8.1-linux]# more proxy.toml

##################################################
#                                                #
#                  Codis-Proxy                   #
#                                                #
##################################################

# Set Codis Product Name/Auth.
product_name = "codis-zijinph"
product_auth = "zijinph123"

# Set auth for client session
#   1. product_auth is used for auth validation among codis-dashboard,
#      codis-proxy and codis-server.
#   2. session_auth is different from product_auth, it requires clients
#      to issue AUTH <PASSWORD> before processing any other commands.
session_auth = "zijin123456"

# Set bind address for admin(rpc), tcp only.
admin_addr = "10.1.28.110:11080"

# Set bind address for proxy, proto_type can be "tcp", "tcp4", "tcp6", "unix" or "unixpacket".
proto_type = "tcp4"
proxy_addr = "10.1.28.110:19000"

# Set jodis address & session timeout
#   1. jodis_name is short for jodis_coordinator_name, only accept "zookeeper" & "etcd".
#   2. jodis_addr is short for jodis_coordinator_addr
#   3. proxy will be registered as node:
#        if jodis_compatible = true (not suggested):
#          /zk/codis/db_{PRODUCT_NAME}/proxy-{HASHID} (compatible with Codis2.0)
#        or else
#          /jodis/{PRODUCT_NAME}/proxy-{HASHID}
jodis_name = "zookeeper"
jodis_addr = "10.1.30.101:2181,10.1.30.102:2181,10.1.30.103:2181,10.1.30.104:2181"
jodis_timeout = "20s"
jodis_compatible = false

# Set datacenter of proxy.
proxy_datacenter = ""

# Set max number of alive sessions.
proxy_max_clients = 1000

# Set max offheap memory size. (0 to disable)
proxy_max_offheap_size = "1024mb"

# Set heap placeholder to reduce GC frequency.
proxy_heap_placeholder = "256mb"

# Proxy will ping backend redis (and clear 'MASTERDOWN' state) in a predefined interval. (0 to disable)
backend_ping_period = "5s"

# Set backend recv buffer size & timeout.
backend_recv_bufsize = "128kb"
backend_recv_timeout = "30s"

# Set backend send buffer & timeout.
backend_send_bufsize = "128kb"
backend_send_timeout = "30s"

# Set backend pipeline buffer size.
backend_max_pipeline = 20480

# Set backend never read replica groups, default is false
backend_primary_only = false

# Set backend parallel connections per server
backend_primary_parallel = 1
backend_replica_parallel = 1

# Set backend tcp keepalive period. (0 to disable)
backend_keepalive_period = "75s"

# Set number of databases of backend.
backend_number_databases = 16

# If there is no request from client for a long time, the connection will be closed. (0 to disable)
# Set session recv buffer size & timeout.
session_recv_bufsize = "128kb"
session_recv_timeout = "30m"

# Set session send buffer size & timeout.
session_send_bufsize = "64kb"
session_send_timeout = "30s"

# Make sure this is higher than the max number of requests for each pipeline request, or your client may be blocked.
# Set session pipeline buffer size.
session_max_pipeline = 10000

# Set session tcp keepalive period. (0 to disable)
session_keepalive_period = "75s"

# Set session to be sensitive to failures. Default is false, instead of closing socket, proxy will send an error response to client.
session_break_on_failure = false

# Set metrics server (such as http://localhost:28000), proxy will report json formatted metrics to specified server in a predefined period.
metrics_report_server = ""
metrics_report_period = "1s"

# Set influxdb server (such as http://localhost:8086), proxy will report metrics to influxdb.
metrics_report_influxdb_server = ""
metrics_report_influxdb_period = "1s"
metrics_report_influxdb_username = ""
metrics_report_influxdb_password = ""
metrics_report_influxdb_database = ""

# Set statsd server (such as localhost:8125), proxy will report metrics to statsd.
metrics_report_statsd_server = ""
metrics_report_statsd_period = "1s"
metrics_report_statsd_prefix = ""