redis集群部署_redis集群方案之Codis集群部署

部署 codis 集群简介

集群需要依赖于 zookeeper 或者 etcd 服务。这里使用的是 zookeeper 。

zookeeper 最好选择使用集群模式部署，提高可用性。

codis-dashboard 通过操作 zookeeper 保存 dashboard、proxy、slot 等相关信息。

通过 codis-dashboard 可以添加 codis-proxy、codis-server、分组、sentinel、迁移 slot 等。

codis-fe 可以通过 zookeeper，获取对应的信息，来展示，并获取 dashboard 地址，来进行控制。

Codis 下载地址

wget -c https://github.com/CodisLabs/codis/releases/download/3.2.2/codis3.2.2-go1.9.2-linux.tar.gztar zxvf codis3.2.2-go1.9.2-linux.tar.gz

cd codis3.2.2-go1.9.2-linuxmkdir etc

配置 codis-dashboard

可以使用 codis-dashboard 生成默认配置文件

./codis-dashboard --default-config  | tee etc/dashboard.toml

修改对应的 zookeeper 地址，为了安全性可以修改 product_name 和 product_auth，整个集群的 product 相关信息要保持一致。

###################################################                                                ##                  Codis-Dashboard               ##                                                #################################################### Set Coordinator, only accept "zookeeper" & "etcd" & "filesystem".# for zookeeper/etcd, coorinator_auth accept "user:password"# Quick Start#coordinator_name = "filesystem"#coordinator_addr = "/tmp/codis"# 这里修改zookeeper的地址coordinator_name = "zookeeper"coordinator_addr = "192.168.122.101:2181,192.168.122.102:2181,192.168.122.103:2181"#coordinator_auth = ""# Set Codis Product Name/Auth.product_name = "codis3"product_auth = "h6oewuoaudPgCJRzYg0Uo"# Set bind address for admin(rpc), tcp only.admin_addr = "0.0.0.0:18080"# Set arguments for data migration (only accept 'sync' & 'semi-async').migration_method = "semi-async"migration_parallel_slots = 100migration_async_maxbulks = 200migration_async_maxbytes = "32mb"migration_async_numkeys = 500migration_timeout = "30s"# Set configs for redis sentinel.sentinel_client_timeout = "10s"sentinel_quorum = 2sentinel_parallel_syncs = 1sentinel_down_after = "30s"sentinel_failover_timeout = "5m"sentinel_notification_script = ""sentinel_client_reconfig_script = ""

启动命令

nohup ./codis-dashboard --ncpu=2 --config=./etc/dashboard.toml --log=/data/codis/logs/dashboard.log --log-level=WARN &

--ncpu 可以指定使用几个 CPU--config 指定配置文件的路径--log 指定日志文件路径--log-devel 指定日志级别

配置 codis-proxy

使用 codis-proxy 生成默认配置文件

./codis-proxy  --default-config   | tee etc/proxy.toml

修改 product_name 和 product_auth，整个集群的 product 相关信息要保持一致。session_auth 是用于连接的验证，可以跟集群的验证不一样。如果短连接比较多可以调整一下 proxy_max_clients 参数。

###################################################                                                ##                  Codis-Proxy                   ##                                                #################################################### Set Codis Product Name/Auth.product_name = "codis3"product_auth = "h6oewuoaudPgCJRzYg0Uo"# Set auth for client session#   1. product_auth is used for auth validation among codis-dashboard,#      codis-proxy and codis-server.#   2. session_auth is different from product_auth, it requires clients#      to issue AUTH  before processing any other commands.session_auth = "L5SqVOieIOk7IyfYscCJcCdQBtHlK5Uq"# Set bind address for admin(rpc), tcp only.admin_addr = "0.0.0.0:11080"# Set bind address for proxy, proto_type can be "tcp", "tcp4", "tcp6", "unix" or "unixpacket".proto_type = "tcp4"proxy_addr = "0.0.0.0:19000"# Set jodis address & session timeout#   1. jodis_name is short for jodis_coordinator_name, only accept "zookeeper" & "etcd".#   2. jodis_addr is short for jodis_coordinator_addr#   3. jodis_auth is short for jodis_coordinator_auth, for zookeeper/etcd, "user:password" is accepted.#   4. proxy will be registered as node:#        if jodis_compatible = true (not suggested):#          /zk/codis/db_{PRODUCT_NAME}/proxy-{HASHID} (compatible with Codis2.0)#        or else#          /jodis/{PRODUCT_NAME}/proxy-{HASHID}jodis_name = ""jodis_addr = ""jodis_auth = ""jodis_timeout = "20s"jodis_compatible = false# Set datacenter of proxy.proxy_datacenter = ""# Set max number of alive sessions.proxy_max_clients = 10000# Set max offheap memory size. (0 to disable)proxy_max_offheap_size = "1024mb"# Set heap placeholder to reduce GC frequency.proxy_heap_placeholder = "256mb"# Proxy will ping backend redis (and clear 'MASTERDOWN' state) in a predefined interval. (0 to disable)backend_ping_period = "5s"# Set backend recv buffer size & timeout.backend_recv_bufsize = "128kb"backend_recv_timeout = "30s"# Set backend send buffer & timeout.backend_send_bufsize = "128kb"backend_send_timeout = "30s"# Set backend pipeline buffer size.backend_max_pipeline = 20480# Set backend never read replica groups, default is falsebackend_primary_only = false# Set backend parallel connections per serverbackend_primary_parallel = 1backend_replica_parallel = 1# Set backend tcp keepalive period. (0 to disable)backend_keepalive_period = "75s"# Set number of databases of backend.backend_number_databases = 16# If there is no request from client for a long time, the connection will be closed. (0 to disable)# Set session recv buffer size & timeout.session_recv_bufsize = "128kb"session_recv_timeout = "30m"# Set session send buffer size & timeout.session_send_bufsize = "64kb"session_send_timeout = "30s"# Make sure this is higher than the max number of requests for each pipeline request, or your client may be blocked.# Set session pipeline buffer size.session_max_pipeline = 10000# Set session tcp keepalive period. (0 to disable)session_keepalive_period = "75s"# Set session to be sensitive to failures. Default is false, instead of closing socket, proxy will send an error response to client.session_break_on_failure = false# Set metrics server (such as http://localhost:28000), proxy will report json formatted metrics to specified server in a predefined period.metrics_report_server = ""metrics_report_period = "1s"# Set influxdb server (such as http://localhost:8086), proxy will report metrics to influxdb.metrics_report_influxdb_server = ""metrics_report_influxdb_period = "1s"metrics_report_influxdb_username = ""metrics_report_influxdb_password = ""metrics_report_influxdb_database = ""# Set statsd server (such as localhost:8125), proxy will report metrics to statsd.metrics_report_statsd_server = ""metrics_report_statsd_period = "1s"metrics_report_statsd_prefix = ""

启动命令

nohup ./codis-proxy --ncpu=2 --config=./etc/proxy.toml --log=/data/codis/logs/proxy.log --log-level=WARN &

--ncpu 可以指定使用几个 CPU--config 指定配置文件的路径--log 指定日志文件路径--log-devel 指定日志级别

配置 codis-server

codis-server 跟 redis-server 配置是一样的。设置的密码要跟 product_auth 这个参数一样，集群才能验证通过。

例如配置一个 7001.conf 的配置文件，如果多个节点，复制复制一份，替换文本中 7001 关键字即可。

daemonize yespidfile "/data/codis/redis_7001.pid"port 7001timeout 60loglevel warninglogfile "/data/codis/logs/codis-server-7001.log"databases 16rdbcompression yesdbfilename "dump-7001.rdb"dir "/data/codis/data"maxmemory 5gbmaxmemory-policy volatile-lruappendonly yesappendfilename "appendonly7001.aof"appendfsync everysecno-appendfsync-on-rewrite yesauto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mbslowlog-log-slower-than 10000slowlog-max-len 1024hz 50aof-rewrite-incremental-fsync yesrequirepass "h6oewuoaudPgCJRzYg0Uo"# Generated by CONFIG REWRITEmasterauth "h6oewuoaudPgCJRzYg0Uo"

启动命令

./codis-server etc/7001.conf...

配置 codis-fe

nohup ./codis-fe --ncpu=2 --log=/data/codis/logs/fe.log --log-level=WARN --zookeeper=192.168.122.101:2181,192.168.122.102:2181,192.168.122.103:2181  --listen=0.0.0.0:8080 &

--ncpu 指定CPU个数--log 指定日志文件--log-level 日志级别--zookeeper 指定zookeeper地址--listen 监听地址:端口

配置 sentinel

cat > etc/sentinel.confport 26379protected-mode nodaemonize yes

启动命令

/redis-sentinel etc/sentinel.conf

Web 界面

在 codis-fe 启动的节点，访问 web 页面。

通过 New Proxy 添加 codis-proxy 节点，默认端口是 11080

通过 New Group 添加分组 ，按照数字 1 2 3 …来即可。

通过 Add Server 添加节点到对应的分组，例如 192.168.122.101:7001 to 1

通过 Migrate Slots 分配 slot 到对应的分组，例如：0 ~ 40 to 1。

通过 Add Sentinel 添加 sentinel，例如：192.168.122.101:26379，添加后，主节点会有 HA 的标记