Centos7.0上用docker下搭建codis集群

1.环境

参数	值
宿主机ip	10.210.234.203
宿主机内核版本	3.10.0-123.el7.x86_64
宿主机cpu	4个单核cpu
docker版本	1.9.1
网络模式	host
用户	root

2.运行方式

• 1.在docker容器中运行codis服务程序
• 2.在宿主机上存储配置文件，docker配置文件映射到宿主机上

3.开始搭建

1.在宿主机上创建以下目录：

mkdir -p  /data1/codis/config;
mkdir -p /data1/codis/logs;
mkdir -p /data1/codis/pid;
mkdir -p /data1/codis/out;
mkdir -p /data1/codis/storage;
mkdir -p /data1/codis/data;

目录说明:

目录	说明
config	配置文件目录
logs	日志文件目录
pid	pid文件目录
assets	存放FE文件(js,css)
stroge	codis元信息存储目录
data	存储redis数据文件

2.启动docker容器

docker run -it  --net=host --name=codis_container -v /data1:/data1 centos /bin/bash 

(如果本地没有centos的镜像,会从远程拉取,稍微会慢一点)

  说明：
      --name:           容器的名称
      --net:            容器与主机的连接方式
      -v /data1:/data1  目录映射，程序配置文件及日志都会在这个目录中，docker中访问/data1目录会直接访问宿主机的/data1目录

3.进入容器,下载安装codis二进制文件包(这里不使用源码安装的方式)

# 下载:
wget https://github.com/CodisLabs/codis/releases/download/3.2.1/codis3.2.1-go1.7.6-linux.tar.gz

# 解压

tar -zxvf codis3.2.1-go1.7.6-linux.tar.gz

# 拷贝目录到/usr/local/bin/codis目录

mv codis3.2.1-go1.7.6-linux /usr/local/bin/codis

# 拷贝assets目录到宿主机(这个是web界面,拷出来方便修改)在宿主机下执行

docker cp codis_container:/usr/local/bin/codis/assets /data1/codis/assets

4.启动FE

1.进入到容器:docker exec -it codis_container
2.执行命令：

/usr/local/bin/codis/codis-fe \
--assets-dir=/data1/codis/assets \
--filesystem=/data1/codis/storage \
--log=/data1/codis/logs/fe.log \
--pidfile=/data1/codis/pid/fe.pid \
--log-level=INFO  \
--listen=0.0.0.0:9090 \
>/data1/codis/out/fe.log 2>/dev/null &

说明：
    --assets 指定web界面所在目录
    --filesystem 以文件的形式存储codis集群元信息
    --log  fe日志文件
    --pidfile pid文件
    --listen  web服务端口

3.访问：http://x.x.x.x:9090 出现以下界面则说明启动成功,如果失败则查看log日志

5.启动dashboard

1.在宿主机/data1/codis/config目录新增dashboard.toml文件
2.内容如下

coordinator_name = "filesystem"
coordinator_addr = "/data1/codis/storage"
product_name = "codis-demo"
product_auth = ""
admin_addr = "0.0.0.0:18080"
migration_method = "semi-async"
migration_parallel_slots = 100
migration_async_maxbulks = 200
migration_async_maxbytes = "32mb"
migration_async_numkeys = 500
migration_timeout = "30s"
sentinel_client_timeout = "10s"
sentinel_quorum = 2
sentinel_parallel_syncs = 1
sentinel_down_after = "30s"
sentinel_failover_timeout = "5m"
sentinel_notification_script = ""
sentinel_client_reconfig_script = ""

3.在codis_container中执行

/usr/local/bin/codis/codis-dashboard \
--config=/data1/codis/config/dashboard.toml \
--log=/data1/codis/logs/dashboard.log \
--log-level=INFO \
--pidfile=/data1/codis/pid/dashboard.pid &

4.成功web界面中看到如下，失败查看日志信息

6.启动proxy

1.在宿主机/data1/codis/config目录新增proxy.toml文件
2.添加如下内容

product_name = "codis-demo"
product_auth = ""
session_auth = ""
admin_addr = "0.0.0.0:11080"
proto_type = "tcp4"
proxy_addr = "0.0.0.0:19000"
jodis_name = ""
jodis_addr = ""
jodis_auth = ""
jodis_timeout = "20s"
jodis_compatible = false
proxy_datacenter = ""
proxy_max_clients = 1000
proxy_max_offheap_size = "1024mb"
proxy_heap_placeholder = "256mb"
backend_ping_period = "5s"
backend_recv_bufsize = "128kb"
backend_recv_timeout = "30s"
backend_send_bufsize = "128kb"
backend_send_timeout = "30s"
backend_max_pipeline = 20480
backend_primary_only = false
backend_primary_parallel = 1
backend_replica_parallel = 1
backend_keepalive_period = "75s"
backend_number_databases = 16
session_recv_bufsize = "128kb"
session_recv_timeout = "30m"
session_send_bufsize = "64kb"
session_send_timeout = "30s"
session_max_pipeline = 10000
session_keepalive_period = "75s"
session_break_on_failure = false
metrics_report_server = ""
metrics_report_period = "1s"
metrics_report_influxdb_server = ""
metrics_report_influxdb_period = "1s"
metrics_report_influxdb_username = ""
metrics_report_influxdb_password = ""
metrics_report_influxdb_database = ""
metrics_report_statsd_server = ""
metrics_report_statsd_period = "1s"
metrics_report_statsd_prefix = ""

3.执行命令

  /usr/local/bin/codis/codis-proxy \
  --config=/data1/codis/config/proxy.toml \
  --dashboard=127.0.0.1:18080  \
  --log=/data1/codis/logs/proxy.log \
  --pidfile=/data1/codis/pid/proxy.pid \
  > /data1/codis/out/proxy.log 2>/dev/null &

4.成功看到如下信息

7.启动redis服务组(这里启动两组)

1.在/data1/codis/config新增redis.conf文件
2.添加内容：

protected-mode no
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize yes
supervised no
loglevel notice
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
dir "/data1/codis/data"
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

group1:
master

/usr/local/bin/codis/codis-server /data1/codis/config/redis.conf \
--port 6379 \
--pidfile /data1/codis/pid/redis_6379.pid  \
--logfile /data1/codis/logs/redis_6379.log

slave1

/usr/local/bin/codis/codis-server \
/data1/codis/config/redis.conf \
--port 16379 \
--slaveof 127.0.0.1 6379 \
--pidfile /data1/codis/pid/redis_1637.pid  \
--logfile /data1/codis/logs/redis_16379.log

slave2

/usr/local/bin/codis/codis-server \
/data1/codis/config/redis.conf \
--port 26379 \
--slaveof 10.210.234.203 6379 \
--pidfile /data1/codis/pid/redis_26379.pid  \
--logfile /data1/codis/logs/redis_26379.log

group2:

master:
/usr/local/bin/codis/codis-server \
/data1/codis/config/redis.conf \
--port 6380 \
--pidfile /data1/codis/pid/redis_6380.pid  \
--logfile /data1/codis/logs/redis_6380.log

slave1:
/usr/local/bin/codis/codis-server \
/data1/codis/config/redis.conf \
--port 16380  \
--slaveof 127.0.0.1 6380 \
--pidfile /data1/codis/pid/redis_16380.pid  \
--logfile /data1/codis/logs/redis_16380.log

slave2
 /usr/local/bin/codis/codis-server \
 /data1/codis/config/redis.conf \
 --port 26380  \
 --slaveof 127.0.0.1 6380 \
 --pidfile /data1/codis/pid/redis_26380.pid  \
 --logfile /data1/codis/logs/redis_26380.log

7.添加分组添加到集群中

8.添加sentinel

1.添加sentinel.conf.001到/data1/codis/config

   protected-mode no
port 25379
dir /data1/codis/data
sentinel monitor mymaster1 127.0.0.1 6379 2
sentinel monitor mymaster2 127.0.0.1 6380 2
sentinel down-after-milliseconds mymaster1 30000
sentinel down-after-milliseconds mymaster2 30000
sentinel parallel-syncs mymaster1 1
sentinel parallel-syncs mymaster2 1
sentinel failover-timeout mymaster1 180000
sentinel failover-timeout mymaster2 180000
sentinel client-reconfig-script mymaster1 /data1/codis/sentinal.sh
sentinel client-reconfig-script mymaster2 /data1/codis/sentinal.sh

2.添加sentinel.conf.002到/data1/codis/config

protected-mode no
port 25380
dir /data1/codis/data
sentinel monitor mymaster1 127.0.0.1 6379 2
sentinel monitor mymaster2 127.0.0.1 6380 2
sentinel down-after-milliseconds mymaster1 30000
sentinel down-after-milliseconds mymaster2 30000
sentinel parallel-syncs mymaster1 1
sentinel parallel-syncs mymaster2 1
sentinel failover-timeout mymaster1 180000
sentinel failover-timeout mymaster2 180000
sentinel client-reconfig-script mymaster1 /data1/codis/sentinal.sh
sentinel client-reconfig-script mymaster2 /data1/codis/sentinal.sh

3.添加sentinel.conf.003到/data1/codis/config

protected-mode no
port 25381
dir /data1/codis/data
sentinel monitor mymaster1 127.0.0.1 6379 2
sentinel monitor mymaster2 127.0.0.1 6380 2
sentinel down-after-milliseconds mymaster1 30000
sentinel down-after-milliseconds mymaster2 30000
sentinel parallel-syncs mymaster1 1
sentinel parallel-syncs mymaster2 1
sentinel failover-timeout mymaster1 180000
sentinel failover-timeout mymaster2 180000
sentinel client-reconfig-script mymaster1 /data1/codis/sentinal.sh
sentinel client-reconfig-script mymaster2 /data1/codis/sentinal.sh

4.添加到哨兵通知脚本到/data1/codis

vi /data1/codis/sentinal.sh

添加如下内容：

#!/bin/bash
echo $1 $2 $3 $4 $5 $6 $7 >> ./notice.txt

当redis故障迁移时完成时会调用：

格式：/data1/codis/sentinal.sh master-name role state from-ip from-port to-ip to-port

脚本必须以 #!/bin/bash 开头，一般会用来报警,通知客户端更新主服务器ip什么的

chmod +x /data1/codis/sentinal.sh

5.启动哨兵：

/usr/local/bin/codis/redis-sentinel   /data1/codis/config/sentinel.conf.003 \
>/dev/null 2>/dev/null &

/usr/local/bin/codis/redis-sentinel \
 /data1/codis/config/sentinel.conf.002 \
 >/dev/null 2>/dev/null &

/usr/local/bin/codis/redis-sentinel \
/data1/codis/config/sentinel.conf.001 \
>/dev/null 2>/dev/null &

5.添加到集群中

9.启动域名服务(在宿主机上安装)

1.yum -y install bind bind-utils bind-chroot
2.vim /etc/named.conf 修改如下图所示

3 .vi /etc/named.rfc1912.zones 添加一组域名服务

4.vim /var/named/named.service.com 新增内容

$TTL 1D
@       IN SOA  @ test01.service. (
                                        0       ; serial
                                        1D      ; refresh
                                        1H      ; retry
                                        1W      ; expire
                                        3H )    ; minimum
@ IN NS test01.service.
test01.service. IN A 10.210.234.203
redis19000.huiyuan.comm.service. IN A 10.210.234.203
redis19001.huiyuan.comm.service. IN A 10.210.234.203
redis19001.huiyuan.activity.service. IN A 10.210.234.203

5.重启named服务器 (如果service不存在，执行yum install initscripts )

service named restart

6.在/etc/resolv.conf文件中新增一行

nameserver 10.210.234.203

7.ping redis19000.huiyuan.comm.service 则说明配置成功

8.此时redis服务信息：

主机： redis19000.huiyuan.comm.service
端口： 19000

9.登录redis服务

redis-cli -h redis19000.huiyuan.comm.service -p 19000