kubernetes 1.5安装 Redis 3.2.6 Cluster StatefulSet

最新推荐文章于 2024-09-06 15:10:02 发布

wenwst

最新推荐文章于 2024-09-06 15:10:02 发布

阅读量5.6k

点赞数 1

分类专栏： k8s kubernetes docker redis cluster

本文链接：https://blog.csdn.net/wenwst/article/details/54022129

版权

kubernetes 同时被 3 个专栏收录

34 篇文章 0 订阅

订阅专栏

k8s

26 篇文章 0 订阅

订阅专栏

docker

17 篇文章 0 订阅

订阅专栏

听说Redis 3以后安装集群比较简单，几个命令就可以完全搞定，在自己的电脑上面安装试了下，果然比较爽。

今天准备把Redis 3的集群安装在K8s上面，通常的做法都是在github上面查找是不是有人已经做过，果然很多。下面的代码基本上来自于github上面。不过就算用这个面的代码，在自己部署的时候也存在不少的问题。因为github上面的都是大神，很比比较小的问题不会写的很清楚。

所以：如果这个文档（blog）如何解决不了问题，就到github上面去查找别的方法。

1>>>>>>>>>>>镜像说明

使用的镜像名: ausov/redis:3.2-alpine

先使用下面命令下载镜像:

docker pull ausov/redis:3.2-alpine

这个镜像如果无法下载，则需要自己手动build一下.

这个是编译需要使用的文件

[root@centos-master alpine]# pwd
/root/k8s-redis/3.2/alpine
[root@centos-master alpine]# ls
docker-entrypoint.sh  Dockerfile  redis.conf

cat docker-entrypoint.sh

#!/bin/sh

set -eox pipefail
#shopt -s nullglob

REDIS_CONF=${REDIS_CONF:-"/opt/k8s-redis/redis.conf"}

if [ "${1#-}" != "$1" ] || [ "${1%.conf}" != "$1" ]; then
  set -- redis-server "$@"
fi

if [ "$1" = 'redis-server' ] && [ -n "$SLAVEOF" ] && [ -z "$SENTINEL" ]; then
  echo "Starting Redis replica"
  set -- $@ "$REDIS_CONF" --slaveof "$SLAVEOF" 6379

elif [ "$1" = 'redis-server' ] && [ -n "$SENTINEL" ]; then
  echo "Starting Redis sentinel"

  while true; do
    redis-cli -h $SENTINEL INFO
    if [[ "$?" == "0" ]]; then
      break
    fi
    echo "Connecting to master failed.  Waiting..."
    sleep 10
  done

  echo "sentinel monitor primary $SENTINEL 6379 2" >> "$REDIS_CONF"
  echo "sentinel down-after-milliseconds primary 5000" >> "$REDIS_CONF"
  echo "sentinel failover-timeout primary 10000" >> "$REDIS_CONF"
  echo "sentinel parallel-syncs primary 1" >> "$REDIS_CONF"

  set -- $@ "$REDIS_CONF" --port 26379 --sentinel --protected-mode no

elif [ "$1" = 'redis-server' ]; then  
  echo "Starting Redis master"
  set -- $@ "$REDIS_CONF"
fi

exec "$@"

这个文件内容太多，我们在这里看下可用的就OK。

cat redis.conf | grep -Ev '(^#|^$)'

protected-mode no
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize no
supervised no
pidfile /var/run/redis_6379.pid
loglevel notice
logfile ""
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir ./
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

cat Dockerfile

FROM redis:3.2-alpine

COPY ["redis.conf", "/opt/k8s-redis/"]
COPY ["docker-entrypoint.sh", "/usr/local/bin/"]

以上就是编译需要使用到的文件。cd到这个目录，执行下面的命令。

docker build -t "ausov/redis:3.2-alpine" .

镜像编译完成以后，需要确保所有的节点都有这个镜像。

2>>>>>>>>>>>部署安装

部署安装我们需要使用三个文件，分别是:

primary.yml secondary.yml sentinel.yml

下面是文件内容:

[root@centos-master example]# cat primary.yml 
apiVersion: v1
kind: Service
metadata:
  name: redis-primary
  labels:
    app: redis-primary
spec:
  ports:
  - port: 6379
    name: redis-primary
  clusterIP: None
  selector:
    app: redis-primary
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: redis-primary
spec:
  serviceName: redis-primary
  replicas: 1
  template:
    metadata:
      labels:
        app: redis-primary
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: redis-primary
        image: ausov/redis:3.2-alpine
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 6379
          name: redis-primary
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
      annotations:
        volume.alpha.kubernetes.io/storage-class: anything
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

[root@centos-master example]# cat secondary.yml 
apiVersion: v1
kind: Service
metadata:
  name: redis-secondary
  labels:
    app: redis-secondary
spec:
  ports:
  - port: 6379
    name: redis-secondary
  clusterIP: None
  selector:
    app: redis-secondary
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: redis-secondary
spec:
  serviceName: redis-secondary
  replicas: 2
  template:
    metadata:
      labels:
        app: redis-secondary
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: redis-secondary
        image: ausov/redis:3.2-alpine
        imagePullPolicy: IfNotPresent
        env:
        - name: SLAVEOF
          value: redis-primary-0.redis-primary
        ports:
        - containerPort: 6379
          name: redis-secondary
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
      annotations:
        volume.alpha.kubernetes.io/storage-class: anything
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

[root@centos-master example]# cat sentinel.yml 
apiVersion: v1
kind: Service
metadata:
  name: redis-sentinel
  labels:
    app: redis-sentinel
spec:
  ports:
  - port: 26379
    name: redis-sentinel
  clusterIP: None
  selector:
    app: redis-sentinel
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: redis-sentinel
spec:
  serviceName: redis-sentinel
  replicas: 3
  template:
    metadata:
      labels:
        app: redis-sentinel
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: redis-sentinel
        image: ausov/redis:3.2-alpine
        imagePullPolicy: IfNotPresent
        env:
        - name: SENTINEL
          value: redis-primary-0.redis-primary
        ports:
        - containerPort: 26379
          name: redis-sentinel

三个文件都存在以后:

 kubectl create -f primary.yml  -f secondary.yml  -f  sentinel.yml

创建完成后，查看一下pod

kubectl get pod

会得到以下信息:

NAME                                READY     STATUS    RESTARTS   AGE

redis-primary-0                     1/1       Running   0          2d
redis-secondary-0                   1/1       Running   0          2d
redis-secondary-1                   1/1       Running   0          2d
redis-sentinel-0                    1/1       Running   0          2d
redis-sentinel-1                    1/1       Running   0          2d
redis-sentinel-2                    1/1       Running   0          2d

下面是查看StatefulSet的信息：

[root@centos-master example]# kubectl get statefulset
NAME              DESIRED   CURRENT   AGE
redis-primary     1         1         2d
redis-secondary   2         2         2d
redis-sentinel    3         3         2d

必须要确保DESIRED和CURRENT的值是一样的。

查看持久存储的信息:

[root@centos-master example]# kubectl get pvc

NAME                                       STATUS    VOLUME                                     CAPACITY   ACCESSMODES   AGE

data-redis-primary-0                       Bound     pvc-94534255-d614-11e6-955d-1866daed1b04   5Gi        RWO           2d
data-redis-secondary-0                     Bound     pvc-9a8f2080-d614-11e6-955d-1866daed1b04   5Gi        RWO           2d
data-redis-secondary-1                     Bound     pvc-9a9029b4-d614-11e6-955d-1866daed1b04   5Gi        RWO           2d

我们在注意这里的STATUS，是Bound。

如果不是，用命令

describe pvc data-redis-primary-0

如果出现下面的问题：

cannot find volume plugin for alpha provisioning

则需要修改文件:

/etc/kubernetes/manifests/kube-controller-manager.json

增加参数: --enable-hostpath-provisioner=true

以下是截取了这个文件的一部分:

        "name": "kube-controller-manager",
        "image": "gcr.io/google_containers/kube-controller-manager-amd64:v1.5.1",
        "command": [
          "kube-controller-manager",
          "--address=127.0.0.1",
          "--leader-elect",
          "--master=127.0.0.1:8080",
          "--cluster-name=kubernetes",
          "--root-ca-file=/etc/kubernetes/pki/ca.pem",
          "--service-account-private-key-file=/etc/kubernetes/pki/apiserver-key.pem",
          "--cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem",
          "--cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem",
          "--insecure-experimental-approve-all-kubelet-csrs-for-group=system:kubelet-bootstrap",
          "--allocate-node-cidrs=true",
          "--enable-hostpath-provisioner=true",
          "--cluster-cidr=10.245.0.0/16"
        ],

修改保存，重启一下docker或kubelet.

通过上面的配置，Redis基本就配置完成:

这两个WARNING暂不影响redis使用，解决方法过两天再写上。

11:M 04 Jan 09:45:23.515 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
11:M 04 Jan 09:45:23.515 # Server started, Redis version 3.2.1
11:M 04 Jan 09:45:23.515 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
11:M 04 Jan 09:45:23.516 * The server is now ready to accept connections on port 6379
OK