需求
- k8s(Kubernetes)中安装Redis集群
- 使用nfs存储
- 部署Redis集群 3主3从
- 服务器内部ip:192.168.0.254
必备环境安装
ubuntu20.x 服务器 nfs服务安装
https://foxwho.blog.csdn.net/article/details/116994098
#创建nfs redis 存储目录 共 6个
mkdir -p /nfs/redis/pv{1..6}
#编辑 nfs 存储配置文件
vim /etc/exports
增加或修改如下
/nfs/redis/pv1 *(ro,sync,no_root_squash,no_subtree_check)
/nfs/redis/pv2 *(ro,sync,no_root_squash,no_subtree_check)
/nfs/redis/pv3 *(ro,sync,no_root_squash,no_subtree_check)
/nfs/redis/pv4 *(ro,sync,no_root_squash,no_subtree_check)
/nfs/redis/pv5 *(ro,sync,no_root_squash,no_subtree_check)
/nfs/redis/pv6 *(ro,sync,no_root_squash,no_subtree_check)
重新 刷新配置共享配置文件中的共享目录
exportfs -rv
查看 nfs共享目录
showmount -e 192.168.0.254
输出
Export list for 192.168.0.254:
/nfs/redis/pv6 *
/nfs/redis/pv5 *
/nfs/redis/pv4 *
/nfs/redis/pv3 *
/nfs/redis/pv2 *
/nfs/redis/pv1 *
k8s 安装略
请自行搜索 k8s 安装
Kubernetes部署redis集群
创建ConfigMap
redis-cm.yml
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-cluster
data:
update-node.sh: |
#!/bin/sh
REDIS_NODES="/data/nodes.conf"
sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${REDIS_NODES}
exec "$@"
redis.conf: |+
cluster-enabled yes
cluster-require-full-coverage no
cluster-node-timeout 15000
cluster-config-file /data/nodes.conf
cluster-migration-barrier 1
dir /data
appendonly yes
protected-mode no
执行创建命令
kubectl apply -f redis-cm.yml
查看ConfigMap
kubectl get cm
输出
NAME DATA AGE
kube-root-ca.crt 1 42h
redis-cluster 2 16h
创建pv
redis-pv.yml
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-pv1
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: "redis-cluster"
nfs:
path: /nfs/redis/pv1
server: 192.168.0.254
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-pv2
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: "redis-cluster"
nfs:
path: /nfs/redis/pv2
server: 192.168.0.254
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-pv3
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: "redis-cluster"
nfs:
path: /nfs/redis/pv3
server: 192.168.0.254
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-pv4
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: "redis-cluster"
nfs:
path: /nfs/redis/pv4
server: 192.168.0.254
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-pv5
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: "redis-cluster"
nfs:
path: /nfs/redis/pv5
server: 192.168.0.254
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-pv6
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: "redis-cluster"
nfs:
path: /nfs/redis/pv6
server: 192.168.0.254
执行创建命令
kubectl apply -f redis-pv.yml
输出
persistentvolume/redis-pv1 created
persistentvolume/redis-pv2 created
persistentvolume/redis-pv3 created
persistentvolume/redis-pv4 created
persistentvolume/redis-pv5 created
persistentvolume/redis-pv6 created
查看pv
kubectl get pv
输出
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
redis-pv1 5Gi RWO Recycle Bound default/data-redis-cluster-0 redis-cluster 16h
redis-pv2 5Gi RWO Recycle Bound default/data-redis-cluster-1 redis-cluster 16h
redis-pv3 5Gi RWO Recycle Bound default/data-redis-cluster-2 redis-cluster 16h
redis-pv6 5Gi RWO Recycle Bound default/data-redis-cluster-3 redis-cluster 16h
redis-pv4 5Gi RWO Recycle Bound default/data-redis-cluster-4 redis-cluster 16h
redis-pv5 5Gi RWO Recycle Bound default/data-redis-cluster-5 redis-cluster 16h
创建 StatefulSet
redis-sts.yml
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-cluster
spec:
serviceName: redis-cluster
replicas: 6
selector:
matchLabels:
app: redis-cluster
template:
metadata:
labels:
app: redis-cluster
spec:
containers:
- name: redis
image: redis:6.2-alpine
ports:
- containerPort: 6379
name: client
- containerPort: 16379
name: gossip
command: ["/conf/update-node.sh", "redis-server", "/conf/redis.conf"]
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- name: conf
mountPath: /conf
readOnly: false
- name: data
mountPath: /data
readOnly: false
volumes:
- name: conf
configMap:
name: redis-cluster
defaultMode: 0755
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 5Gi
storageClassName: redis-cluster
执行创建命令
kubectl apply -f redis-sts.yml
输出
statefulset.apps/redis-cluster created
查看
kubectl get pods -l app=redis-cluster
输出
NAME READY STATUS RESTARTS AGE
redis-cluster-0 1/1 Running 0 16h
redis-cluster-1 1/1 Running 0 16h
redis-cluster-2 1/1 Running 0 16h
redis-cluster-3 1/1 Running 0 16h
redis-cluster-4 1/1 Running 0 16h
redis-cluster-5 1/1 Running 0 16h
创建 svc
redis-svc.yml
---
apiVersion: v1
kind: Service
metadata:
name: redis-cluster
spec:
type: ClusterIP
ports:
- port: 6379
targetPort: 6379
name: client
- port: 16379
targetPort: 16379
name: gossip
selector:
app: redis-cluster
执行创建命令
kubectl apply -f redis-svc.yml
输出
service/redis-cluster created
查看 svc redis-cluster
kubectl get svc redis-cluster
输出
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis-cluster ClusterIP 10.43.216.117 <none> 6379/TCP,16379/TCP 17h
初始化 Redis Cluster
在命令执行中有Can I set the above configuration? (type 'yes' to accept):
时,要输入yes
创建集群
kubectl exec -it redis-cluster-0 -- redis-cli --cluster create --cluster-replicas 1 $(kubectl get pods -l app=redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 {end}')
输出
····
Can I set the above configuration? (type 'yes' to accept): yes
····
验证集群部署
kubectl exec -it redis-cluster-0 -- redis-cli cluster info
输出
····
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:57351
cluster_stats_messages_pong_sent:55991
cluster_stats_messages_sent:113342
cluster_stats_messages_ping_received:55986
cluster_stats_messages_pong_received:57351
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:113342
····
查看各个redis的主从状态
for x in $(seq 0 5); do echo "redis-cluster-$x"; kubectl exec redis-cluster-$x -- redis-cli role; echo; done
输出
redis-cluster-0
master
80332
10.42.0.19
6379
80332
redis-cluster-1
master
80332
10.42.0.20
6379
80332
redis-cluster-2
master
80332
10.42.0.18
6379
80332
redis-cluster-3
slave
10.42.0.17
6379
connected
80332
redis-cluster-4
slave
10.42.0.15
6379
connected
80332
redis-cluster-5
slave
10.42.0.16
6379
connected
80346
参考
https://github.com/zuxqoj/kubernetes-redis-cluster
https://github.com/llmgo/redis-sts.git
https://rancher.com/blog/2019/deploying-redis-cluster/
https://www.cnblogs.com/winstom/p/11881882.html
https://blog.csdn.net/zhutongcloud/article/details/90768390