目录
1.创建configmap
前置条件:创建pv,我这里使用的是nfs,做sc
cat redis-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-conf
namespace: redis
data:
redis.conf: |
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 10000
protected-mode no
daemonize no
pidfile /var/run/redis.pid
port 6379
tcp-backlog 511
bind 0.0.0.0
timeout 3600
tcp-keepalive 1
loglevel verbose
logfile /data/redis.log
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /data
requirepass lishanbin123456
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 20000
slowlog-log-slower-than 10000
slowlog-max-len 128
#rename-command FLUSHALL ""
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
cluster-announce-port 6379
cluster-announce-bus-port 16379
2.创建svc
apiVersion: v1
kind: Service
metadata:
name: redis-service
labels:
app: redis
spec:
ports:
- name: redis-port
port: 6379
clusterIP: None
selector:
app: redis
appCluster: redis-cluster
3.创建statefulset
cat redis-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
spec:
serviceName: "redis-service"
selector:
matchLabels:
app: redis
replicas: 6
template:
metadata:
labels:
app: redis
appCluster: redis-cluster
spec:
terminationGracePeriodSeconds: 20
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- redis
topologyKey: kubernetes.io/hostname
containers:
- name: redis
image: "redis:6.2.6"
imagePullPolicy: IfNotPresent
command:
- "redis-server" #redis启动命令
args:
- "/etc/redis/redis.conf" #redis-server后面跟的参数,换行代表空格
- "--protected-mode" #允许外网访问
- "no"
# command: redis-server /etc/redis/redis.conf --protected-mode no
resources: #资源
requests: #请求的资源
cpu: "100m" #m代表千分之,相当于0.1 个cpu资源
memory: "100Mi" #内存100m大小
ports:
- name: redis
containerPort: 6379
protocol: "TCP"
- name: cluster
containerPort: 16379
protocol: "TCP"
volumeMounts:
- name: "redis-conf" #挂载configmap生成的文件
mountPath: "/etc/redis" #挂载到哪个路径下
- name: "redis-data" #挂载持久卷的路径
mountPath: "/var/lib/redis"
volumes:
- name: "redis-conf" #引用configMap卷
configMap:
name: "redis-conf"
items:
- key: "redis.conf" #创建configMap指定的名称
path: "redis.conf" #里面的那个文件--from-file参数后面的文件
volumeClaimTemplates: #进行pvc持久卷声明,
- metadata:
name: redis-data
spec:
accessModes: ["ReadWriteMany"]
storageClassName: "managed-nfs-storage"
resources:
requests:
storage: 10G
4.集群初始化
方式一:使用redis-trib
kubectl run -it ubuntu --image=registry.cn-shenzhen.aliyuncs.com/lishanbin/redis-tools-ubuntu:v0.5.1 --restart=Never /bin/bash
redis-trib.py create \
`dig +short redis-0.redis-service.redis.svc.cluster.local`:6379 \
`dig +short redis-1.redis-service.redis.svc.cluster.local`:6379 \
`dig +short redis-2.redis-service.redis.svc.cluster.local`:6379
redis-trib.py replicate \
--master-addr `dig +short redis-0.redis-service.redis.svc.cluster.local`:6379 \
--slave-addr `dig +short redis-3.redis-service.redis.svc.cluster.local`:6379
redis-trib.py replicate \
--master-addr `dig +short redis-1.redis-service.redis.svc.cluster.local`:6379 \
--slave-addr `dig +short redis-4.redis-service.redis.svc.cluster.local`:6379
redis-trib.py replicate \
--master-addr `dig +short redis-2.redis-service.redis.svc.cluster.local`:6379 \
--slave-addr `dig +short redis-5.redis-service.redis.svc.cluster.local`:6379
方式二:使用ip,重启pod,ip会变,redis集群状态使用的是nodeid,不影响
kubectl get pods -l app=redis -n redis -o jsonpath='{range.items[*]}{.status.podIP}:6379 '
kubectl exec -it redis-0 -n redis bash
redis-cli --cluster create --cluster-replicas 1 10.203.36.212:6379 10.203.216.177:6379 10.203.191.5:6379 10.203.150.121:6379 10.203.36.204:6379 10.203.216.178:6379 -a lishanbin123456
出现下图表示成功
5.测试验证
6.遗留问题思考
方案只能提供给k8s集群内的应用使用,对集群外的应用根本用不了,因为一旦涉及到move命令,redis节点只会给出内部的pod ip,这个使得集群外的应用根本连不上,因为这个涉及到redis的源码,redis集群节点的相互通讯使用的redis进程所在的环境的ip,而这个ip就是pod ip,相对的节点发送给客户端的move的ip也是pod ip
nodeport可以让外部机器连上集群,但是redis集群是客户端重定向机制,在set key的时候,会不断的计算slot,并重定向对应的master机器上,此时外部机器redis客户端连接的ip会被强制重定向到slot计算定位的节点机器上,但是这个定位后的ip是k8s集群内部分配的pod ip,仅限于k8s集群内部node访问,这个外部机器压根连不上这个重定向后的节点;所以问题就出来了,等同于外部没法使用k8s集群内的redis集群,所以如何解决这个问题?或者说,k8s集群内部的redis服务到底该不该对外(k8s集群外部)提供访问,还是说要在k8s集群内部开个对外提供数据访问的中心服务来解耦这种直连形式,达到redis只内部运行而不涉及外连;
待验证:
单个节点故障可以恢复。但是全部节点都down之后是不行,这个场景我在其他的文章中了解说使用initcontainer,把pod的ip替换到nodes.conf里面自己的新ip可以解决。但是我测试了,全部伸缩到0之后,重新伸缩到6,集群状态还是不能恢复。。