本次部署为基于k8s部署redis一主两从三哨兵集群,本次服务器为:
host176 192.168.212.176
host177 192.168.212.177
host178 192.168.212.178
host179 192.168.212.179
1.设置nfs存储
(1) yum下载nfs,rpcbind,四台机器上都执行
yum install -y nfs-utils rpcbind
(2) 在176上执行,设置nfs共享目录,在exports文件中新增下列内容
vi /etc/exports
/k8s/redis-sentinel/0 *(rw,sync,no_subtree_check,no_root_squash)
/k8s/redis-sentinel/1 *(rw,sync,no_subtree_check,no_root_squash)
/k8s/redis-sentinel/2 *(rw,sync,no_subtree_check,no_root_squas
(3) 176节点上创建数据存储路径
mkdir -p /k8s/redis-sentinel
(4)进入目录
cd /k8s/redis-sentinel
(5) 176节点上创建数据存储路径
mkdir 0 1 2
(6) 176节点上执行,设置redis数据存储路径文件权限
chmod -R 755 /k8s/redis-sentinel
(7) 176节点上执行,使新增的nfs共享目录生效
exportfs -arv
(8) 四台机器上都执行,启动rpcbind并设置开机自启
systemctl enable rpcbind && systemctl start rpcbind
(9) 四台机器上都执行,启动nfs并设置开机自启
systemctl enable nfs && systemctl start nfs
以下操作均在主节点176节点上以root用户操作,可新建redis-sentinel目录存放下面所属yml文件,本人在此路径/opt/redis-sentinel下
2.创建pv
(1) vim redis-sentinel-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-redis-sentinel-0
spec:
capacity:
storage: 4Gi
accessModes:
- ReadWriteMany
volumeMode: Filesystem
persistentVolumeReclaimPolicy: Recycle
storageClassName: "redis-sentinel-storage-class"
nfs:
# real share directory
path: /k8s/redis-sentinel/0
# nfs real ip
server: 192.168.212.176
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-redis-sentinel-1
spec:
capacity:
storage: 4Gi
accessModes:
- ReadWriteMany
volumeMode: Filesystem
persistentVolumeReclaimPolicy: Recycle
storageClassName: "redis-sentinel-storage-class"
nfs:
# real share directory
path: /k8s/redis-sentinel/1
# nfs real ip
server: 192.168.212.176
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-redis-sentinel-2
spec:
capacity:
storage: 4Gi
accessModes:
- ReadWriteMany
volumeMode: Filesystem
persistentVolumeReclaimPolicy: Recycle
storageClassName: "redis-sentinel-storage-class"
nfs:
# real share directory
path: /k8s/redis-sentinel/2
# nfs real ip
server: 192.168.212.176
注:上述配置文件中需要将server:192.168.212.176这个ip改为自己的ip,其他一样即可
(2) 使redis-sentinel-pv.yml文件生效
kubectl apply -f redis-sentinel-pv.yaml
(3) 查看之前配置文件生成的pv卷是否创建成功
kubectl get pv | grep redis
3.创建namespace
(1) 创建命名空间publi-service
kubectl create namespace public-service
(2) 查看是否创建成功
kubectl get namespace
4.创建ConfigMap
(1) vim redis-sentinel-configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: redis-sentinel-config
namespace: public-service
labels:
addonmanager.kubernetes.io/mode: Reconcile
data:
redis-master.conf: |
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 0
loglevel notice
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /data/
slave-serve-stale-data yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
requirepass Idc&57$S6z
masterauth Idc&57$S6z
protected-mode no
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 64mb 16mb 60
hz 10
aof-rewrite-incremental-fsync yes
redis-slave.conf: |
port 6379
slaveof redis-sentinel-master-ss-0.redis-sentinel-master-ss.public-service.svc.cluster.local 6379
tcp-backlog 511
timeout 0
tcp-keepalive 0
loglevel notice
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /data/
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
requirepass Idc&57$S6z
masterauth Idc&57$S6z
protected-mode no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 64mb 16mb 60
hz 10
aof-rewrite-incremental-fsync yes
redis-sentinel.conf: |
port 26379
dir /data
protected-mode no
sentinel monitor mymaster 192.168.212.176 32379 2
sentinel down-after-milliseconds mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel auth-pass mymaster Idc&57$S6z
注:需更改redis-sentinel.conf下的 sentinel monitor mymaster 192.168.212.176 32379 2 这个设置,192.168.212.176更改为自己的ip 即可.
(2) 使redis-sentinel-configmap.yaml配置文件生效
kubectl apply -f redis-sentinel-configmap.yaml
(3) 查看configmap配置文件是否创建成功
kubectl get configmap -n public-service
5.创建service
(1) vim redis-sentinel-service-master.yaml
kind: Service
apiVersion: v1
metadata:
labels:
app: redis-sentinel-master-ss
name: redis-sentinel-master-ss
namespace: public-service
spec:
clusterIP: None
ports:
- name: redis
port: 6379
targetPort: 6379
selector:
app: redis-sentinel-master-ss
(2) vim redis-sentinel-service-slave.yaml
kind: Service
apiVersion: v1
metadata:
labels:
app: redis-sentinel-slave-ss
name: redis-sentinel-slave-ss
namespace: public-service
spec:
clusterIP: None
ports:
- name: redis
port: 6379
targetPort: 6379
selector:
app: redis-sentinel-slave-ss
(3) vim redis-sentinel-svc.yml
apiVersion: v1
kind: Service
metadata:
labels:
app: redis-sentinel-master-ss
name: redis-sentinel-master-ss-svc
namespace: public-service
spec:
externalTrafficPolicy: Cluster
ports:
- name: redis
nodePort: 32379
port: 6379
protocol: TCP
targetPort: 6379
selector:
app: redis-sentinel-master-ss
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
(4) vim sentinel-svc.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: redis-sentinel-sentinel-ss
name: redis-sentinel-sentinel-ss-svc
namespace: public-service
spec:
ports:
- name: redis
port: 26379
protocol: TCP
targetPort: 26379
selector:
app: redis-sentinel-sentinel-ss
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
(5) 使上述四个yml文件生效
kubectl apply -f redis-sentinel-service-master.yaml -f redis-sentinel-service-slave.yaml -f redis-sentinel-svc.yml -f sentinel-svc.yaml
6.创建StatefulSet
(1) vim redis-sentinel-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: redis-sentinel
namespace: public-service
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: redis-sentinel
namespace: public-service
rules:
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: redis-sentinel
namespace: public-service
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: redis-sentinel
subjects:
- kind: ServiceAccount
name: redis-sentinel
namespace: public-service
(2) vim redis-sentinel-ss-master.yaml
kind: StatefulSet
apiVersion: apps/v1
metadata:
labels:
app: redis-sentinel-master-ss
name: redis-sentinel-master-ss
namespace: public-service
spec:
replicas: 1
selector:
matchLabels:
app: redis-sentinel-master-ss
serviceName: redis-sentinel-master-ss
template:
metadata:
labels:
app: redis-sentinel-master-ss
spec:
containers:
- args:
- -c
- cp /mnt/redis-master.conf /data/ ; redis-server /data/redis-master.conf
command:
- sh
image: redis:5.0.5-alpine
imagePullPolicy: IfNotPresent
name: redis-master
ports:
- containerPort: 6379
name: masterport
protocol: TCP
volumeMounts:
- mountPath: /mnt/
name: config-volume
readOnly: false
- mountPath: /data/
name: redis-sentinel-master-storage
readOnly: false
serviceAccountName: redis-sentinel
terminationGracePeriodSeconds: 30
volumes:
- configMap:
items:
- key: redis-master.conf
path: redis-master.conf
name: redis-sentinel-config
name: config-volume
volumeClaimTemplates:
- metadata:
name: redis-sentinel-master-storage
spec:
accessModes:
- ReadWriteMany
storageClassName: "redis-sentinel-storage-class"
resources:
requests:
storage: 4Gi
(3) vim redis-sentinel-ss-slave.yaml
kind: StatefulSet
apiVersion: apps/v1
metadata:
labels:
app: redis-sentinel-slave-ss
name: redis-sentinel-slave-ss
namespace: public-service
spec:
replicas: 2
selector:
matchLabels:
app: redis-sentinel-slave-ss
serviceName: redis-sentinel-slave-ss
template:
metadata:
labels:
app: redis-sentinel-slave-ss
spec:
containers:
- args:
- -c
- cp /mnt/redis-slave.conf /data/ ; redis-server /data/redis-slave.conf
command:
- sh
image: redis:5.0.5-alpine
imagePullPolicy: IfNotPresent
name: redis-slave
ports:
- containerPort: 6379
name: slaveport
protocol: TCP
volumeMounts:
- mountPath: /mnt/
name: config-volume
readOnly: false
- mountPath: /data/
name: redis-sentinel-slave-storage
readOnly: false
serviceAccountName: redis-sentinel
terminationGracePeriodSeconds: 30
volumes:
- configMap:
items:
- key: redis-slave.conf
path: redis-slave.conf
name: redis-sentinel-config
name: config-volume
volumeClaimTemplates:
- metadata:
name: redis-sentinel-slave-storage
spec:
accessModes:
- ReadWriteMany
storageClassName: "redis-sentinel-storage-class"
resources:
requests:
storage: 4Gi
(4) 使上述三个配置文件生效
kubectl apply -f redis-sentinel-rbac.yaml -f redis-sentinel-ss-master.yaml -f redis-sentinel-ss-slave.yaml
(5) 查看命名空间public-service下各pod情况,STATUS皆为running则为正常
kubectl get pods -n public-service
(6) 进入容器并查看状态
kubectl exec -ti redis-sentinel-slave-ss-1 -n public-service – redis-cli -a ‘Idc&57$S6z’ -h redis-sentinel-master-ss-0.redis-sentinel-master-ss.public-service.svc.cluster.local info replication
至此,一主两从搭建完毕
7. 创建sentinel
(1) vim redis-sentinel-ss-sentinel.yaml
kind: StatefulSet
apiVersion: apps/v1
metadata:
labels:
app: redis-sentinel-sentinel-ss
name: redis-sentinel-sentinel-ss
namespace: public-service
spec:
replicas: 3
selector:
matchLabels:
app: redis-sentinel-sentinel-ss
serviceName: redis-sentinel-sentinel-ss
template:
metadata:
labels:
app: redis-sentinel-sentinel-ss
spec:
containers:
- args:
- -c
- cp /mnt/redis-sentinel.conf /data/ ; redis-sentinel /data/redis-sentinel.conf
command:
- sh
image: redis:5.0.5-alpine
imagePullPolicy: IfNotPresent
name: redis-sentinel
ports:
- containerPort: 26379
name: sentinel-port
protocol: TCP
volumeMounts:
- mountPath: /mnt/
name: config-volume
readOnly: false
serviceAccountName: redis-sentinel
terminationGracePeriodSeconds: 30
volumes:
- configMap:
items:
- key: redis-sentinel.conf
path: redis-sentinel.conf
name: redis-sentinel-config
name: config-volume
(2) vim redis-sentinel-service-sentinel.yaml
kind: Service
apiVersion: v1
metadata:
labels:
app: redis-sentinel-sentinel-ss
name: redis-sentinel-sentinel-ss
namespace: public-service
spec:
clusterIP: None
ports:
- name: redis
port: 26379
targetPort: 26379
selector:
app: redis-sentinel-sentinel-ss
(3) kubectl apply -f redis-sentinel-ss-sentinel.yaml -f redis-sentinel-service-sentinel.yaml
(4) 查看命名空间public-service下各个服务状态
kubectl get service -n public-service
至此哨兵搭建完成.之后用代码检验.
参考文档: https://www.cnblogs.com/dukuan/p/9913420.html
单节点部署参考: https://zhuanlan.zhihu.com/p/160959892 如果镜像拉取报错,可添加拉取策略IfNotPresent