K8S 上部署 Redis

小强签名设计

已于 2024-09-19 14:17:06 修改

阅读量490

点赞数 1

分类专栏：容器化技术文章标签： kubernetes 大数据容器

于 2024-07-08 16:16:17 首次发布

本文链接：https://blog.csdn.net/m0_37739193/article/details/140271802

版权

容器化技术专栏收录该内容

5 篇文章 0 订阅

订阅专栏

文章目录

一、前言

Artifact Hub 是一个专注于云原生应用的集中式搜索和发布平台。它旨在简化开发者在 CNCF（Cloud Native Computing Foundation）项目中寻找、安装和分享包与配置的过程。用户可以通过这个平台方便地发现、安装各类云原生工具的组件，如Argo模板、Helm 图表、Kubernetes 插件等。

在这里插入图片描述

二、Redis

Redis 有三种集群模式：主从模式，Sentinel（哨兵）模式，Cluster 模式，我这里目前安装的是 主从模式。想了解更多关于 redis 概念与原理介绍，可参考这篇文章：Redis原理介绍与环境部署（主从模式、哨兵模式、集群模式）

$ helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories

$ kubectl get sc
NAME                  PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
nfs                   nfs-provisioner-01   Retain          Immediate              false                  3d10h

2.1 安装方式一：

# 执行安装命令并执行参数存储类的名称，得确保你集群中有这个存储类并且能够正常的提供PV动态供给
$ helm install redis-cluster oci://registry-1.docker.io/bitnamicharts/redis --set global.storageClass=nfs
Pulled: registry-1.docker.io/bitnamicharts/redis:19.6.1
Digest: sha256:0d077ee5947e26645e3bc05e3d6e6bd62e24e3082cf5df43d89664099336a78d
NAME: redis-cluster
LAST DEPLOYED: Mon Jul  8 03:42:57 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis
CHART VERSION: 19.6.1
APP VERSION: 7.2.5

** Please be patient while the chart is being deployed **

Redis&reg; can be accessed on the following DNS names from within your cluster:

    redis-cluster-master.default.svc.cluster.local for read/write operations (port 6379)
    redis-cluster-replicas.default.svc.cluster.local for read-only operations (port 6379)



To get your password run:

    export REDIS_PASSWORD=$(kubectl get secret --namespace default redis-cluster -o jsonpath="{.data.redis-password}" | base64 -d)

To connect to your Redis&reg; server:

1. Run a Redis&reg; pod that you can use as a client:

   kubectl run --namespace default redis-client --restart='Never'  --env REDIS_PASSWORD=$REDIS_PASSWORD  --image docker.io/bitnami/redis:7.2.5-debian-12-r2 --command -- sleep infinity

   Use the following command to attach to the pod:

   kubectl exec --tty -i redis-client \
   --namespace default -- bash

2. Connect using the Redis&reg; CLI:
   REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h redis-cluster-master
   REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h redis-cluster-replicas

To connect to your database from outside the cluster execute the following commands:

    kubectl port-forward --namespace default svc/redis-cluster-master 6379:6379 &
    REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h 127.0.0.1 -p 6379

WARNING: There are "resources" sections in the chart not set. Using "resourcesPreset" is not recommended for production. For production installations, please set the following values according to your workload needs:
  - replica.resources
  - master.resources
+info https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

$ kubectl get pods
NAME                       READY   STATUS              RESTARTS   AGE
redis-cluster-master-0     1/1     Running             0          8m44s
redis-cluster-replicas-0   1/1     Running             0          8m44s
redis-cluster-replicas-1   1/1     Running             0          6m43s
redis-cluster-replicas-2   0/1     ContainerCreating   0          4m50s

# 将密码保存到 REDIS_PASSWORD 这个环境变量中
$ export REDIS_PASSWORD=$(kubectl get secret --namespace default redis-cluster -o jsonpath="{.data.redis-password}" | base64 -d)

# 再创建一个 Redis 客户端
$ kubectl run --namespace default redis-client --restart='Never'  --env REDIS_PASSWORD=$REDIS_PASSWORD  --image docker.io/bitnami/redis:7.2.5-debian-12-r2 --command -- sleep infinity
pod/redis-client created

$ kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
redis-client               1/1     Running   0          28s
redis-cluster-master-0     1/1     Running   0          10m
redis-cluster-replicas-0   1/1     Running   0          10m
redis-cluster-replicas-1   1/1     Running   0          8m50s
redis-cluster-replicas-2   1/1     Running   0          6m57s

$ kubectl exec -it redis-client -- bash
I have no name!@redis-client:/$ REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h redis-cluster-master
redis-cluster-master:6379> info replication
# Replication
role:master
connected_slaves:3
slave0:ip=redis-cluster-replicas-0.redis-cluster-headless.default.svc.cluster.local,port=6379,state=online,offset=924,lag=1
slave1:ip=redis-cluster-replicas-1.redis-cluster-headless.default.svc.cluster.local,port=6379,state=online,offset=924,lag=1
slave2:ip=redis-cluster-replicas-2.redis-cluster-headless.default.svc.cluster.local,port=6379,state=online,offset=924,lag=1
master_failover_state:no-failover
master_replid:239753ae516fad9e7fb6230d053861b80bc0bbb3
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:924
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:924

$ kubectl get svc
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                               AGE
kubernetes               ClusterIP   10.68.0.1       <none>        443/TCP                               3d12h
redis-cluster-headless   ClusterIP   None            <none>        6379/TCP                              16m
redis-cluster-master     ClusterIP   10.68.240.7     <none>        6379/TCP                              16m
redis-cluster-replicas   ClusterIP   10.68.170.149   <none>        6379/TCP                              16m

改为暴露端口从外部访问：

$ kubectl edit svc redis-cluster-master
# 原先内容
  ports:
  - name: tcp-redis
    port: 6379
    protocol: TCP
    targetPort: redis
  selector:
    app.kubernetes.io/component: master
    app.kubernetes.io/instance: redis-cluster
    app.kubernetes.io/name: redis
  sessionAffinity: None
  type: ClusterIP
# 修改为
  ports:
  - name: tcp-redis
    nodePort: 31379
    port: 6379
    protocol: TCP
    targetPort: redis
  selector:
    app.kubernetes.io/component: master
    app.kubernetes.io/instance: redis-cluster
    app.kubernetes.io/name: redis
  sessionAffinity: None
  type: NodePort
# 注意：nodePort 值不在指定范围会报错而无法保存（比如设置成 11379）：
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
# services "redis-cluster-master" was not valid:
# * spec.ports[0].nodePort: Invalid value: 11379: provided port is not in the valid range. The range of valid ports is 30000-32767

# 重启服务（删除之后会自动重启）
$ kubectl delete svc redis-cluster-master -n default

$ kubectl get svc
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                               AGE
kubernetes               ClusterIP   10.68.0.1       <none>        443/TCP                               4d11h
redis-cluster-headless   ClusterIP   None            <none>        6379/TCP                              22h
redis-cluster-master     NodePort    10.68.240.7     <none>        6379:31379/TCP                        22h
redis-cluster-replicas   ClusterIP   10.68.170.149   <none>        6379/TCP                              22h

补充：安装时参数自定义

# 安装的时候增加 namespace 的创建
$ helm install redis-cluster oci://registry-1.docker.io/bitnamicharts/redis -n redis --create-namespace --set global.storageClass=nfs

# redis 默认会安装1个master，3个node，可以通过以下参数来修改 --set replica.replicaCount=2 --set master.count=1
$ helm install --set replica.persistence.size=2Gi --set master.persistence.size=1Gi \
 --set global.storageClass=manual  --set  replica.replicaCount=2 --set master.count=1 linkage-redis bitnami/redis
 
# redis-cluster 默认创建6个 nodes（每个nodes包括一个master及一个replica），可以调整参数该边node数目，但调整后的nodes数不能<3
$ helm install --set cluster.nodes=3 --set replica.persistence.size=2Gi --set master.persistence.size=2Gi \
 --set global.storageClass=manual  linkage-redis bitnami/redis-cluster

参考这篇文章搭建出来的 pods 一直处于 pending 状态：k8s 如何访问redis集群

2.2 安装方式二：

$ helm pull bitnami/redis --version 17.3.7
$ tar -zxvf redis-17.3.7.tgz
redis/Chart.yaml
redis/Chart.lock
redis/values.yaml
redis/values.schema.json
redis/templates/NOTES.txt
redis/templates/_helpers.tpl
redis/templates/configmap.yaml
redis/templates/extra-list.yaml
redis/templates/headless-svc.yaml
redis/templates/health-configmap.yaml
redis/templates/master/application.yaml
redis/templates/master/psp.yaml
redis/templates/master/pvc.yaml
redis/templates/master/service.yaml
redis/templates/master/serviceaccount.yaml
redis/templates/metrics-svc.yaml
redis/templates/networkpolicy.yaml
redis/templates/pdb.yaml
redis/templates/prometheusrule.yaml
redis/templates/replicas/hpa.yaml
redis/templates/replicas/service.yaml
redis/templates/replicas/serviceaccount.yaml
redis/templates/replicas/statefulset.yaml
redis/templates/role.yaml
redis/templates/rolebinding.yaml
redis/templates/scripts-configmap.yaml
redis/templates/secret.yaml
redis/templates/sentinel/hpa.yaml
redis/templates/sentinel/node-services.yaml
redis/templates/sentinel/ports-configmap.yaml
redis/templates/sentinel/service.yaml
redis/templates/sentinel/statefulset.yaml
redis/templates/serviceaccount.yaml
redis/templates/servicemonitor.yaml
redis/templates/tls-secret.yaml
redis/.helmignore
redis/README.md
redis/img/redis-cluster-topology.png
redis/img/redis-topology.png
redis/charts/common/Chart.yaml
redis/charts/common/values.yaml
redis/charts/common/templates/_affinities.tpl
redis/charts/common/templates/_capabilities.tpl
redis/charts/common/templates/_errors.tpl
redis/charts/common/templates/_images.tpl
redis/charts/common/templates/_ingress.tpl
redis/charts/common/templates/_labels.tpl
redis/charts/common/templates/_names.tpl
redis/charts/common/templates/_secrets.tpl
redis/charts/common/templates/_storage.tpl
redis/charts/common/templates/_tplvalues.tpl
redis/charts/common/templates/_utils.tpl
redis/charts/common/templates/_warnings.tpl
redis/charts/common/templates/validations/_cassandra.tpl
redis/charts/common/templates/validations/_mariadb.tpl
redis/charts/common/templates/validations/_mongodb.tpl
redis/charts/common/templates/validations/_mysql.tpl
redis/charts/common/templates/validations/_postgresql.tpl
redis/charts/common/templates/validations/_redis.tpl
redis/charts/common/templates/validations/_validations.tpl
redis/charts/common/.helmignore
redis/charts/common/README.md

$ helm install redis ./redis -n redis --create-namespace --set global.storageClass=nfs
NAME: redis
LAST DEPLOYED: Tue Jul  9 11:18:15 2024
NAMESPACE: redis
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis
CHART VERSION: 17.3.7
APP VERSION: 7.0.5

** Please be patient while the chart is being deployed **

Redis&reg; can be accessed on the following DNS names from within your cluster:

    redis-master.redis.svc.cluster.local for read/write operations (port 6379)
    redis-replicas.redis.svc.cluster.local for read-only operations (port 6379)



To get your password run:

    export REDIS_PASSWORD=$(kubectl get secret --namespace redis redis -o jsonpath="{.data.redis-password}" | base64 -d)

To connect to your Redis&reg; server:

1. Run a Redis&reg; pod that you can use as a client:

   kubectl run --namespace redis redis-client --restart='Never'  --env REDIS_PASSWORD=$REDIS_PASSWORD  --image docker.io/bitnami/redis:7.0.5-debian-11-r7 --command -- sleep infinity

   Use the following command to attach to the pod:

   kubectl exec --tty -i redis-client \
   --namespace redis -- bash

2. Connect using the Redis&reg; CLI:
   REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h redis-master
   REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h redis-replicas

To connect to your database from outside the cluster execute the following commands:

    kubectl port-forward --namespace redis svc/redis-master 6379:6379 &
    REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h 127.0.0.1 -p 6379

$ kubectl get pods,svc -n redis
NAME                   READY   STATUS              RESTARTS   AGE
pod/redis-master-0     0/1     ContainerCreating   0          53s
pod/redis-replicas-0   0/1     ContainerCreating   0          53s

NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/redis-headless   ClusterIP   None            <none>        6379/TCP   53s
service/redis-master     ClusterIP   10.68.242.183   <none>        6379/TCP   53s
service/redis-replicas   ClusterIP   10.68.122.142   <none>        6379/TCP   53s

$ kubectl get pods -n redis
NAME                   READY   STATUS    RESTARTS   AGE
pod/redis-master-0     1/1     Running   0          99s
pod/redis-replicas-0   0/1     Running   0          99s

$ kubectl get pods -n redis
NAME                   READY   STATUS    RESTARTS   AGE
pod/redis-master-0     1/1     Running             0          2m3s
pod/redis-replicas-0   1/1     Running             0          2m3s
pod/redis-replicas-1   0/1     ContainerCreating   0          15s

$ kubectl get pods -n redis
NAME                   READY   STATUS    RESTARTS   AGE
pod/redis-master-0     1/1     Running   0          4m27s
pod/redis-replicas-0   1/1     Running   0          4m27s
pod/redis-replicas-1   1/1     Running   0          2m39s
pod/redis-replicas-2   0/1     Running   0          82s

$ kubectl describe pod redis-master-0 -n redis
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  9m31s  default-scheduler  0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod..
  Normal   Scheduled         9m29s  default-scheduler  Successfully assigned redis/redis-master-0 to worker-01
  Normal   Pulling           9m28s  kubelet            Pulling image "docker.io/bitnami/redis:7.0.5-debian-11-r7"
  Normal   Pulled            8m17s  kubelet            Successfully pulled image "docker.io/bitnami/redis:7.0.5-debian-11-r7" in 1m11.811062916s (1m11.811075708s including waiting)
  Normal   Created           8m17s  kubelet            Created container redis
  Normal   Started           8m16s  kubelet            Started container redis

卸载：

$ helm uninstall redis -n redis
release "redis" uninstalled

# delete ns 
$ kubectl delete ns redis --force
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
namespace "redis" force deleted

# delete pv
kubectl delete pv `kubectl get pv|grep ^redis|awk '{print $1}'` --force

2.3 使用本地存储安装：

参考：【云原生】Redis on k8s 编排部署讲解与实战操作

下载 chart 包：

$ helm repo add bitnami https://charts.bitnami.com/bitnami

$ helm pull bitnami/redis --version 17.3.7

$ tar -xf redis-17.3.7.tgz

构建镜像（非必须，可以提前下载好镜像，可能会因为下载镜像失败而安装失败。）：这里就不重新构建镜像了，只是把远程镜像 tag 一下，推到本地 harbor 仓库加速下载镜像。

$ docker pull docker.io/bitnami/redis:7.0.5-debian-11-r7

# tag
$ docker tag docker.io/bitnami/redis:7.0.5-debian-11-r7 myharbor.com/bigdata/redis:7.0.5-debian-11-r7

# 推送镜像到本地harbor仓库
$ docker push myharbor.com/bigdata/redis:7.0.5-debian-11-r7

修改 yaml 编排：

新增 redis/templates/master/pv.yaml 文件，内容如下：

{{- range .Values.master.persistence.local }}
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: {{ .name }}
  labels:
    name: {{ .name }}
spec:
  storageClassName: {{ $.Values.master.persistence.storageClass }}
  capacity:
    storage: {{ $.Values.master.persistence.size }}
  accessModes:
    - ReadWriteOnce
  local:
    path: {{ .path }}
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - {{ .host }}
---
{{- end }}

新增 redis/templates/replicas/pv.yaml 文件，内容如下：

{{- range .Values.replica.persistence.local }}
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: {{ .name }}
  labels:
    name: {{ .name }}
spec:
  storageClassName: {{ $.Values.replica.persistence.storageClass }}
  capacity:
    storage: {{ $.Values.replica.persistence.size }}
  accessModes:
    - ReadWriteOnce
  local:
    path: {{ .path }}
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - {{ .host }}
---
{{- end }}

修改 redis/values.yaml ：

global:
  redis:
    password: "123456"

...

image:
  registry: myharbor.com
  repository: bigdata/redis
  tag: 7.0.5-debian-11-r7

master:
  count: 1
  persistence:
    enabled: true
    size: 8Gi
    storageClass: "local-redis-storage"
    local:
    - name: redis-0
      host: "master01"
      path: "/dataspace1/redis-data/redis"

replica:
  replicaCount: 2
  persistence:
    enabled: true
    size: 8Gi
    storageClass: "local-redis-storage"
    local:
    - name: redis-1
      host: "node01"
      path: "/dataspace1/redis-data/redis"
    - name: redis-2
      host: "node02"
      path: "/dataspace1/redis-data/redis"

创建本地存储。一开始参考的是这篇文章并没有成功在K8S中使用Local持久卷，不知道是我哪里整错了，等有时间可以再研究下这篇文章：StorageClass-动态PVC 和 helm或者k8s部署pod时遇到pod一直处于pending状态

$ vim local-redis-storage-bak.yml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-redis-storage
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

$ kubectl create -f local-redis-storage-bak.yml

参考：K8s 部署 MySQL 8.0.20 主从复制结构可以成功。本次使用 OpenEBS 来作为存储引擎，OpenEBS 是一个开源的、可扩展的存储平台，它提供了一种简单的方式来创建和管理持久化存储卷。它支持各种存储后端，包括但不限于 ZFS、Btrfs、XFS 等。同时，OpenEBS 具有高度的可扩展性和可配置性，可以满足不同的存储需求。

安装 OpenEBS ：

$ kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml

验证 OpenEBS 是否正确安装：

在这里插入图片描述
创建命名空间：

$ vim local-redis-storage.yml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-redis-storage
  annotations:
    openebs.io/cas-type: local
    cas.openebs.io/config: |
      - name: StorageType
        value: hostpath
      - name: BasePath
        value: /dataspace1/redis-data/redis
provisioner: openebs.io/local
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

$ kubectl apply -f local-redis-storage.yml
storageclass.storage.k8s.io/local-redis-storage created

$ kubectl get sc
NAME                  PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-redis-storage   kubernetes.io/no-provisioner   Retain          WaitForFirstConsumer   false                  30h

开始部署：

# 先检查语法
$ helm lint ./redis

# 开始安装
$ helm install redis ./redis -n redis --create-namespace

测试验证：

$ kubectl get pods,svc -n redis -owide
NAME                   READY   STATUS    RESTARTS   AGE   IP              NODE        NOMINATED NODE   READINESS GATES
pod/redis-master-0     1/1     Running   0          86s   172.20.85.242   worker-01   <none>           <none>
pod/redis-replicas-0   1/1     Running   0          86s   172.20.58.226   worker-02   <none>           <none>
pod/redis-replicas-1   1/1     Running   0          47s   172.20.32.151   master-01   <none>           <none>

NAME                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE   SELECTOR
service/redis-headless   ClusterIP   None           <none>        6379/TCP   86s   app.kubernetes.io/instance=redis,app.kubernetes.io/name=redis
service/redis-master     ClusterIP   10.68.8.36     <none>        6379/TCP   86s   app.kubernetes.io/component=master,app.kubernetes.io/instance=redis,app.kubernetes.io/name=redis
service/redis-replicas   ClusterIP   10.68.91.126   <none>        6379/TCP   86s   app.kubernetes.io/component=replica,app.kubernetes.io/instance=redis,app.kubernetes.io/name=redis

root@bigdata-master01:/home/bigdata# ll /dataspace1/redis-data/redis/
total 0
drwxr-xr-x 4 root root 102 Jul 12 10:27 ./
drwxr-xr-x 3 root root  19 Jul 12 10:19 ../
drwxrwsrwx 3 root 1001  43 Jul 12 10:27 pvc-0a70e76e-b638-4a12-8291-658ae80c0431/
drwxrwxrwx 2 root root   6 Jul 12 10:19 pvc-9ac87175-8c45-4f63-bafa-16f9d8d704c7/

root@bigdata-node01:/home/bigdata# ll /dataspace1/redis-data/redis/
total 0
drwxr-xr-x 4 root root 102 Jul 12 10:26 ./
drwxr-xr-x 3 root root  19 Jul 12 10:17 ../
drwxrwsrwx 3 root 1001  27 Jul 12 10:27 pvc-0d1e3c61-2a1c-4d29-9ad9-4b75c6f908b4/
drwxrwsrwx 3 root 1001  43 Jul 12 10:18 pvc-80697195-5346-4e05-9607-cc1b4a2e59fa/

# 登录master，可读可写
$ kubectl exec -it redis-master-0 -n redis -- redis-cli -h redis-master -a $(kubectl get secret --namespace redis redis -o jsonpath="{.data.redis-password}" | base64 -d)

# 登录slave，只读
$ kubectl exec -it redis-master-0 -n redis -- redis-cli -h redis-replicas -a $(kubectl get secret --namespace redis redis -o jsonpath="{.data.redis-password}" | base64 -d)

在这里插入图片描述
实测可以执行的命令（上面的配置完事后其实一条命令指定参数就可以安装）：

helm install my-redis ./redis -n redis --create-namespace --set global.storageClass=local-redis-storage --set master.service.type=NodePort --set master.service.nodePorts.redis=31379 --set global.redis.password=123456