Kubernetes的有状态应用示例：使用StatefulSet部署Cassandra

环境

• RHEL 9.3
• Docker Community 24.0.7
• minikube v1.32.0

概述

本例展示了如何在Kubernetes上运行Apache Cassandra。Cassandra是一个数据库，需要永久性存储来提供数据持久性（应用状态）。在此示例中，自定义的Cassandra种子提供者使得数据库在新的Cassandra实例接入Cassandra集群时能够发现它们。

注意：Cassandra和Kubernetes都使用术语 node 来表示集群的成员。在本例中，StatefulSet的pod是Cassandra node，并且是Cassandra集群的成员（称为 ring ）。当这些pod在Kubernetes集群中运行时，Kubernetes control plane会把这些pod调度到Kubernetes的node上。

当 Cassandra node启动时，使用 seed 列表来引导发现ring中的其它node。本例部署了一个自定义的Cassandra种子提供者，使得数据库在新的Cassandra pod出现在Kubernetes集群中时，可以发现它们。

准备

Minikube默认需要2048MB内存和2个CPU。本例中使用默认资源配置运行Minikube会出现资源不足的错误。为避免这些错误，请使用以下设置启动Minikube：

$ minikube start --memory 5120 --cpus=4
😄  minikube v1.32.0 on Redhat 9.3
✨  Using the docker driver based on existing profile
❗  You cannot change the memory size for an existing minikube cluster. Please first delete the cluster.
❗  You cannot change the CPUs for an existing minikube cluster. Please first delete the cluster.
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

注意这两行：

❗  You cannot change the memory size for an existing minikube cluster. Please first delete the cluster.
❗  You cannot change the CPUs for an existing minikube cluster. Please first delete the cluster.

注：如果不管，按默认设置启动minikube，则在后面的步骤会报错，如：

$ kubectl get pods -l="app=cassandra"
Unable to connect to the server: http2: client connection lost

$ kubectl get nodes
Unable to connect to the server: net/http: TLS handshake timeout

要改变设置，貌似需要先删除cluster才行：

$ minikube delete
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /home/ding/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.

然后再运行：

$ minikube start --memory 5120 --cpus=4
😄  minikube v1.32.0 on Redhat 9.3
✨  Automatically selected the docker driver
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
❗  minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.42, but successfully downloaded docker.io/kicbase/stable:v0.0.42 as a fallback image
🔥  Creating docker container (CPUs=4, Memory=5120MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

为Cassandra创建headless service

以下 Service 用于在Cassandra pod和客户端之间进行DNS查找：

创建文件 cassandra-service.yaml 如下：

apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  clusterIP: None
  ports:
  - port: 9042
  selector:
    app: cassandra

kubectl apply -f cassandra-service.yaml

$ kubectl get svc cassandra
NAME        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
cassandra   ClusterIP   None         <none>        9042/TCP   24s

创建文件 cassandra-statefulset.yaml 如下：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra
  labels:
    app: cassandra
spec:
  serviceName: cassandra
  replicas: 3
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra
        # image: gcr.io/google-samples/cassandra:v13
        image: docker.io/kaiding1/cassandra:v13
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: 
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.default.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "K8Demo"
          - name: CASSANDRA_DC
            value: "DC1-K8Demo"
          - name: CASSANDRA_RACK
            value: "Rack1-K8Demo"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - /ready-probe.sh
          initialDelaySeconds: 15
          timeoutSeconds: 5
        # These volume mounts are persistent. They are like inline claims,
        # but not exactly because the names need to match exactly one of
        # the stateful pod volumes.
        volumeMounts:
        - name: cassandra-data
          mountPath: /cassandra_data
  # These are converted to volume claims by the controller
  # and mounted at the paths mentioned above.
  # do not use these in production until ssd GCEPersistentDisk or other ssd pd
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: fast
      resources:
        requests:
          storage: 1Gi
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fast
provisioner: k8s.io/minikube-hostpath
parameters:
  type: pd-ssd

注：因为访问不了 gcr.io ，所以事先把image pull下来，并push到了可访问的位置。

$ kubectl apply -f cassandra-statefulset.yaml
statefulset.apps/cassandra created
storageclass.storage.k8s.io/fast created

验证Cassandra StatefulSet

注：启动pod特别慢，本例中花了半个多小时。

$ kubectl get statefulset cassandra
NAME        READY   AGE
cassandra   3/3     53m

$ kubectl get pods -l="app=cassandra"
NAME          READY   STATUS    RESTARTS   AGE
cassandra-0   1/1     Running   0          53m
cassandra-1   1/1     Running   0          47m
cassandra-2   1/1     Running   0          37m

$ kubectl exec -it cassandra-0 -- nodetool status
Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.244.0.4  142.7 KiB  32           72.8%             913518a2-c96e-4915-9b95-dc26c8c4e9a7  Rack1-K8Demo
UN  10.244.0.5  143.1 KiB  32           70.6%             e5689d71-a4b9-4a20-af0b-147ea2ec4c57  Rack1-K8Demo
UN  10.244.0.3  135.94 KiB  32           56.6%             af529f81-7da7-4129-b379-d41554390c6f  Rack1-K8Demo

修改Cassandra StatefulSet

kubectl edit statefulset cassandra

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: 2016-08-13T18:40:58Z
  generation: 1
  labels:
  app: cassandra
  name: cassandra
  namespace: default
  resourceVersion: "323"
  uid: 7a219483-6185-11e6-a910-42010a8a0fc0
spec:
  replicas: 3

把replica数量从3改为4，然后保存退出。

$ kubectl edit statefulset cassandra
statefulset.apps/cassandra edited

大约5分钟后，查看状态：

$ kubectl get pods -l="app=cassandra"
NAME          READY   STATUS    RESTARTS   AGE
cassandra-0   1/1     Running   0          101m
cassandra-1   1/1     Running   0          96m
cassandra-2   1/1     Running   0          85m
cassandra-3   1/1     Running   0          5m

$ kubectl get statefulset cassandra
NAME        READY   AGE
cassandra   4/4     101m

清理

删除或缩小StatefulSet不会删除与之关联的volume。这是出于安全考虑，因为与自动清除所有相关的StatefulSet资源相比，数据更有价值。

注意：根据Storage Class和回收策略，删除PVC可能导致相关联的volume也被删除。不要想当然认为PVC被删除后，还能访问数据。

$ grace=$(kubectl get pod cassandra-0 -o=jsonpath='{.spec.terminationGracePeriodSeconds}') \
  && kubectl delete statefulset -l app=cassandra \
  && echo "Sleeping ${grace} seconds" 1>&2 \
  && sleep $grace \
  && kubectl delete persistentvolumeclaim -l app=cassandra
statefulset.apps "cassandra" deleted
Sleeping 1800 seconds
persistentvolumeclaim "cassandra-data-cassandra-0" deleted
persistentvolumeclaim "cassandra-data-cassandra-1" deleted
persistentvolumeclaim "cassandra-data-cassandra-2" deleted
persistentvolumeclaim "cassandra-data-cassandra-3" deleted

$ kubectl delete service -l app=cassandra
service "cassandra" deleted

Cassandra容器环境变量

本例中的pod 使用来自Google容器registry的 gcr.io/google-samples/cassandra:v13 image。该Docker image基于debian-base，且包含 OpenJDK 8。

注：因为访问不了 gcr.io ，所以事先把image pull下来，并push到了可访问的位置。

该镜像包括Apache Debian repo的标准Cassandra安装。通过环境变量，你可以更改插入到 cassandra.yaml 中的值。

环境变量	缺省值
CASSANDRA_CLUSTER_NAME	'Test Cluster'
CASSANDRA_NUM_TOKENS	32
CASSANDRA_RPC_ADDRESS	0.0.0.0

参考

• https://kubernetes.io/docs/tutorials/stateful-application/cassandra