文章目录
环境
- RHEL 9.3
- Docker Community 24.0.7
- minikube v1.32.0
概述
本例展示了如何在Kubernetes上运行Apache Cassandra。Cassandra是一个数据库,需要永久性存储来提供数据持久性(应用状态)。在此示例中,自定义的Cassandra种子提供者使得数据库在新的Cassandra实例接入Cassandra集群时能够发现它们。
注意:Cassandra和Kubernetes都使用术语 node
来表示集群的成员。在本例中,StatefulSet的pod是Cassandra node,并且是Cassandra集群的成员(称为 ring
)。当这些pod在Kubernetes集群中运行时,Kubernetes control plane会把这些pod调度到Kubernetes的node上。
当 Cassandra node启动时,使用 seed
列表来引导发现ring中的其它node。本例部署了一个自定义的Cassandra种子提供者,使得数据库在新的Cassandra pod出现在Kubernetes集群中时,可以发现它们。
准备
Minikube默认需要2048MB内存和2个CPU。本例中使用默认资源配置运行Minikube会出现资源不足的错误。为避免这些错误,请使用以下设置启动Minikube:
$ minikube start --memory 5120 --cpus=4
😄 minikube v1.32.0 on Redhat 9.3
✨ Using the docker driver based on existing profile
❗ You cannot change the memory size for an existing minikube cluster. Please first delete the cluster.
❗ You cannot change the CPUs for an existing minikube cluster. Please first delete the cluster.
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🔄 Restarting existing docker container for "minikube" ...
🐳 Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
🔗 Configuring bridge CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
注意这两行:
❗ You cannot change the memory size for an existing minikube cluster. Please first delete the cluster.
❗ You cannot change the CPUs for an existing minikube cluster. Please first delete the cluster.
注:如果不管,按默认设置启动minikube,则在后面的步骤会报错,如:
$ kubectl get pods -l="app=cassandra"
Unable to connect to the server: http2: client connection lost
$ kubectl get nodes
Unable to connect to the server: net/http: TLS handshake timeout
要改变设置,貌似需要先删除cluster才行:
$ minikube delete
🔥 Deleting "minikube" in docker ...
🔥 Deleting container "minikube" ...
🔥 Removing /home/ding/.minikube/machines/minikube ...
💀 Removed all traces of the "minikube" cluster.
然后再运行:
$ minikube start --memory 5120 --cpus=4
😄 minikube v1.32.0 on Redhat 9.3
✨ Automatically selected the docker driver
📌 Using Docker driver with root privileges
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
❗ minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.42, but successfully downloaded docker.io/kicbase/stable:v0.0.42 as a fallback image
🔥 Creating docker container (CPUs=4, Memory=5120MB) ...
🐳 Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring bridge CNI (Container Networking Interface) ...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎 Verifying Kubernetes components...
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
为Cassandra创建headless service
以下 Service 用于在Cassandra pod和客户端之间进行DNS查找:
创建文件 cassandra-service.yaml
如下:
apiVersion: v1
kind: Service
metadata:
labels:
app: cassandra
name: cassandra
spec:
clusterIP: None
ports:
- port: 9042
selector:
app: cassandra
kubectl apply -f cassandra-service.yaml
$ kubectl get svc cassandra
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cassandra ClusterIP None <none> 9042/TCP 24s
创建文件 cassandra-statefulset.yaml
如下:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: cassandra
labels:
app: cassandra
spec:
serviceName: cassandra
replicas: 3
selector:
matchLabels:
app: cassandra
template:
metadata:
labels:
app: cassandra
spec:
terminationGracePeriodSeconds: 1800
containers:
- name: cassandra
# image: gcr.io/google-samples/cassandra:v13
image: docker.io/kaiding1/cassandra:v13
imagePullPolicy: Always
ports:
- containerPort: 7000
name: intra-node
- containerPort: 7001
name: tls-intra-node
- containerPort: 7199
name: jmx
- containerPort: 9042
name: cql
resources:
limits:
cpu: "500m"
memory: 1Gi
requests:
cpu: "500m"
memory: 1Gi
securityContext:
capabilities:
add:
- IPC_LOCK
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- nodetool drain
env:
- name: MAX_HEAP_SIZE
value: 512M
- name: HEAP_NEWSIZE
value: 100M
- name: CASSANDRA_SEEDS
value: "cassandra-0.cassandra.default.svc.cluster.local"
- name: CASSANDRA_CLUSTER_NAME
value: "K8Demo"
- name: CASSANDRA_DC
value: "DC1-K8Demo"
- name: CASSANDRA_RACK
value: "Rack1-K8Demo"
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
readinessProbe:
exec:
command:
- /bin/bash
- -c
- /ready-probe.sh
initialDelaySeconds: 15
timeoutSeconds: 5
# These volume mounts are persistent. They are like inline claims,
# but not exactly because the names need to match exactly one of
# the stateful pod volumes.
volumeMounts:
- name: cassandra-data
mountPath: /cassandra_data
# These are converted to volume claims by the controller
# and mounted at the paths mentioned above.
# do not use these in production until ssd GCEPersistentDisk or other ssd pd
volumeClaimTemplates:
- metadata:
name: cassandra-data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: fast
resources:
requests:
storage: 1Gi
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast
provisioner: k8s.io/minikube-hostpath
parameters:
type: pd-ssd
注:因为访问不了 gcr.io
,所以事先把image pull下来,并push到了可访问的位置。
$ kubectl apply -f cassandra-statefulset.yaml
statefulset.apps/cassandra created
storageclass.storage.k8s.io/fast created
验证Cassandra StatefulSet
注:启动pod特别慢,本例中花了半个多小时。
$ kubectl get statefulset cassandra
NAME READY AGE
cassandra 3/3 53m
$ kubectl get pods -l="app=cassandra"
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 53m
cassandra-1 1/1 Running 0 47m
cassandra-2 1/1 Running 0 37m
$ kubectl exec -it cassandra-0 -- nodetool status
Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.244.0.4 142.7 KiB 32 72.8% 913518a2-c96e-4915-9b95-dc26c8c4e9a7 Rack1-K8Demo
UN 10.244.0.5 143.1 KiB 32 70.6% e5689d71-a4b9-4a20-af0b-147ea2ec4c57 Rack1-K8Demo
UN 10.244.0.3 135.94 KiB 32 56.6% af529f81-7da7-4129-b379-d41554390c6f Rack1-K8Demo
修改Cassandra StatefulSet
kubectl edit statefulset cassandra
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: 2016-08-13T18:40:58Z
generation: 1
labels:
app: cassandra
name: cassandra
namespace: default
resourceVersion: "323"
uid: 7a219483-6185-11e6-a910-42010a8a0fc0
spec:
replicas: 3
把replica数量从3改为4,然后保存退出。
$ kubectl edit statefulset cassandra
statefulset.apps/cassandra edited
大约5分钟后,查看状态:
$ kubectl get pods -l="app=cassandra"
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 101m
cassandra-1 1/1 Running 0 96m
cassandra-2 1/1 Running 0 85m
cassandra-3 1/1 Running 0 5m
$ kubectl get statefulset cassandra
NAME READY AGE
cassandra 4/4 101m
清理
删除或缩小StatefulSet不会删除与之关联的volume。这是出于安全考虑,因为与自动清除所有相关的StatefulSet资源相比,数据更有价值。
注意:根据Storage Class和回收策略,删除PVC可能导致相关联的volume也被删除。不要想当然认为PVC被删除后,还能访问数据。
$ grace=$(kubectl get pod cassandra-0 -o=jsonpath='{.spec.terminationGracePeriodSeconds}') \
&& kubectl delete statefulset -l app=cassandra \
&& echo "Sleeping ${grace} seconds" 1>&2 \
&& sleep $grace \
&& kubectl delete persistentvolumeclaim -l app=cassandra
statefulset.apps "cassandra" deleted
Sleeping 1800 seconds
persistentvolumeclaim "cassandra-data-cassandra-0" deleted
persistentvolumeclaim "cassandra-data-cassandra-1" deleted
persistentvolumeclaim "cassandra-data-cassandra-2" deleted
persistentvolumeclaim "cassandra-data-cassandra-3" deleted
$ kubectl delete service -l app=cassandra
service "cassandra" deleted
Cassandra容器环境变量
本例中的pod 使用来自Google容器registry的 gcr.io/google-samples/cassandra:v13
image。该Docker image基于debian-base,且包含 OpenJDK 8。
注:因为访问不了 gcr.io
,所以事先把image pull下来,并push到了可访问的位置。
该镜像包括Apache Debian repo的标准Cassandra安装。通过环境变量,你可以更改插入到 cassandra.yaml
中的值。
环境变量 | 缺省值 |
---|---|
CASSANDRA_CLUSTER_NAME | 'Test Cluster' |
CASSANDRA_NUM_TOKENS | 32 |
CASSANDRA_RPC_ADDRESS | 0.0.0.0 |
参考
https://kubernetes.io/docs/tutorials/stateful-application/cassandra