本文目的是使用statefulset部署3节点的etcd集群
一。准备双AZ 4节点的集群
构建集群参考另一篇帖子CentOS7安装k8s v1.23.4集群
k8s-node1,k8s-node2打上标签topology.kubernetes.io/zone=az1,表示AZ1;
k8s-node3,k8s-node4打上标签topology.kubernetes.io/zone=az2,表示AZ2;
二。构建etcd v3.5.4镜像
自行构建docker镜像,从etcd官网下载etcd-v3.5.4-linux-amd64.tar.gz并解压,
Dockerfile配置:
FROM centos:7
USER root
ADD etcd-v3.5.4-linux-amd64/etcd* /usr/bin/
COPY etcd.sh /
RUN chmod 777 /etcd.sh
RUN chmod 777 /usr/bin/etcd
CMD ["/etcd.sh"]
容器启动脚本etcd.sh,环境变量由pod注入,见statefulset中的配置
#!/bin/bash
echo ${MY_POD_NAME}
echo ${MY_POD_IP}
echo ${HEADLESS_SERVICE_NAME}
echo ${CLUSTER_NAMESPACE}
echo ${INITIAL_CLUSTER}
/usr/bin/etcd --data-dir=/var/lib/etcd --name=${MY_POD_NAME} --listen-peer-urls=http://0.0.0.0:2380 --listen-client-urls=http://0.0.0.0:2379 --advertise-client-urls=http://${MY_POD_NAME}.${HEADLESS_SERVICE_NAME}.${CLUSTER_NAMESPACE}.svc.cluster.local:2379 --initial-advertise-peer-urls=http://${MY_POD_NAME}.${HEADLESS_SERVICE_NAME}.${CLUSTER_NAMESPACE}.svc.cluster.local:2380 --initial-cluster-state='new' --initial-cluster-token='etcd-cluster-token' --initial-cluster=${INITIAL_CLUSTER}
构建镜像并推送到阿里个人镜像仓库
也可直接docker pull registry.cn-shenzhen.aliyuncs.com/k8s_go_java/etcd:v1
三。使用local-storage存储类提供pv
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
创建pv前,先在每个节点创建pv对应的目录/mnt/statefulset/etcd,注意pv配置了节点亲和
apiVersion: v1
kind: PersistentVolume
metadata:
name: etcd-pv
spec:
capacity:
storage: 2Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /mnt/statefulset/etcd
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- az1
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: etcd-pv-2
spec:
capacity:
storage: 2Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /mnt/statefulset/etcd
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- az1
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: etcd-pv-3
spec:
capacity:
storage: 2Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /mnt/statefulset/etcd
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- az2
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: etcd-pv-4
spec:
capacity:
storage: 2Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /mnt/statefulset/etcd
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- az2
四。创建svc
一个名为etcd-headless-svc的无头服务,一个NodePort类型的svc
apiVersion: v1
kind: Service
metadata:
name: etcd-headless-svc
namespace: etcd
labels:
app: etcd
spec:
clusterIP: None
ports:
- name: etcd-client
port: 2379
targetPort: 2379
protocol: TCP
- name: etcd-peer
port: 2380
targetPort: 2380
protocol: TCP
selector:
app: etcd
---
apiVersion: v1
kind: Service
metadata:
name: etcd-nodeport
namespace: etcd
labels:
app: etcd
spec:
type: NodePort
ports:
- name: etcd-cluster
port: 2379
targetPort: 2379
protocol: TCP
sessionAffinity: None
selector:
app: etcd
五。创建statefulset
这里创建了两个sts:etcd-az1和etcd-az2,目的是限制sts中的pod跨AZ调度,即etcd-az1中的pod只能在AZ1下的k8s-node1和k8s-node2中调度,而不能在AZ2的节点中调度,保障AZ级别的高可用,注意nodeSelector中对AZ的配置。
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: etcd-az1
namespace: etcd
labels:
app.kubernetes.io/name: etcd-cluster-az1
app: etcd
spec:
podManagementPolicy: Parallel
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: etcd-cluster-az1
serviceName: etcd-headless-svc
template:
metadata:
labels:
app: etcd
app.kubernetes.io/name: etcd-cluster-az1
spec:
nodeSelector:
topology.kubernetes.io/zone: az1
containers:
- name: etcd
image: registry.cn-shenzhen.aliyuncs.com/k8s_go_java/etcd:v1
imagePullPolicy: IfNotPresent
ports:
- name: client
containerPort: 2379
protocol: TCP
- name: peer
containerPort: 2380
protocol: TCP
env:
- name: MY_POD_NAME #当前pod名
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_POD_IP #当前pod ip
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CLUSTER_NAMESPACE #名称空间
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: HEADLESS_SERVICE_NAME #内部通信的无头服务名称
value: "etcd-headless-svc"
- name: INITIAL_CLUSTER #initial-cluster的值
value: "etcd-az1-0=http://etcd-az1-0.etcd-headless-svc.etcd.svc.cluster.local:2380,etcd-az1-1=http://etcd-az1-1.etcd-headless-svc.etcd.svc.cluster.local:2380,etcd-az2-0=http://etcd-az2-0.etcd-headless-svc.etcd.svc.cluster.local:2380"
resources:
requests:
cpu: "1000m"
memory: "1Gi"
limits:
cpu: "1000m"
memory: "1Gi"
volumeMounts:
- name: data
mountPath: /var/lib/etcd
updateStrategy:
type: OnDelete
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: local-storage
resources:
requests:
storage: 2Gi
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: etcd-az2
namespace: etcd
labels:
app.kubernetes.io/name: etcd-cluster-az2
app: etcd
spec:
podManagementPolicy: Parallel
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: etcd-cluster-az2
serviceName: etcd-headless-svc
template:
metadata:
labels:
app: etcd
app.kubernetes.io/name: etcd-cluster-az2
spec:
nodeSelector:
topology.kubernetes.io/zone: az2
containers:
- name: etcd
image: registry.cn-shenzhen.aliyuncs.com/k8s_go_java/etcd:v1
imagePullPolicy: IfNotPresent
ports:
- name: client
containerPort: 2379
protocol: TCP
- name: peer
containerPort: 2380
protocol: TCP
env:
- name: MY_POD_NAME #当前pod名
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_POD_IP #当前pod ip
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CLUSTER_NAMESPACE #名称空间
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: HEADLESS_SERVICE_NAME #内部通信的无头服务名称
value: "etcd-headless-svc"
- name: INITIAL_CLUSTER #initial-cluster的值
value: "etcd-az1-0=http://etcd-az1-0.etcd-headless-svc.etcd.svc.cluster.local:2380,etcd-az1-1=http://etcd-az1-1.etcd-headless-svc.etcd.svc.cluster.local:2380,etcd-az2-0=http://etcd-az2-0.etcd-headless-svc.etcd.svc.cluster.local:2380"
resources:
requests:
cpu: "1000m"
memory: "1Gi"
limits:
cpu: "1000m"
memory: "1Gi"
volumeMounts:
- name: data
mountPath: /var/lib/etcd
updateStrategy:
type: OnDelete
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: local-storage
resources:
requests:
storage: 2Gi
六。验证
用NodePort方式访问etcd集群