statefulset部署etcd v3.5.4集群

greatsharp

已于 2022-07-06 22:21:00 修改

阅读量996

点赞数 1

分类专栏： Kubernetes 文章标签： etcd kubernetes docker

于 2022-07-05 22:37:16 首次发布

本文链接：https://blog.csdn.net/greatsharp/article/details/125629575

版权

Kubernetes 专栏收录该内容

4 篇文章 0 订阅

订阅专栏

本文目的是使用statefulset部署3节点的etcd集群

一。准备双AZ 4节点的集群

构建集群参考另一篇帖子CentOS7安装k8s v1.23.4集群

k8s-node1，k8s-node2打上标签topology.kubernetes.io/zone=az1，表示AZ1；

k8s-node3，k8s-node4打上标签topology.kubernetes.io/zone=az2，表示AZ2；

二。构建etcd v3.5.4镜像

自行构建docker镜像，从etcd官网下载etcd-v3.5.4-linux-amd64.tar.gz并解压，

Dockerfile配置：

FROM centos:7

USER root

ADD etcd-v3.5.4-linux-amd64/etcd* /usr/bin/

COPY etcd.sh /

RUN chmod 777 /etcd.sh

RUN chmod 777 /usr/bin/etcd

CMD ["/etcd.sh"]

容器启动脚本etcd.sh，环境变量由pod注入，见statefulset中的配置

#!/bin/bash
echo ${MY_POD_NAME}
echo ${MY_POD_IP}
echo ${HEADLESS_SERVICE_NAME}
echo ${CLUSTER_NAMESPACE}
echo ${INITIAL_CLUSTER}

/usr/bin/etcd --data-dir=/var/lib/etcd --name=${MY_POD_NAME} --listen-peer-urls=http://0.0.0.0:2380 --listen-client-urls=http://0.0.0.0:2379 --advertise-client-urls=http://${MY_POD_NAME}.${HEADLESS_SERVICE_NAME}.${CLUSTER_NAMESPACE}.svc.cluster.local:2379 --initial-advertise-peer-urls=http://${MY_POD_NAME}.${HEADLESS_SERVICE_NAME}.${CLUSTER_NAMESPACE}.svc.cluster.local:2380 --initial-cluster-state='new' --initial-cluster-token='etcd-cluster-token' --initial-cluster=${INITIAL_CLUSTER}

构建镜像并推送到阿里个人镜像仓库

也可直接docker pull registry.cn-shenzhen.aliyuncs.com/k8s_go_java/etcd:v1

三。使用local-storage存储类提供pv

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

创建pv前，先在每个节点创建pv对应的目录/mnt/statefulset/etcd，注意pv配置了节点亲和

apiVersion: v1
kind: PersistentVolume
metadata:
  name: etcd-pv
spec:
  capacity:
    storage: 2Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /mnt/statefulset/etcd
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values:
          - az1


---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: etcd-pv-2
spec:
  capacity:
    storage: 2Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /mnt/statefulset/etcd
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values:
          - az1


---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: etcd-pv-3
spec:
  capacity:
    storage: 2Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /mnt/statefulset/etcd
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values:
          - az2


---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: etcd-pv-4
spec:
  capacity:
    storage: 2Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /mnt/statefulset/etcd
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values:
          - az2

四。创建svc

一个名为etcd-headless-svc的无头服务，一个NodePort类型的svc

apiVersion: v1
kind: Service
metadata:
  name: etcd-headless-svc
  namespace: etcd
  labels:
    app: etcd      
spec:
  clusterIP: None
  ports:
  - name: etcd-client
    port: 2379
    targetPort: 2379
    protocol: TCP
  - name: etcd-peer
    port: 2380
    targetPort: 2380
    protocol: TCP
  selector:
    app: etcd


---


apiVersion: v1
kind: Service
metadata:
  name: etcd-nodeport
  namespace: etcd
  labels:
    app: etcd
spec:
  type: NodePort
  ports:
  - name: etcd-cluster
    port: 2379
    targetPort: 2379
    protocol: TCP
  sessionAffinity: None
  selector:
    app: etcd

五。创建statefulset

这里创建了两个sts：etcd-az1和etcd-az2，目的是限制sts中的pod跨AZ调度，即etcd-az1中的pod只能在AZ1下的k8s-node1和k8s-node2中调度，而不能在AZ2的节点中调度，保障AZ级别的高可用，注意nodeSelector中对AZ的配置。

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: etcd-az1
  namespace: etcd
  labels:
    app.kubernetes.io/name: etcd-cluster-az1
    app: etcd
spec:
  podManagementPolicy: Parallel
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: etcd-cluster-az1
  serviceName: etcd-headless-svc
  template:
    metadata:
      labels:
        app: etcd
        app.kubernetes.io/name: etcd-cluster-az1
    spec:
      nodeSelector:
        topology.kubernetes.io/zone: az1
      containers:
      - name: etcd
        image: registry.cn-shenzhen.aliyuncs.com/k8s_go_java/etcd:v1
        imagePullPolicy: IfNotPresent
        ports:
        - name: client
          containerPort: 2379
          protocol: TCP
        - name: peer
          containerPort: 2380
          protocol: TCP
        env:
        - name: MY_POD_NAME  #当前pod名
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: MY_POD_IP  #当前pod ip
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: CLUSTER_NAMESPACE  #名称空间
          valueFrom: 
            fieldRef:
              fieldPath: metadata.namespace
        - name: HEADLESS_SERVICE_NAME   #内部通信的无头服务名称
          value: "etcd-headless-svc"
        - name: INITIAL_CLUSTER   #initial-cluster的值
          value: "etcd-az1-0=http://etcd-az1-0.etcd-headless-svc.etcd.svc.cluster.local:2380,etcd-az1-1=http://etcd-az1-1.etcd-headless-svc.etcd.svc.cluster.local:2380,etcd-az2-0=http://etcd-az2-0.etcd-headless-svc.etcd.svc.cluster.local:2380"
        resources:
          requests:
            cpu: "1000m"
            memory: "1Gi"
          limits:
            cpu: "1000m"
            memory: "1Gi"
        volumeMounts:
        - name: data
          mountPath: /var/lib/etcd
  updateStrategy:
    type: OnDelete
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: local-storage
      resources:
        requests:
          storage: 2Gi

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: etcd-az2
  namespace: etcd
  labels:
    app.kubernetes.io/name: etcd-cluster-az2
    app: etcd
spec:
  podManagementPolicy: Parallel
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: etcd-cluster-az2
  serviceName: etcd-headless-svc
  template:
    metadata:
      labels:
        app: etcd
        app.kubernetes.io/name: etcd-cluster-az2
    spec:
      nodeSelector:
        topology.kubernetes.io/zone: az2
      containers:
      - name: etcd
        image: registry.cn-shenzhen.aliyuncs.com/k8s_go_java/etcd:v1
        imagePullPolicy: IfNotPresent
        ports:
        - name: client
          containerPort: 2379
          protocol: TCP
        - name: peer
          containerPort: 2380
          protocol: TCP
        env:
        - name: MY_POD_NAME  #当前pod名
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: MY_POD_IP  #当前pod ip
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: CLUSTER_NAMESPACE  #名称空间
          valueFrom: 
            fieldRef:
              fieldPath: metadata.namespace
        - name: HEADLESS_SERVICE_NAME   #内部通信的无头服务名称
          value: "etcd-headless-svc"
        - name: INITIAL_CLUSTER   #initial-cluster的值
          value: "etcd-az1-0=http://etcd-az1-0.etcd-headless-svc.etcd.svc.cluster.local:2380,etcd-az1-1=http://etcd-az1-1.etcd-headless-svc.etcd.svc.cluster.local:2380,etcd-az2-0=http://etcd-az2-0.etcd-headless-svc.etcd.svc.cluster.local:2380"
        resources:
          requests:
            cpu: "1000m"
            memory: "1Gi"
          limits:
            cpu: "1000m"
            memory: "1Gi"
        volumeMounts:
        - name: data
          mountPath: /var/lib/etcd
  updateStrategy:
    type: OnDelete
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: local-storage
      resources:
        requests:
          storage: 2Gi