K8S集群维护之etcd集群数据备份还原
一、背景描述
k8s版本:v1.16.6
etcd版本:3.4.3
k8s、etcd复用三台服务器!
服务器地址分别是:
172.16.1.11
172.16.1.12
172.16.1.13
二、备份etcd数据
备份etcd集群任意一个节点的数据就行
①、无证书版本:
export ETCD_ENDPOINTS="https://172.16.1.11:2379,https://172.16.1.12:2379,https://172.16.1.13:2379"
etcdctl --endpoints=${ETCD_ENDPOINTS} snapshot save "/home/snapshot.db"
save 数据备份目录
②、证书版本备份
[root@k8s01 autoshell]# cat /mnt/autoshell/etcd_backup.sh
#!/bin/bash
#时间戳,用来区分不同备份
timestamp=`date +%Y%m%d-%H%M%S`
#备份到哪个文件夹
#back_dir="/data/kubernetes/etcd/datas_bak"
back_dir="/mnt/data_backup/etcd_backup"
#etcd集群列表
#endpoints="https://172.16.1.11:2379,https://172.16.1.12:2379,https://172.16.1.13:2379"
endpoints="https://172.16.1.11:2379"
#etcd证书路径
cert_file="/etc/etcd/cert/etcd.pem"
#etcd证书的key路径
key_file="/etc/etcd/cert/etcd-key.pem"
#ca证书路径
cacert_file="/etc/kubernetes/cert/ca.pem"
mkdir -p $back_dir
ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
--endpoints=$endpoints \
--cert=$cert_file \
--key=$key_file \
--cacert=$cacert_file \
snapshot save $back_dir/snapshot_$timestamp.db
三、etcd集群数据还原
①、还原注意事项
1、将备份文件”snapshot_20210509-144401.db“ 分别拷贝到另外两台服务器。
2、还原备份数据到该路径:–data-dir="/data/k8s/etcd_new/etcd.restore" 注:etcd启动服务配置文件数据目录也需要修改为相应目录!修改如下文件: /etc/systemd/system/etcd.service
②、172.16.1.11 节点数据还原
后续服务器需要修改 --name “hostname” ,修改hostname 为对应服务器名称。
修改 --initial-advertise-peer-urls “https://IP:2380” ,修改IP 为对应服务器IP地址。
export ETCDCTL_API=3
etcdctl snapshot restore /usr/local/src/snapshot_20210509-144401.db \
--data-dir="/data/k8s/etcd_new/etcd.restore" \
--name k8s01 \
--initial-cluster "k8s01=https://172.16.1.11:2380,k8s02=https://172.16.1.12:2380,k8s03=https://172.16.1.13:2380" \
--initial-cluster-token etcd-cluster \
--initial-advertise-peer-urls "https://172.16.1.11:2380"
③、172.16.1.12 节点数据还原
export ETCDCTL_API=3
etcdctl snapshot restore /usr/local/src/snapshot_20210509-144401.db \
--data-dir="/data/k8s/etcd_new/etcd.restore" \
--name k8s02 \
--initial-cluster "k8s01=https://172.16.1.11:2380,k8s02=https://172.16.1.12:2380,k8s03=https://172.16.1.13:2380" \
--initial-cluster-token etcd-cluster \
--initial-advertise-peer-urls "https://172.16.1.12:2380"
④、172.16.1.13 节点数据还原
export ETCDCTL_API=3
etcdctl snapshot restore /usr/local/src/snapshot_20210509-144401.db \
--data-dir="/data/k8s/etcd_new/etcd.restore" \
--name k8s03 \
--initial-cluster "k8s01=https://172.16.1.11:2380,k8s02=https://172.16.1.12:2380,k8s03=https://172.16.1.13:2380" \
--initial-cluster-token etcd-cluster \
--initial-advertise-peer-urls "https://172.16.1.13:2380"
⑤、停止各个节点的etcd服务
[root@k8s01 autoshell]# systemctl stop etcd
⑥、修改各个节点etcd 配置文件,我这里配置到启动文件 /etc/systemd/system/etcd.service中。
修改了三个点:
WorkingDirectory=
–data-dir=
–wal-dir=
[root@k8s01 autoshell]# cat /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/data/k8s/etcd_new
ExecStart=/opt/k8s/bin/etcd \
--data-dir=/data/k8s/etcd_new/etcd.restore \
--wal-dir=/data/k8s/etcd_new/wal \
--name=k8s01 \
--cert-file=/etc/etcd/cert/etcd.pem \
--key-file=/etc/etcd/cert/etcd-key.pem \
--trusted-ca-file=/etc/kubernetes/cert/ca.pem \
--peer-cert-file=/etc/etcd/cert/etcd.pem \
--peer-key-file=/etc/etcd/cert/etcd-key.pem \
--peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \
--peer-client-cert-auth \
--client-cert-auth \
--listen-peer-urls=https://172.16.1.11:2380 \
--initial-advertise-peer-urls=https://172.16.1.11:2380 \
--listen-client-urls=https://172.16.1.11:2379,http://127.0.0.1:2379 \
--advertise-client-urls=https://172.16.1.11:2379 \
--initial-cluster-token=etcd-cluster-0 \
--initial-cluster=k8s01=https://172.16.1.11:2380,k8s02=https://172.16.1.12:2380,k8s03=https://172.16.1.13:2380 \
--initial-cluster-state=new \
--auto-compaction-mode=periodic \
--auto-compaction-retention=1 \
--max-request-bytes=33554432 \
--quota-backend-bytes=6442450944 \
--heartbeat-interval=250 \
--election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
⑦、启动各个节点etcd服务(这里演示K8S01节点)
[root@k8s01 autoshell]# systemctl start etcd
查看集群leader
export ETCD_ENDPOINTS="https://172.16.1.11:2379,https://172.16.1.12:2379,https://172.16.1.13:2379"
/opt/k8s/bin/etcdctl \
-w table --cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/etcd/cert/etcd.pem \
--key=/etc/etcd/cert/etcd-key.pem \
--endpoints=${ETCD_ENDPOINTS} endpoint status
四、k8s 集群其他服务
重启K8S集群master节点服务
systemctl restart kubelet
systemctl restart kube-apiserver
systemctl restart kube-controller-manager
systemctl restart kube-scheduler
systemctl restart kube-proxy
systemctl restart kube-nginx
systemctl restart containerd.service
重启K8S集群work节点服务
systemctl restart kubelet
systemctl restart kube-nginx
systemctl restart kube-proxy
参考:https://www.cnblogs.com/golinux/p/12576331.html