记etcd节点故障恢复

• 故障原因：
  电脑断电，导致etcd集群有一个节点启动失败，比对了数据目录，应该是数据不一致导致的。
• 故障现象：
  etcd服务启动失败，日志中有如下报错：
  recovering backend from snapshot error: database snapshot file path error

故障处理过程

• 集群中删除故障节点

etcdctl --endpoints="https://192.168.171.200:2379,https://192.168.171.201:2379,https://192.168.171.202:2379" member list   //获取member ID 
36b6dcf065a1b19f: name=etcd03 peerURLs=https://192.168.171.202:2380 clientURLs=https://192.168.171.202:2379 isLeader=false
7a3d1a92c3588a59: name=etcd02 peerURLs=https://192.168.171.201:2380 clientURLs=https://192.168.171.201:2379 isLeader=false
a0ac85faa030bb7e: name=etcd01 peerURLs=https://192.168.171.200:2380 clientURLs=https://192.168.171.200:2379 isLeader=true
etcdctl --endpoints="https://192.168.171.200:2379,https://192.168.171.201:2379,https://192.168.171.202:2379" member remove 36b6dcf065a1b19f  //删除故障节点

• 删除故障节点数据目录 确保member数据被清除

cd /var/lib/etcd   //默认目录
rm -rf *

• 将故障节点重新加入集群

etcdctl --endpoints="https://192.168.171.200:2379,https://192.168.171.201:2379,https://192.168.171.202:2379" member add etcd03 https://192.168.171.202:2380
命令会输出节点加入集群需要的启动参数，具体如下：
ETCD_NAME="etcd03"
ETCD_INITIAL_CLUSTER="etcd03=https://192.168.171.202:2380,etcd02=https://192.168.171.201:2380,etcd01=https://192.168.171.200:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

• 启动故障节点etcd服务

由于是旧节点重新加入集群，只需要修改ETCD_INITIAL_CLUSTER_STATE参数为existing
sed -i 's/new/existing/g' /opt/etcd/cfg/etcd
systemctl restart etcd  //启动服务
systemctl status etcd		//检查服务状态

• 检查etcd集群状态

etcdctl --endpoints="https://192.168.171.200:2379,https://192.168.171.201:2379,https://192.168.171.202:2379" cluster-health

小结：

该适用于集群中某个节点故障，例如数据丢失，服务无法启动。