- 故障原因:
电脑断电,导致etcd集群有一个节点启动失败,比对了数据目录,应该是数据不一致导致的。 - 故障现象:
etcd服务启动失败,日志中有如下报错:
etcdserver: request timed out
故障处理过程
- 集群中删除故障节点
etcdctl --endpoints="https://192.168.171.200:2379,https://192.168.171.201:2379,https://192.168.171.202:2379" member list //获取member ID
36b6dcf065a1b19f: name=etcd03 peerURLs=https://192.168.171.202:2380 clientURLs=https://192.168.171.202:2379 isLeader=false
7a3d1a92c3588a59: name=etcd02 peerURLs=https://192.168.171.201:2380 clientURLs=https://192.168.171.201:2379 isLeader=false
a0ac85faa030bb7e: name=etcd01 peerURLs=https://192.168.171.200:2380 clientURLs=https://192.168.171.200:2379 isLeader=true
etcdctl --endpoints="https://192.168.171.200:2379,https://192.168.171.201:2379,https://192.168.171.202:2379" member remove 36b6dcf065a1b19f //删除故障节点
- 1
- 2
- 3
- 4
- 5
- 删除故障节点数据目录 确保member数据被清除
cd /var/lib/etcd //默认目录
rm -rf *
- 1
- 2
- 将故障节点重新加入集群
etcdctl --endpoints="https://192.168.171.200:2379,https://192.168.171.201:2379,https://192.168.171.202:2379" member add etcd03 --peer-urls=https://192.168.171.202:2380
命令会输出节点加入集群需要的启动参数,具体如下:
ETCD_NAME="etcd03"
ETCD_INITIAL_CLUSTER="etcd03=https://192.168.171.202:2380,etcd02=https://192.168.171.201:2380,etcd01=https://192.168.171.200:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
- 1
- 2
- 3
- 4
- 5
- 启动故障节点etcd服务
由于是旧节点重新加入集群,只需要修改ETCD_INITIAL_CLUSTER_STATE参数为existing
sed -i 's/new/existing/g' /opt/etcd/cfg/etcd
systemctl restart etcd //启动服务
systemctl status etcd //检查服务状态
- 1
- 2
- 3
- 4
- 检查etcd集群状态
etcdctl --endpoints="https://192.168.171.200:2379,https://192.168.171.201:2379,https://192.168.171.202:2379" endpoint status -w table
- 1
小结:
该适用于集群中某个节点故障,例如数据丢失,服务无法启动。