1.问题描述
panic: 49630da07712d85a state.commit 91684558 is out of range [91511394, 91519050]
2.问题解决步骤
# 登录异常节点,停止etcd
# systemctl stop etcd
# 登录异常节点,先备份这个节点的数据信息
# cp -r /var/lib/etcd/member ~/member
# 登录异常节点,删除member下面的数据
# rm -rf /var/lib/etcd/member/*
# 登录etcd的leader节点,进入容器
# docker ps|grep etcd
# docker exec -it c8e8f6618bab sh
# 登录etcd的leader节点,查看member信息
# cd /usr/local/bin/
# ./etcdctl member list
/usr/local/bin # ./etcdctl member list
2bc4bab341916457: name=etcd1 peerURLs=https://30.23.11.48:2380 clientURLs=https://30.23.11.48:2379 isLeader=true
49630da07712d85a: name=etcd3 peerURLs=https://30.23.13.75:2380 clientURLs=https://30.23.13.75:2379 isLeader=false
d7ee0d7ab8c4f414: name=etcd2 peerURLs=https://30.23.11.152:2380 clientURLs=https://30.23.11.152:2379 isLeader=false
# 登录etcd的leader节点,删除该异常节点
# ./etcdctl member remove 49630da07712d85a
/usr/local/bin # ./etcdctl member remove 49630da07712d85a
Removed member 49630da07712d85a from cluster
# 登录etcd的leader节点,再查看节点列表
# ./etcdctl member list
/usr/local/bin # ./etcdctl member list
2bc4bab341916457: name=etcd1 peerURLs=https://30.23.11.48:2380 clientURLs=https://30.23.11.48:2379 isLeader=true
d7ee0d7ab8c4f414: name=etcd2 peerURLs=https://30.23.11.152:2380 clientURLs=https://30.23.11.152:2379 isLeader=false
# 登录异常节点,启动etcd
# systemctl restart etcd
# 登录etcd的leader节点,将异常节点重新添加到集群
# ./etcdctl member add 49630da07712d85a https://30.23.13.75:2380
/usr/local/bin # ./etcdctl member add 49630da07712d85a https://30.23.13.75:2380
Added member named 49630da07712d85a with ID 624063775fe61328 to cluster
ETCD_NAME="49630da07712d85a"
ETCD_INITIAL_CLUSTER="etcd1=https://30.23.11.48:2380,49630da07712d85a=https://30.23.13.75:2380,etcd2=https://30.23.11.152:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
/usr/local/bin # ./etcdctl member list
2bc4bab341916457: name=etcd1 peerURLs=https://30.23.11.48:2380 clientURLs=https://30.23.11.48:2379 isLeader=true
624063775fe61328: name=etcd3 peerURLs=https://30.23.13.75:2380 clientURLs=https://30.23.13.75:2379 isLeader=false
d7ee0d7ab8c4f414: name=etcd2 peerURLs=https://30.23.11.152:2380 clientURLs=https://30.23.11.152:2379 isLeader=false
# 登录etcd的leader节点,重新查看列表
# ./etcdctl member list
/usr/local/bin # ./etcdctl member list
2bc4bab341916457: name=etcd1 peerURLs=https://30.23.11.48:2380 clientURLs=https://30.23.11.48:2379 isLeader=true
624063775fe61328: name=etcd3 peerURLs=https://30.23.13.75:2380 clientURLs=https://30.23.13.75:2379 isLeader=false
d7ee0d7ab8c4f414: name=etcd2 peerURLs=https://30.23.11.152:2380 clientURLs=https://30.23.11.152:2379 isLeader=false
# 登录异常节点查看日志,没有报错即可
# docker logs -f 94e03f9fa8b2