rancher报错连接etcd 503

rancher报错连接etcd 503

错误信息

2022/01/24 10:01:12 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:01:13 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:01:14 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:01:15 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:01:16 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:02:05 [INFO] Rancher version 4384647 is starting
2022/01/24 10:02:05 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:auto Embedded:false KubeConfig: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false NoCACerts:false ListenConfig:<nil> AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features:}
2022/01/24 10:02:05 [INFO] Listening on /tmp/log.sock
2022/01/24 10:02:05 [INFO] Running etcd --data-dir=management-state/etcd
2022-01-24 10:02:06.070629 W | pkg/flags: unrecognized environment variable ETCD_URL_arm64=https://github.com/etcd-io/etcd/releases/download/v3.3.14/etcd-v3.3.14-linux-arm64.tar.gz
2022-01-24 10:02:06.070685 W | pkg/flags: unrecognized environment variable ETCD_URL_amd64=http://10.170.130.153:9001/2.3.2/etcd-v3.3.14-linux-amd64.tar.gz
2022-01-24 10:02:06.070690 W | pkg/flags: unrecognized environment variable ETCD_UNSUPPORTED_ARCH=amd64
2022-01-24 10:02:06.070694 W | pkg/flags: unrecognized environment variable ETCD_URL=ETCD_URL_amd64
2022-01-24 10:02:06.070720 I | etcdmain: etcd Version: 3.3.14
2022-01-24 10:02:06.070724 I | etcdmain: Git SHA: 5cf5d88a1
2022-01-24 10:02:06.070726 I | etcdmain: Go Version: go1.12.9
2022-01-24 10:02:06.070729 I | etcdmain: Go OS/Arch: linux/amd64
2022-01-24 10:02:06.070733 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8
2022-01-24 10:02:06.070799 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2022-01-24 10:02:06.071238 I | embed: listening for peers on http://localhost:2380
2022-01-24 10:02:06.071371 I | embed: listening for client requests on localhost:2379
2022-01-24 10:02:16.071759 W | etcdserver: another etcd process is using "management-state/etcd/member/snap/db" and holds the file lock, or loading backend file is taking >10 seconds
2022-01-24 10:02:16.071786 W | etcdserver: waiting for it to exit before starting...
2022-01-24 10:02:26.393768 I | etcdserver: recovered store from snapshot at index 12500125
2022-01-24 10:02:26.402541 I | mvcc: restore compact to 8371858
2022-01-24 10:02:29.463731 I | etcdserver: name = default
2022-01-24 10:02:29.463759 I | etcdserver: data dir = management-state/etcd
2022-01-24 10:02:29.463766 I | etcdserver: member dir = management-state/etcd/member
2022-01-24 10:02:29.463770 I | etcdserver: heartbeat = 100ms
2022-01-24 10:02:29.463773 I | etcdserver: election = 1000ms
2022-01-24 10:02:29.463776 I | etcdserver: snapshot count = 100000
2022-01-24 10:02:29.463822 I | etcdserver: advertise client URLs = http://localhost:2379
2022-01-24 10:02:32.195901 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 12592794
2022-01-24 10:02:32.198720 I | raft: 8e9e05c52164694d became follower at term 15380
2022-01-24 10:02:32.198748 I | raft: newRaft 8e9e05c52164694d [peers: [8e9e05c52164694d], term: 15380, commit: 12592794, applied: 12500125, lastindex: 12592794, lastterm: 15380]
2022-01-24 10:02:32.198897 I | etcdserver/api: enabled capabilities for version 3.3
2022-01-24 10:02:32.198918 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32 from store
2022-01-24 10:02:32.198924 I | etcdserver/membership: set the cluster version to 3.3 from store
2022-01-24 10:02:32.503001 I | mvcc: restore compact to 8371858
2022-01-24 10:02:35.457532 W | auth: simple token is not cryptographically signed
2022-01-24 10:02:35.505883 I | etcdserver: starting server... [version: 3.3.14, cluster version: 3.3]
2022-01-24 10:02:35.506109 I | etcdserver: 8e9e05c52164694d as single-node; fast-forwarding 9 ticks (election ticks 10)
2022-01-24 10:02:36.199408 I | raft: 8e9e05c52164694d is starting a new election at term 15380
2022-01-24 10:02:36.199448 I | raft: 8e9e05c52164694d became candidate at term 15381
2022-01-24 10:02:36.199471 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 15381
2022-01-24 10:02:36.199486 I | raft: 8e9e05c52164694d became leader at term 15381
2022-01-24 10:02:36.199492 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 15381
2022-01-24 10:02:36.300671 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2022-01-24 10:02:36.301042 I | embed: ready to serve client requests
2022-01-24 10:02:36.302735 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
2022/01/24 10:02:37 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:02:38 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:02:39 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:02:40 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:02:41 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:02:42 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:02:43 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:02:44 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:02:45 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:02:46 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:02:47 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:02:48 [INFO] Waiting on etcd startup: status 503
2022/01/24 10:02:49 [INFO] Waiting on etcd startup: status 503

进入rancher容器手动清理etcd碎片

docker exec -it rancher bash

使用API3

export ETCDCTL_API=3

查看告警信息，告警信息一般 memberID:8630161756594109333 alarm:NOSPACE

etcdctl --endpoints=http://127.0.0.1:2379 alarm list

获取当前版本

rev=$(etcdctl --endpoints=http://127.0.0.1:2379 endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9].*')

压缩掉所有旧版本

etcdctl --endpoints=http://127.0.0.1:2379 compact $rev

整理多余的空间

etcdctl --endpoints=http://127.0.0.1:2379 defrag

取消告警信息

etcdctl --endpoints=http://127.0.0.1:2379 alarm disarm