搭建k8s集群错误集

最新推荐文章于 2024-05-21 11:31:40 发布

VIP文章 nsydgs

最新推荐文章于 2024-05-21 11:31:40 发布

阅读量4.4k

点赞数

文章标签： kubernetes docker 运维

本文链接：https://blog.csdn.net/nsydgs/article/details/126265598

版权

1

etcd

8月 10 14:12:32 k8master-1 etcd[23435]: {
   "level":"warn","ts":"2022-08-10T14:12:32.069+0800","caller":"rafthttp/http.go:500","msg":"request cluster ID mismatch","local-member-id":"44ec88b2ad8081e","local-member-cluster-id":"ced548654624706f","local-member-server-version":"3.5.0","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"1d412b7cdf0f5787","remote-peer-server-version":"3.5.0","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"8c96ad28e090da8f"}

kube-apiserver

E0810 14:15:31.208449   22888 controller.go:223] unable to sync kubernetes service: etcdserver: requested lease not found
E0810 14:15:41.208772   22888 controller.go:223] unable to sync kubernetes service: etcdserver: requested lease not found

排查：

[root@k8master-1 work]#  /app/k8s/bin/etcdctl --cacert=/etc/kubernetes/cert/ca.pem  --cert=/etc/etcd/cert/etcd.pem --key=/etc/etcd/cert/etcd-key.pem --endpoints=https://192.168.159.156:2379,https://192.168.159.158:2379,https://192.168.159.159:2379 member list  -w table
+------------------+---------+------------+------------------------------+------------------------------+------------+
|        ID        | STATUS  |    NAME    |          PEER ADDRS          |         CLIENT ADDRS         | IS LEARNER |
+------------------+---------+------------+------------------------------+------------------------------+------------+
|  44ec88b2ad8081e | started | k8master-1 | https://192.168.159.156:2380 |                              |      false |
|  7d173c333430d55 | started | k8worker-2 | https://192.168.159.159:2380 | https://192.168.159.159:2379 |      false |
| 1d412b7cdf0f5787 | started | k8worker-1 | https://192.168.159.158:2380 | https://192.168.159.158:2379 |      false |
+------------------+---------+------------+------------------------------+------------------------------+------------+

[root@k8master-1 work]#  /app/k8s/bin/etcdctl --cacert=/etc/kubernetes/cert/ca.pem  --cert=/etc/etcd/cert/etcd.pem --key=/etc/etcd/cert/etcd-key.pem --endpoints=https://192.168.159.156:2379,https://192.168.159.158:2379,https://192.168.159.159:2379 endpoint status  -w table
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.159.156:2379 |  44ec88b2ad8081e |   3.5.0 |  741 kB |      true |      false |        14 |       6986 |               6986 |        |
| https://192.168.159.158:2379 | 1d412b7cdf0f5787 |   3.5.0 |  1.3 MB |      true |      false |        17 |      40171 |              40171 |        |
| https://192.168.159.159:2379 |  7d173c333430d55 |   3.5.0 |  1.3 MB |     false |      false |        17 |      40171 |              40171 |        |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

如果出现 IS LEADER 2个true,检查日志发现： request cluster ID mismatch
需要删除：
/app/k8s/etcd/work/* #
/app/k8s/etcd/wal/* #
再重启服务。

解决方法：

 1031  systemctl stop  etcd.service
 1032  systemctl status  etcd.service
 1033  rm -f /app/k8s/etcd/work/*
 1034  rm -f /app/k8s/etcd/wal/*
 1035  systemctl start  etcd.service

正常日志：

 1070  8月 10 15:04:35 k8worker-2 etcd[56620]: {
   "level":"info","ts":"2022-08-10T15:04:35.319+0800","caller":"mvcc/index.go:214","msg":"compact tree index","revision":2245}
 1071  8月 10 15:04:35 k8worker-2 etcd[56620]: {
   "level":"info","ts":"2022-08-10T15:04:35.319+0800","caller":"mvcc/kvstore_compaction.go:57","msg":"finished scheduled compaction","compact-revision":2245,"took":"63.833µs"}
 1072  8月 10 15:09:35 k8worker-2 etcd[56620]: {
   "level":"info","ts":"2022-08-10T15:09:35.326+0800","caller":"mvcc/index.go:214","msg":"compact tree index","revision":2247}
 1073  8月 10 15:09:35 k8worker-2 etcd[56620]: {
   "level":"info","ts":"2022-08-10T15:09:35.327+0800","caller":"mvcc/kvstore_compaction.go:57","msg":"finished scheduled compaction","compact-revision":2247,"took":"46.555µs"

2

etcd

{
   "level":"fatal","ts":"2022-08-10T15:03:50.046+0800","caller":"etcdmain/etcd.go:203","msg":"discovery failed","error":"cannot fetch cluster info from peer urls: could not retrieve cluster information from the given URLs","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdmain/etcd.go:203\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdmain/main.go:40\nmain.main\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/main.go:32\nruntime.main\n\t/home/remote/sbatsche/.gvm/gos/go1.16.3/src/runtime/proc.go:225"}
8月 10 15:03:50 k8master-1 systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
8月 10 15:03:50 k8master-1 systemd[1]: Failed to start Etcd Server.
8月 10 15:03:50 k8master-1 systemd[1]: Unit etcd.service entered failed state.
8月 10 15:03:50 k8master-1 systemd[1]: etcd.service failed.

其他节点没有启动，其他节点启动即可。

3

kube-controller-manager

8月 10 15:23:01 k8master-1 kube-controller-manager[35641]: unable to load configmap based request-header-client-ca-file: Get "https://192.168.159.156:6443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": x509: certificate signed by unknown authority

排查过程：

[root@k8master-1 work]# cat /etc/systemd/system/kube-controller-manager.service |grep pem
  --client-ca-file=/etc/kubernetes/cert/ca.pem \
  --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \
  --root-ca-file=/etc/kubernetes/cert/ca.pem \
  --service-account-private-key-file=/etc/kubernetes/cert/apiserver-key.pem \
  --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \
[root@k8master-1 work]# cfssl certinfo -cert /etc/kubernetes/cert/ca.pem
{
   
  "subject":