k8s集群添加节点遇到的问题

加入集群卡住不动

kubeadm join 172.25.42.235:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:98de276382a20d872f579a74ee924b38d3705ebfcf268b1658a0b6ddb3dccaaa

开debug模式

kubeadm join 172.25.42.235:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:98de276382a20d872f579a74ee924b38d3705ebfcf268b1658a0b6ddb3dccaaa  -v=10

报错

I0531 05:04:20.734795    8706 round_trippers.go:435] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.21.1 (linux/amd64) kubernetes/5e58841" 'https://172.25.42.235:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s'
I0531 05:04:20.736850    8706 round_trippers.go:454] GET https://172.25.42.235:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s 200 OK in 2 milliseconds
I0531 05:04:20.736865    8706 round_trippers.go:460] Response Headers:
I0531 05:04:20.736870    8706 round_trippers.go:463]     Cache-Control: no-cache, private
I0531 05:04:20.736873    8706 round_trippers.go:463]     Content-Type: application/json
I0531 05:04:20.736876    8706 round_trippers.go:463]     X-Kubernetes-Pf-Flowschema-Uid: aabbe46b-393b-42d9-886b-aaf4b2fc7ce2
I0531 05:04:20.736879    8706 round_trippers.go:463]     X-Kubernetes-Pf-Prioritylevel-Uid: 3120be8e-1c11-4b11-ad37-0de38bb70030
I0531 05:04:20.736882    8706 round_trippers.go:463]     Content-Length: 2353
I0531 05:04:20.736915    8706 round_trippers.go:463]     Date: Mon, 31 May 2021 09:04:20 GMT
I0531 05:04:20.737280    8706 request.go:1123] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"cluster-info","namespace":"kube-public","uid":"4e7f9c17-5dc6-41ce-a18d-1076606b97d1","resourceVersion":"5183381","creationTimestamp":"2021-04-14T07:01:55Z","managedFields":[{"manager":"kubeadm","operation":"Update","apiVersion":"v1","time":"2021-04-14T07:01:55Z","fieldsType":"FieldsV1","fieldsV1":{"f:data":{".":{},"f:kubeconfig":{}}}},{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-05-31T09:03:26Z","fieldsType":"FieldsV1","fieldsV1":{"f:data":{"f:jws-kubeconfig-oumnnc":{}}}}]},"data":{"jws-kubeconfig-oumnnc":"eyJhbGciOiJIUzI1NiIsImtpZCI6Im91bW5uYyJ9..2Omg4lzZWg82FA-nUq1UxpnLL2qUR7-cgls62ciTvOE","kubeconfig":"apiVersion: v1\nclusters:\n- cluster:\n    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeE1EUXhOREEzTURBME5sb1hEVE14TURReE1qQTNNREEwTmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSy9uCk9wZEMyNlpBNnRHODA5NkxlcDVXY0w3eTdwRmNmcmVYVWVjNDlxNFlQekI4cW1NK2I0Vy9lTE45dnBRa3l1WHcKbWZCbVJsOVp0aDlXWGVDVjgxVzN5OXEyQUgrWTgwSmJiOGZEYkxsREwrWTZQa25DOW45bEFDVXRmMTlHNHlCbwprdTFtYlVMM215c2pKYWY3RC81RFhsaEZpaGFQVkRaT1M3RVZJNlRqRU4xWlV2VDQraEMxRkgrcUhVS0hIUGNYCnhtUXQzMUtxR1gyOW5sNkd3TzlQajRGbnVTV3JxQnpCM01YRHpESGlnSXBhMm9mVjY3OU9nZ1B3c0duVkRFbzUKU1k0dmFTK0djR29ZVFhoWGMyYm5kbFVuSDBIc0l3UmpPeWFldWpwREdYclhvMzh0Mk1hR0F2L3g1Z2NvTVM3KwplWVdYclFtKzJXRURDNlpDUkpjQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZLcTBobnNTMHNDUDcyQmFTaGR6cHRjc3lKRHhNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFBVkgvNDVRUzVUb2VJMk4xaHVOQUZqMDJtb0RCQzQ3VXZIbmpva3NieVA2K2dYSE9wNQpCL1lKS0Njd2hwUTlWODAwUFRtc2FHK0kwZFliMjFEQW1SVnJTYVFsR1RtSnFmc1dpeFJQUERsRmRSOXg3dGNsCm85THV4ZDRzZnJmNG1GZHJuU1QzSndJWElzZXA2ZVRYWm1WK1NMREplR0ZmejVYSDlFUkRVRkJiTWxQSE1SbHUKNHp1anVIeENEa0JxSHhCMnpRZTA5YlN5TE1jNndtejJnL1dSM0xWakxpVkpRTWVaU0YySzduMmkzN29KY05EcgpEb29QSE40NWhESHNiRnFMU3FNRUU4eGUreFV5Y3JrVFdiWTRWQUpnUUFHMGpGV1prUmc4Ymg4RC9seXArNC81Cit3RHBUVW5lQU12L2VBdjBDQ01KWGIwdGZNbEhYYnBhSTVBNAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\n    server: https://172.25.42.235:6443\n  name: \"\"\ncontexts: null\ncurrent-context: \"\"\nkind: Config\npreferences: {}\nusers: null\n"}}
I0531 05:04:20.737406    8706 token.go:221] [discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again

原因

官方说是为了安全，初始化token后会在24小时候会被master删除,

解决办法

在master节点重新生成一个

[root@k8s-master01 data]# kubeadm token create
oumnnc.aqlxuvdbntlvzoiv

在master查看Token

[root@k8s-master01 data]# kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
oumnnc.aqlxuvdbntlvzoiv   23h         2021-06-01T17:03:25+08:00   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

重新加入，又报错

[root@k8s-node240 docker]# kubeadm join 172.25.42.235:6443 --token oumnnc.aqlxuvdbntlvzoiv --discovery-token-ca-cert-hash sha256:98de276382a20d872f579a74ee924b38d3705ebfcf268b1658a0b6ddb3dccaaa 
[preflight] Running pre-flight checks
error execution phase preflight: couldn't validate the identity of the API Server: cluster CA found in cluster-info ConfigMap is invalid: none of the public keys "sha256:46a4b70fca4adb7ec0315e25e0712ff1a4eaebf002e4ded963a981de071431fe" are pinned
To see the stack trace of this error execute with --v=5 or higher

原因：

如果找不到hash可以在master节点执行

[root@k8s-master01 data]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
46a4b70fca4adb7ec0315e25e0712ff1a4eaebf002e4ded963a981de071431fe

重新加入，成功

[root@k8s-node240 docker]# kubeadm join 172.25.42.235:6443 --token oumnnc.aqlxuvdbntlvzoiv --discovery-token-ca-cert-hash sha256:46a4b70fca4adb7ec0315e25e0712ff1a4eaebf002e4ded963a981de071431fe
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

成功加入集群

原文：k8s集群添加node节点