【问题解决】删除node节点后如何把node节点重新加入

已于 2024-02-22 15:35:22 修改
阅读量675 收藏 9
点赞数 8
分类专栏： Kubernetes 文章标签： kubernetes node节点 加入集群
于 2024-02-21 22:33:59 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u011709380/article/details/136221914
版权

环境明细

  • docker版本： 25.0.3
  • kubeadm 版本：v1.25.0

1 在master节点删除node节点

[root@k8s-master ~]# kubectl delete nodes k8s-node-02
node "k8s-node-02" deleted
[root@k8s-master ~]# kubectl get  nodes  -o wide
NAME          STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
k8s-master    Ready    control-plane   2d    v1.25.0   10.10.10.100   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://25.0.3
k8s-node-01   Ready    <none>          47h   v1.25.0   10.10.10.177   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://25.0.3

2 在node节点重置kubeadm

- 正常重置kubeadm是：kubeadm reset  
- 由于我目前测试使用的事cri-docker，安装的k8s版本是v1.25.0，所以在重置的时候需要加入选项：kubeadm reset  --cri-socket /var/run/cri-dockerd.sock 如果不加入cri-dockerd的socket连接会有如下报错

在这里插入图片描述

[root@k8s-node-02 ~]# kubeadm reset  --cri-socket /var/run/cri-dockerd.sock
W0221 22:18:30.321890  108186 preflight.go:55] [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
W0221 22:18:32.838690  108186 removeetcdmember.go:85] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
W0221 22:18:37.911609  108186 cleanupnode.go:94] [reset] Failed to remove containers: [failed to stop running pod I0221: output: I0221 22:18:33.113881  108218 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
E0221 22:18:33.118604  108218 remote_runtime.go:205] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = [failed to get checkpoint for sandbox \"I0221\": checkpoint is not found, failed to get sandbox status: Error response from daemon: No such container: I0221]" podSandboxID="I0221"
time="2024-02-21T22:18:33+08:00" level=fatal msg="stopping the pod sandbox \"I0221\": rpc error: code = Unknown desc = [failed to get checkpoint for sandbox \"I0221\": checkpoint is not found, failed to get sandbox status: Error response from daemon: No such container: I0221]"
: exit status 1, failed to stop running pod 22:18:32.911038: output: I0221 22:18:33.310362  108238 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
E0221 22:18:33.318550  108238 remote_runtime.go:205] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: could not retrieve port mappings: invalid key: \"22:18:32.911038\"" podSandboxID="22:18:32.911038"
time="2024-02-21T22:18:33+08:00" level=fatal msg="stopping the pod sandbox \"22:18:32.911038\": rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: could not retrieve port mappings: invalid key: \"22:18:32.911038\""
: exit status 1, failed to remove running container 108192: output: I0221 22:18:33.704212  108298 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
getting sandbox status of pod "108192": rpc error: code = Unknown desc = Error response from daemon: No such container: 108192
: exit status 1, failed to stop running pod util_unix.go:103]: output: I0221 22:18:33.925259  108319 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
E0221 22:18:33.933909  108319 remote_runtime.go:205] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: could not retrieve port mappings: invalid key: \"util_unix.go:103]\"" podSandboxID="util_unix.go:103]"
time="2024-02-21T22:18:33+08:00" level=fatal msg="stopping the pod sandbox \"util_unix.go:103]\": rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: could not retrieve port mappings: invalid key: \"util_unix.go:103]\""
: exit status 1, failed to stop running pod "Using: output: I0221 22:18:34.125710  108339 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
E0221 22:18:34.131325  108339 remote_runtime.go:205] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = [failed to get checkpoint for sandbox \"\\\"Using\": invalid key: \"\\\"Using\", failed to get sandbox status: Error response from daemon: No such container: \"Using]" podSandboxID="\"Using"
time="2024-02-21T22:18:34+08:00" level=fatal msg="stopping the pod sandbox \"\\\"Using\": rpc error: code = Unknown desc = [failed to get checkpoint for sandbox \"\\\"Using\": invalid key: \"\\\"Using\", failed to get sandbox status: Error response from daemon: No such container: \"Using]"
: exit status 1, failed to remove running container this: output: I0221 22:18:34.543335  108393 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
getting sandbox status of pod "this": rpc error: code = Unknown desc = Error response from daemon: No such container: this
: exit status 1, failed to remove running container endpoint: output: I0221 22:18:34.920105  108445 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
getting sandbox status of pod "endpoint": rpc error: code = Unknown desc = Error response from daemon: No such container: endpoint
: exit status 1, failed to remove running container is: output: I0221 22:18:35.303518  108498 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
getting sandbox status of pod "is": rpc error: code = Unknown desc = Error response from daemon: No such container: is
: exit status 1, failed to stop running pod deprecated,: output: I0221 22:18:35.502281  108518 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
E0221 22:18:35.509564  108518 remote_runtime.go:205] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: could not retrieve port mappings: invalid key: \"deprecated,\"" podSandboxID="deprecated,"
time="2024-02-21T22:18:35+08:00" level=fatal msg="stopping the pod sandbox \"deprecated,\": rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: could not retrieve port mappings: invalid key: \"deprecated,\""
: exit status 1, failed to remove running container please: output: I0221 22:18:35.898679  108571 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
getting sandbox status of pod "please": rpc error: code = Unknown desc = Error response from daemon: No such container: please
: exit status 1, failed to remove running container consider: output: I0221 22:18:36.283251  108623 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
getting sandbox status of pod "consider": rpc error: code = Unknown desc = Error response from daemon: No such container: consider
: exit status 1, failed to remove running container using: output: I0221 22:18:36.687468  108675 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
getting sandbox status of pod "using": rpc error: code = Unknown desc = Error response from daemon: No such container: using
: exit status 1, failed to remove running container full: output: I0221 22:18:37.089088  108728 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
getting sandbox status of pod "full": rpc error: code = Unknown desc = Error response from daemon: No such container: full
: exit status 1, failed to stop running pod URL: output: I0221 22:18:37.266203  108749 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
E0221 22:18:37.272216  108749 remote_runtime.go:205] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = [failed to get checkpoint for sandbox \"URL\": checkpoint is not found, failed to get sandbox status: Error response from daemon: No such container: URL]" podSandboxID="URL"
time="2024-02-21T22:18:37+08:00" level=fatal msg="stopping the pod sandbox \"URL\": rpc error: code = Unknown desc = [failed to get checkpoint for sandbox \"URL\": checkpoint is not found, failed to get sandbox status: Error response from daemon: No such container: URL]"
: exit status 1, failed to stop running pod format": output: I0221 22:18:37.466599  108769 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
E0221 22:18:37.474168  108769 remote_runtime.go:205] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: could not retrieve port mappings: invalid key: \"format\\\"\"" podSandboxID="format\""
time="2024-02-21T22:18:37+08:00" level=fatal msg="stopping the pod sandbox \"format\\\"\": rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: could not retrieve port mappings: invalid key: \"format\\\"\""
: exit status 1, failed to stop running pod endpoint="/var/run/cri-dockerd.sock": output: I0221 22:18:37.696206  108789 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
E0221 22:18:37.704741  108789 remote_runtime.go:205] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: could not retrieve port mappings: invalid key: \"endpoint=\\\"/var/run/cri-dockerd.sock\\\"\"" podSandboxID="endpoint=\"/var/run/cri-dockerd.sock\""
time="2024-02-21T22:18:37+08:00" level=fatal msg="stopping the pod sandbox \"endpoint=\\\"/var/run/cri-dockerd.sock\\\"\": rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: could not retrieve port mappings: invalid key: \"endpoint=\\\"/var/run/cri-dockerd.sock\\\"\""
: exit status 1, failed to stop running pod URL="unix:///var/run/cri-dockerd.sock": output: I0221 22:18:37.903473  108809 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/cri-dockerd.sock" URL="unix:///var/run/cri-dockerd.sock"
E0221 22:18:37.909505  108809 remote_runtime.go:205] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = [failed to get checkpoint for sandbox \"URL=\\\"unix:///var/run/cri-dockerd.sock\\\"\": invalid key: \"URL=\\\"unix:///var/run/cri-dockerd.sock\\\"\", failed to get sandbox status: Error response from daemon: No such container: URL=\"unix:/var/run/cri-dockerd.sock\"]" podSandboxID="URL=\"unix:///var/run/cri-dockerd.sock\""
time="2024-02-21T22:18:37+08:00" level=fatal msg="stopping the pod sandbox \"URL=\\\"unix:///var/run/cri-dockerd.sock\\\"\": rpc error: code = Unknown desc = [failed to get checkpoint for sandbox \"URL=\\\"unix:///var/run/cri-dockerd.sock\\\"\": invalid key: \"URL=\\\"unix:///var/run/cri-dockerd.sock\\\"\", failed to get sandbox status: Error response from daemon: No such container: URL=\"unix:/var/run/cri-dockerd.sock\"]"
: exit status 1]
[reset] Deleting contents of directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

3 重新再master节点生成加入集群的token

# 获取token信息
[root@k8s-master ~]# kubeadm  token list
TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
bkld6x.7bn54sic1tbry69k   6h          2024-02-22T14:12:06Z   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

# 获取ca证书sha256编码的hash值
[root@k8s-master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
18ce7af434e99a8edd527be3ea83603e68c50fb676b7cf9eeeb169f5ded88b7f

# 如果token信息失效可以重新获取token
[root@k8s-master ~]# kubeadm token create --print-join-command
kubeadm join 10.10.10.100:6443 --token bkld6x.7bn54sic1tbry69k --discovery-token-ca-cert-hash sha256:18ce7af434e99a8edd527be3ea83603e68c50fb676b7cf9eeeb169f5ded88b7f

4 node节点重新加入到集群中

  • 同理在此处加入节点的时候也需要添加cri-docker的socket连接
[root@k8s-node-02 ~]# kubeadm join 10.10.10.100:6443 --token bkld6x.7bn54sic1tbry69k --discovery-token-ca-cert-hash sha256:18ce7af434e99a8edd527be3ea83603e68c50fb676b7cf9eeeb169f5ded88b7f   --cri-socket /var/run/cri-dockerd.sock
W0221 22:19:16.247305  108905 initconfiguration.go:119] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/cri-dockerd.sock". Please update your configuration!
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

5 在master节点查看node加入情况

[root@k8s-master ~]# kubectl get  nodes  -o wide
NAME          STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
k8s-master    Ready    control-plane   2d    v1.25.0   10.10.10.100   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://25.0.3
k8s-node-01   Ready    <none>          47h   v1.25.0   10.10.10.177   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://25.0.3
k8s-node-02   Ready    <none>          12s   v1.25.0   10.10.10.112   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://25.0.3
嘻哈记
关注
  • 8
    点赞
  • 踩
  • 9
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Node节点树.png
05-20
Node节点树.png
在Linux上安装运行pi node节点
01-14
目前pi node节点99%以上都是运行在window10+docker环境。 但我们知道Pi node节点底层是采用stellar恒星区块链源码改进而来的。因此Pi node节点肯定可以运行在linux上。 本文介绍如何在Linux上直接运行pi node节点。 ...
k8s集群—node节点的删除与添加
我是小bā吖的博客
02-10 3294
在搭建集群过程中，有时候会遇到一个节点处于ready状态，另一个节点处于notready状态，需要把node节点从集群中删除后再次加入。
k8s的master节点上删除node节点，重新再接入
最新发布
qq_30467221的博客
05-08 162
master删除node工作节点：生成加入节点命令：把工作节点重新加入节点：
k8s驱逐节点、重新加入、重装k8s
qq_52787609的博客
05-08 375
安装、卸载Kubernetes。node节点重新加入集群的操作。
删除原有node节点，添加新的节点到kubernets集群中
qq_45034687的博客
01-17 502
在一个集群中有些节点，会莫名出现一些故障，导致在master节点中显示notready状态，一般根据日志信息进行troubleshooting即可，但是有些时候，就是无法恢复，我们就得将原有故障节点删除，重新加入集群当中。一.删除原有node节点（master节点上执行）二、生成token口令（master上执行）获取node节点信息，看新节点是否加入集群。成功看到新节点处于ready状态即为成功！四、检验（master节点）三、node节点加入集群。
K8S 重新删除和重新加入某个node
nvd11的专栏
06-18 353
当我修改某个node的hostname后， k8s就显示这个node状态时not ready了这时只能删除和重新加入这个node(新的node 里面并没有部署任何pod）
k8s集群节点删除再重新加入+ERROR
tuboshu1234的博客
01-08 490
从集群中删除node 再将node 重新加入集群
【Ambari】 NameNode HA 回滚操作
康师傅没有眼泪
08-28 924
目录 一、问题引出 1. 角色查看 二、解决步骤 1. 清理额外的NAMENODE 2. 恢复secondary_namenode 3. NameNodeHA 启动 一、问题引出 启用HDFS HA过程中，被人为的中断或意外的中断导致Secondary NameNone还没有被删除。我是 namenode 启动花了 15 分钟 后 我就强制退出了， 真的是手贱啊，查了网上的文章失败后 可以retry 就可以了，自己挖坑自己填。 参考文档：https://www.jianshu.com/..
Pi Node节点运行，接入数64的建议.txt
02-24
响应官方人人节点
drupal实现在node节点的评论下面添加内容的方法
09-28
主要为大家介绍了drupal实现在node节点的评论下面添加内容的方法,涉及相关函数的修改与hook函数的使用,具有一定的借鉴价值,需要的朋友可以参考下
kubectl-node-shell：通过kubectl执行进入节点
02-03
在Kubernetes环境中，管理员和开发人员经常需要对集群中的节点进行直接操作，例如排查问题、调试服务或执行系统级...理解并熟练使用`kubectl-node-shell`将有助于提升你在Kubernetes环境中的工作效率和问题解决能力。
k8s集群master和node添加
贾文超的博客
02-01 960
k8s集群添加node节点和master节点
使用ambari扩展hdfs节点
妙妙的博客
04-26 4287
1.Ambari添加节点：Hosts->Action->Add New Hosts 2.在已有节点及Ambari server上添加节点IP及hostname对应关系（/etc/hosts 文件），相当于追加新的节点信息，每个服务器保持一致； 修改主机的hostname; vi /etc/sysconfig/network; service network restart 新机器会
K8S集群node的删除和添加
强哥的微博
01-04 1572
【代码】K8S集群node的删除和添加。
k8s二进制集群中删除node节点,并重新加入集群
qq_27156945的博客
10-21 3125
k8s二进制集群中删除node节点,并重新加入集群一、删除node04节点1.查看当前集群的节点信息2.使用kubectl命令删除node节点3.查看当前集群的节点信息4.在node节点上彻底删除集群信息5.node节点停止kubelet服务二、node节点重新加入集群1.重启kubelet服务 原有环境说明 主机名 系统版本 ip地址 docker version kubelet ver...
kubeadm join timeout
qq_15138049的博客
11-11 955
[root@hadoop02 kubernetes]# kubeadm join 192.168.153.103:6443 --token d4zd6v.xibtlpcycy8jlb70 --discovery-token-ca-cert-hash sha256:8f7643fc558b209f226eef1633f8008bd37a7ed551af5525e58a6ea38849a5b8 [preflight] Running pre-flight checks error execution phase
当etcd被误删后，kubeadm安装的集群快速恢复集群
red_sky_blue的专栏
03-09 4568
etcd被删除后，之前集群的数据如果没有备份，数据将全部丢失，即使集群恢复后，也是一个新的集群。 【现象】： # kubectl get nodes No resources found 【恢复操作】 在master节点上 # kubeadm reset [reset] Reading configuration from the cluster... [reset] FYI: You can look at this config file with 'kubectl -n kube-
k8s中node节点宕机后pod怎么迁移
03-31
控制器会检测到node节点宕机后，将该节点上的pod标记为不可用，然后将它们重新调度到其他可用节点上，以确保服务的可用性。这个过程中，k8s会自动选择一个最佳的节点来重新调度pod，并确保pod之间的负载均衡。因此，...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值