kubeadm 重置集群后遇到x509问题

kubeadm重置集群

• 重置集群命令

  [root@master-node ~]# kubeadm reset -f
  [reset] Reading configuration from the cluster...
  [reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
  W1125 15:31:28.439103    2709 reset.go:73] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get config map: Get https://***.***.***.***:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config: dial tcp ***.***.***.***:6443: connect: connection refused
  [preflight] Running pre-flight checks
  W1125 15:31:28.439490    2709 reset.go:234] [reset] No kubeadm config, using etcd pod spec to get data directory
  [reset] Stopping the kubelet service
  [reset] unmounting mounted directories in "/var/lib/kubelet"
  [reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
  [reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
  [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
  
  The reset process does not reset or clean up iptables rules or IPVS tables.
  If you wish to reset iptables, you must do so manually.
  For example:
  iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
  
  If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
  to reset your system's IPVS tables.
  
  

  • 重置成功

kubeadm初始化集群

• 初始化集群

  [root@master-node ~]# kubeadm init
  I1125 15:32:55.630416    2924 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  I1125 15:32:55.630789    2924 version.go:97] falling back to the local client version: v1.14.0
  [init] Using Kubernetes version: v1.14.0
  [preflight] Running pre-flight checks
  	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
  [preflight] Pulling images required for setting up a Kubernetes cluster
  [preflight] This might take a minute or two, depending on the speed of your internet connection
  [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
  [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
  [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
  [kubelet-start] Activating the kubelet service
  [certs] Using certificateDir folder "/etc/kubernetes/pki"
  [certs] Generating "front-proxy-ca" certificate and key
  [certs] Generating "front-proxy-client" certificate and key
  [certs] Generating "ca" certificate and key
  [certs] Generating "apiserver" certificate and key
  [certs] apiserver serving cert is signed for DNS names [master-node kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 ***.***.***.***]
  [certs] Generating "apiserver-kubelet-client" certificate and key
  [certs] Generating "etcd/ca" certificate and key
  [certs] Generating "etcd/healthcheck-client" certificate and key
  [certs] Generating "etcd/server" certificate and key
  [certs] etcd/server serving cert is signed for DNS names [master-node localhost] and IPs [***.***.***.*** 127.0.0.1 ::1]
  [certs] Generating "etcd/peer" certificate and key
  [certs] etcd/peer serving cert is signed for DNS names [master-node localhost] and IPs [***.***.***.*** 127.0.0.1 ::1]
  [certs] Generating "apiserver-etcd-client" certificate and key
  [certs] Generating "sa" key and public key
  [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
  [kubeconfig] Writing "admin.conf" kubeconfig file
  [kubeconfig] Writing "kubelet.conf" kubeconfig file
  [kubeconfig] Writing "controller-manager.conf" kubeconfig file
  [kubeconfig] Writing "scheduler.conf" kubeconfig file
  [control-plane] Using manifest folder "/etc/kubernetes/manifests"
  [control-plane] Creating static Pod manifest for "kube-apiserver"
  [control-plane] Creating static Pod manifest for "kube-controller-manager"
  [control-plane] Creating static Pod manifest for "kube-scheduler"
  [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
  [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
  [apiclient] All control plane components are healthy after 18.504163 seconds
  [upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
  [kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster
  [upload-certs] Skipping phase. Please see --experimental-upload-certs
  [mark-control-plane] Marking the node master-node as control-plane by adding the label "node-role.kubernetes.io/master=''"
  [mark-control-plane] Marking the node master-node as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
  [bootstrap-token] Using token: hfhd7f.t0zoh40o2eogr9ld
  [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
  [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
  [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
  [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
  [bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace
  [addons] Applied essential addon: CoreDNS
  [addons] Applied essential addon: kube-proxy
  
  Your Kubernetes control-plane has initialized successfully!
  To start using your cluster, you need to run the following as a regular user:
  
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
  
  You should now deploy a pod network to the cluster.
  Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
  
  Then you can join any number of worker nodes by running the following on each as root:
  。。。 。。。
  	
  

  • 执行命令

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
  

• 初始化成功（接下来执行kubectl命令会报x509问题）

• systemctl status kubelet 验证kubelet启动成功

问题：执行kubectl命令报错

• x509问题

  [root@master-node ~]# kubectl get nodes 
  Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
  

• 原因：重置集群后$HOME/.kube目录没有被删除掉，导致新的集群认证失败

  [root@master-node ~]# kubeadm reset -f
  [reset] Reading configuration from the cluster...
  [reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
  [preflight] Running pre-flight checks
  [reset] Removing info for node "master-node" from the ConfigMap "kubeadm-config" in the "kube-system" Namespace
  W1125 16:04:06.696024   16390 reset.go:158] [reset] failed to remove etcd member: error syncing endpoints with etc: etcdclient: no available endpoints
  .Please manually remove this etcd member using etcdctl
  [reset] Stopping the kubelet service
  [reset] unmounting mounted directories in "/var/lib/kubelet"
  [reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
  [reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
  [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
  
  The reset process does not reset or clean up iptables rules or IPVS tables.
  If you wish to reset iptables, you must do so manually.
  For example:
  iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
  
  If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
  to reset your system's IPVS tables.
  
  [root@master-node ~]# cd /root/.kube/
  [root@master-node .kube]# ls
  cache  config  http-cache
  [root@master-node .kube]# 
  
  

• 解决
  • 1、手动删除改目录（建议在重置后执行删除）

  rm -rf $HOME/.kube
  

  • 2、再次执行命令（如果$HOME/.kube以清除，可省略）

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
  

• 结果
  x509问题处理完成

  [root@master-node ~]# kubectl get nodes
  NAME          STATUS     ROLES    AGE   VERSION
  master-node   NotReady   master   17s   v1.14.0