二进制 Kubernetes新增node节点以及问题记录

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

关闭selinux

sed -i 's/enforcing/disabled/' /etc/selinux/config  # 永久
setenforce 0  # 临时

关闭swap

swapoff -a  # 临时
sed -ri 's/.*swap.*/#&/' /etc/fstab    # 永久

根据规划设置主机名

hostnamectl set-hostname <hostname>

在master添加hosts

cat >> /etc/hosts << EOF
192.168.111.60 master
192.168.111.61 node1
192.168.111.62 node2
192.168.111.63 node3
EO

F

将桥接的IPv4流量传递到iptables的链

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system  # 生效

时间同步

yum install ntpdate -y
ntpdate time.windows.com

dockr安装

yum install wget -y
wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz

解压二进制包

tar zxvf docker-19.03.9.tgz
mv docker/* /usr/bin

解压二进制包

cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF

创建配置文件

mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"],
  "insecure-registries": ["192.168.111.60:5000"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  }
}

EOF
启动并设置开机启动

systemctl daemon-reload
systemctl start docker
systemctl enable docker

新增node节点
拷贝已部署好的Node相关文件到新节点192.168.111.63

scp -r /opt/kubernetes root@192.168.111.63:/opt/
scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.111.63:/usr/lib/systemd/system
scp -r /opt/cni/ root@192.168.111.63:/opt/
scp /opt/kubernetes/ssl/ca.pem root@192.168.111.63:/opt/kubernetes/ssl
从master节点复制镜像
scp image.tar.gz root@192.168.111.63:/root/
tar zxf image.tar.gz
docker load -i nginx-ingress-controller.tar 
docker load -i coredns.tar 
docker load -i flannel.tar 
docker load -i metrics-scraper.tar 
docker load -i pause.tar

删除kubelet证书和kubeconfig文件

rm /opt/kubernetes/cfg/kubelet.kubeconfig   y
rm -f /opt/kubernetes/ssl/kubelet*

修改主机名

vi /opt/kubernetes/cfg/kubelet.conf
--hostname-override=node3

vi /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: node3
metricsBindAddress: 192.168.111.63:10249

在master执行
配置tls 基于bootstrap自动颁发证书

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

启动并设置开机启动

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet
systemctl start kube-proxy
systemctl enable kube-proxy

在Master上批准新Node kubelet证书申请

kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-O9xAa4LZv8u-WNIqJOR0cSQ_DzS5I_uOTCcUeQ8bohA   20s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending
[root@master60 ~]# kubectl certificate approve node-csr-O9xAa4LZv8u-WNIqJOR0cSQ_DzS5I_uOTCcUeQ8bohA
certificatesigningrequest.certificates.k8s.io/node-csr-O9xAa4LZv8u-WNIqJOR0cSQ_DzS5I_uOTCcUeQ8bohA approved

查看Node状态

kubectl get nodes

遇到问题：

 failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "kubelet-bootstrap" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope

原因是：kubelet-bootstrap并没有权限创建证书。所以要创建这个用户的权限并绑定到这个角色上。

解决方法是在master上执行kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

kubelet 启动失败

systemctl status kubelet -l
node "node3" not found
Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

本机host文件没有自己的

192.168.111.63  node3

添加即可

docker 启动失败
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
没有daemon.json文件就自己新建一个:先写入最简的，在别的机器使用正常，在刚建的机器不一定正常

mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"],
  "insecure-registries": ["192.168.111.60:5000"],
}