K8s 二进制扩容node节点方案
基于以下链接文档，k8s环境扩容
链接: https://pan.baidu.com/s/1_DopTLhdWujC3_6Ed3gyxg 提取码: eu2g
网络上的文档没有能够让你直接copy走拿去就能用的文档，但是大多数作者，都是根据自己的需求来进行的，二进制虽然不如kubeadm方式简单，一条两条命令基本搞定，但是基本上就是按照当时集群的部署文档copy的一遍而已，在扩容node节点的时候首先考虑几个问题。
1.node节点上的必备服务
docker，cni插件（calio，fannel），kubelet，kube-proxy
2.他们直接如果和k8s master建立关系
也就是kubelet和kube-proxy如何于apiserver建立连接呢？
通过下面的例子我们就能看到所有的组件的配置文件都有记录在与apiserver建立连接，kube-proxy和kubelet自然也不会例外。
[root@master kubernetes]# find . | grep -nr 6443
scheduler.kubeconfig:5: server: https://192.168.0.110:6443
kubelet.kubeconfig:5: server: https://192.168.0.110:6443
kube-proxy.kubeconfig:5: server: https://192.168.0.110:6443
bootstrap-kubelet.kubeconfig:5: server: https://192.168.0.110:6443
admin.kubeconfig:5: server: https://192.168.0.110:6443
controller-manager.kubeconfig:5: server: https://192.168.0.110:6443
除此以为需要证书文件，才可以建立连接，因为我们可以看到https://192.168.0.110:6443，这里用的则是https协议，另外k8s二进制的通信原理大家也都知道，ssl证书啦，tls协议啦，不懂的自己先百度一下吧。
3.如果单master节点扩容为多master节点呢
也简单，一通百通，改天我在把方案补充出来，与之前单master节点的部署方式并无太大区别，只是在扩容之前考虑一个问题为什么要多master，肯定大家都知道高可用什么什么的，那高可用的核心问题点在哪里？我们要了解两个最最最核心的组件，apiserver和etcd，apiserver是所有组件的枢纽，etcd存储k8s集群的元数据
apiserver要考虑的是因为所有的组件，任何的交互信息都要通过apiserver来进行，那apiserver如果不堪重负挂掉啦，那集群不就瘫掉了吗，所以多master的好处就显而易见，那要怎么做呢，其实也特别简单，我这里比如用什么haproxy+keeplived，通过master节点的ip:6443端口，实现轮询，这里的问题不久解决了吗。
etcd主要承担的则是数据的冗余，及时我挂了一个master节点，至少还有一个节点的etcd保存了集群的相关信息etcd 目前默认使用 2379 端口提供 HTTP API 服务， 2380 端口和 peer 通信，直接在配置中建立etcd集群的关系即可，如下所示
通过etcd的配置文件即可建立
initial-cluster: 'k8s-master01=https://192.168.0.107:2380,k8s-master02=https://192.168.0.108:2380,k8s-master03=https://192.168.0.109:2380
1.初始化系统配置
1.1 配置hosts
192.168.0.110 master1
192.168.0.102 node1
192.168.0.63 node2
192.168.0.55 node3
1.2 配置yum源
curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo
1.3必备工具安装
yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git wget jq psmisc -y
1.4节点关闭firewalld 、dnsmasq、selinux
systemctl disable --now firewalld
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
1.5节点关闭swap分区，fstab注释swap
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
1.6 节点同步时间
rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum install ntpdate -y
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo 'Asia/Shanghai' >/etc/timezone
ntpdate time2.aliyun.com
# 加入到crontab
*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com
1.7 节点配置limit：
ulimit -SHn 65535
vim /etc/security/limits.conf
# 末尾添加如下内容
* soft nofile 65535
* hard nofile 65535
* soft nproc 65535
* hard nproc 65535
* soft memlock unlimited
* hard memlock unlimited
1.8 免密登陆
#master节点执行
ssh-copy-id -i .ssh/id_rsa.pub 192.168.0.55
1.9 拷贝节点安装文件
#master节点执行
scp -r /root/k8s-ha-install/ node3:/root/
1.10内核升级
yum update -y --exclude=kernel*
cd /root
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm
cd /root && yum localinstall -y kernel-ml*
grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
grubby --default-kernel
reboot
uname -a
1.11更改主机名
hostname node3
hostnamectl set-hostname node3
bash
1.12安装ipvsadm
#所有节点配置ipvs模块，在内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack， 4.18以下使用nf_conntrack_ipv4即可
yum install ipvsadm ipset sysstat conntrack libseccomp -y
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
vim /etc/modules-load.d/ipvs.conf
# 加入以下内容
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
#启动服务
systemctl enable --now systemd-modules-load.service
#检查是否加载
[root@node3 ~]# lsmod | grep -e ip_vs -e nf_conntrack
nf_conntrack_netlink 40960 0
nfnetlink 16384 3 nf_conntrack_netlink,ip_set
ip_vs_ftp 16384 0
nf_nat 32768 2 nf_nat_ipv4,ip_vs_ftp
ip_vs_sed 16384 0
ip_vs_nq 16384 0
ip_vs_fo 16384 0
ip_vs_dh 16384 0
ip_vs_lblcr 16384 0
ip_vs_lblc 16384 0
ip_vs_wlc 16384 0
ip_vs_lc 16384 0
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 13
ip_vs 151552 37 ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_fo,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_vs_sed,ip_vs_ftp
nf_conntrack 143360 6 xt_conntrack,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink,ip_vs
nf_defrag_ipv6 20480 1 nf_conntrack
nf_defrag_ipv4 16384 1 nf_conntrack
libcrc32c 16384 3 nf_conntrack,nf_nat,ip_vs
1.13开启一些k8s集群中必须的内核参数，所有节点配置k8s内核：
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
#执行命令
sysctl --system && sysctl -p
2.安装Docker
yum install docker-ce-19.03.* -y
[root@node3 ~]# vim /etc/docker/daemon.json
#将文本改成如下，推荐从其他节点copy
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"insecure-registries": ["https://hub.test.com:18443"]
}
#启动docker
systemctl daemon-reload && systemctl enable --now docker
3.安装核心服务前准备
mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/ /etc/kubernetes/pki /opt/cni/bin
#在master节点执行
scp /usr/local/bin/kube{let,-proxy} node3:/usr/local/bin/
#复制相关证书到node节点
[root@master1 ~]# for FILE in etcd-ca.pem etcd.pem etcd-key.pem;do scp /etc/etcd/ssl/$FILE node3:/etc/etcd/ssl/; done
etcd-ca.pem 100% 1367 672.2KB/s 00:00
etcd.pem 100% 1452 1.3MB/s 00:00
etcd-key.pem 100% 1675 1.3MB/s 00:00
[root@master1 ~]# for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do scp /etc/kubernetes/$FILE node3:/etc/kubernetes/${FILE}; done
ca.pem 100% 1411 966.2KB/s 00:00
ca-key.pem 100% 1675 1.3MB/s 00:00
front-proxy-ca.pem 100% 1143 899.7KB/s 00:00
bootstrap-kubelet.kubeconfig 100% 2300 139.2KB/s 00:00
4.配置和安装kubelet
vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service
[Service]
ExecStart=/usr/local/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target
vim /etc/systemd/system/kubelet.service.d/10-kubelet.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS
vim /etc/kubernetes/kubelet-conf.yml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
#启动服务
systemctl daemon-reload
systemctl enable --now kubelet
#master执行
[root@master1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready <none> 29d v1.20.0
node1 Ready <none> 29d v1.20.0
node2 Ready <none> 29d v1.20.0
node3 Ready <none> 3m v1.20.0
问题梳理：
[root@node3 ~]# journalctl -xefu kubelet | grep fai
Jun 02 19:33:39 node3 kubelet[24182]: F0602 19:33:39.786231 24182 server.go:269] failed to run Kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"
Jun 02 19:33:39 node3 systemd[1]: Unit kubelet.service entered failed state.
Jun 02 19:33:39 node3 systemd[1]: kubelet.service failed.
Jun 02 19:33:54 node3 kubelet[24431]: E0602 19:33:54.914011 24431 kubelet.go:1271] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
Jun 02 19:33:54 node3 kubelet[24431]: E0602 19:33:54.922192 24431 nodelease.go:49] failed to get node "node3" when trying to set owner ref to the node lease: nodes "node3" not found
Jun 02 19:33:54 node3 kubelet[24431]: E0602 19:33:54.993972 24431 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get node info: node "node3" not found
Jun 02 19:33:55 node3 kubelet[24431]: W0602 19:33:55.518361 24431 driver-call.go:149] FlexVolume: driver call failed: executable: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds/uds, args: [init], error: fork/exec /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds/uds: no such file or directory, output: ""
##该问题检查/etc/docker/daemon.json文件配置，最好从其他健康的node节点直接复制
Jun 02 19:36:34 node3 kubelet[26467]: E0602 19:36:34.012055 26467 pod_workers.go:191] Error syncing pod 5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5 ("calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"), skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "back-off 10s restarting failed container=calico-node pod=calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"
Jun 02 19:36:42 node3 kubelet[26467]: E0602 19:36:42.628358 26467 pod_workers.go:191] Error syncing pod 5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5 ("calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"), skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "back-off 10s restarting failed container=calico-node pod=calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"
Jun 02 19:36:59 node3 kubelet[26467]: E0602 19:36:59.097043 26467 pod_workers.go:191] Error syncing pod 5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5 ("calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"), skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "back-off 20s restarting failed container=calico-node pod=calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"
Jun 02 19:37:02 node3 kubelet[26467]: E0602 19:37:02.628171 26467 pod_workers.go:191] Error syncing pod 5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5 ("calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"), skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "back-off 20s restarting failed container=calico-node pod=calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"
#无关紧要，因为我没安装kube-proxy，因此cni插件CrashLoopBackOff
#还有一些奇怪的日志，但是kubelet如果能够正常启动的话，先不要管这些，因为类似于kube-proxy和cni的依赖问题等等，安装后重启即可解决
5.配置和安装kube-proxy
master节点操作
scp /etc/kubernetes/kube-proxy.kubeconfig node3:/etc/kubernetes/kube-proxy.kubeconfig
scp kube-proxy/kube-proxy.conf node3:/etc/kubernetes/kube-proxy.conf
scp kube-proxy/kube-proxy.service node3:/usr/lib/systemd/system/kube-proxy.service
Node3节点启动kube-proxy
systemctl daemon-reload
systemctl enable --now kube-proxy
6.cni
我这里用的是calico，因为类似于系统插件都是用的daemonset方式部署，所以直接apply即可
kubectl apply -f calico-etcd.yaml
kubectl get po -n kube-system
如果容器状态异常可以使用kubectl describe 或者logs查看容器的日志
547
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
