目录
前言
在上一篇文章中,我们已经做好了前期的集群规划和Etcd数据库的集群部署,现在我们将对Master管理监控节点和Node工作节点配置核心组件。
一、Master管理节点需要安装的组件介绍
Master 是 K8S 的集群控制节点,负责整个集群的管理和控制,基本上 K8S 所有的控制命令都是发给它,它来负责具体的执行过程。下面是Master节点上的组件介绍:
kube-apiserver:
K8S集群中的所有资源的访问以及变更,都要依靠kube-apiserver这个重要的核心的组件。
它是集群的统一入口,各组件协调者,以 HTTP Rest 提供接口服务,所有对象资源的增、删、改、查和监听操作都交给 kube-apiserver 处理后再提交给 Etcd 数据库存储。
- 提供用户、集群中的不同部分和集群外部组件相互通信,各模块之间的数据交互和通信的枢纽
- 查询和操纵集群中的Pod、Namespace、ConfigMap 和 Event 等对象的状态
- 提供集群管理的 REST API 接口,包括认证授权、数据校验以及集群状态变更等
- 是资源配额控制的入口
- 拥有完备的集群安全机制
kube-controller-manager:
是 K8S 里所有资源对象的自动化控制中心,处理集群中常规后台任务,一个资源对应一个控制器,而 kube-controller-manager 就是负责管理这些控制器的。
Kubernetes 控制器管理器是一个守护程序,它嵌入了 Kubernetes 附带的核心控制循环。在机器人和自动化应用中,控制回路是调节系统状态的非终止回路。在 Kubernetes 中,控制器是一个控制循环,它通过 apiserver 监视集群的共享状态,并进行更改,尝试将当前状态移动到所需状态。目前 Kubernetes 附带的控制器示例包括复制控制器、端点控制器、命名空间控制器和服务帐户控制器。
kube-scheduler:
Kubernetes 调度程序是一个控制平面进程,它将 Pod 分配给Node节点。调度程序根据约束和可用资源确定调度队列中每个 Pod 的有效放置节点。然后,调度程序对每个有效节点进行排名,并将 Pod 绑定到合适的节点。
下图是在网上找到的k8s的框架图:
二、配置Master管理节点各组件模块
1.配置kube-apiserver组件
(1)自签ApiServer SSL证书:
先给apiserver签发一份SSL证书,整个过程和上一篇文章中给Etcd自签证书步骤大同小异。
①创建所需目录:
- /k8s/kubernetes/ssl(自签证书存放目录)
- /k8s/kubernetes/cfg(配置文件存放目录)
- /k8s/kubernetes/bin(执行程序存放目录)
- /k8s/kubernetes/logs(日志文件存放目录)
[root@k8s-master-1 ~]# cd /
[root@k8s-master-1 /]# mkdir -p /k8s/kubernetes/{ssl,cfg,bin,logs}
[root@k8s-master-1 /]# cd k8s/kubernetes/
[root@k8s-master-1 kubernetes]# ll
total 0
drwxr-xr-x 2 root root 6 Mar 23 18:42 bin
drwxr-xr-x 2 root root 6 Mar 23 18:42 cfg
drwxr-xr-x 2 root root 6 Mar 23 18:42 logs
drwxr-xr-x 2 root root 6 Mar 23 18:42 ssl
②进入ssl目录创建CA配置文件ca-config.json:
[root@k8s-master-1 ssl]# vim ca-config.json
[root@k8s-master-1 ssl]# cat ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
③配置CA证书签名请求文件ca-csr.json:
[root@k8s-master-1 ssl]# vim ca-csr.json
[root@k8s-master-1 ssl]# cat ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Shanghai",
"L": "Shanghai",
"O": "kubernetes",
"OU": "System"
}
],
"ca": {
"expiry": "87600h"
}
}
[root@k8s-master-1 ssl]# ll
total 8
-rw-r--r-- 1 root root 292 Mar 23 18:46 ca-config.json
-rw-r--r-- 1 root root 262 Mar 23 18:46 ca-csr.json
④生成CA证书和私钥:
[root@k8s-master-1 ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2022/03/23 18:47:53 [INFO] generating a new CA key and certificate from CSR
2022/03/23 18:47:53 [INFO] generate received request
2022/03/23 18:47:53 [INFO] received CSR
2022/03/23 18:47:53 [INFO] generating key: rsa-2048
2022/03/23 18:47:53 [INFO] encoded CSR
2022/03/23 18:47:53 [INFO] signed certificate with serial number 72946279832304111272771850607707125708900150427
[root@k8s-master-1 ssl]# ll
total 20
-rw-r--r-- 1 root root 292 Mar 23 18:46 ca-config.json
-rw-r--r-- 1 root root 1013 Mar 23 18:47 ca.csr
-rw-r--r-- 1 root root 262 Mar 23 18:46 ca-csr.json
-rw------- 1 root root 1675 Mar 23 18:47 ca-key.pem
-rw-r--r-- 1 root root 1383 Mar 23 18:47 ca.pem
⑤配置证书签名请求文件kubernetes-csr.json:
[root@k8s-master-1 ssl]# vim kubernetes-csr.json
[root@k8s-master-1 ssl]# cat kubernetes-csr.json
{
"CN": "kubernetes",
"hosts": [ #指定会直接访问apiserver的IP列表,一般需指定etcd集群、kubernetes master 集群的主机IP和kubernetes服务的服务IP
"127.0.0.1",
"10.0.0.1",
"192.168.61.161",
"192.168.61.162",
"192.168.61.163",
"192.168.61.164",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "kubernetes",
"OU": "System"
}
]
}
⑥为kubernetes生成证书和私钥:
[root@k8s-master-1 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
2022/03/23 18:57:35 [INFO] generate received request
2022/03/23 18:57:35 [INFO] received CSR
2022/03/23 18:57:35 [INFO] generating key: rsa-2048
2022/03/23 18:57:36 [INFO] encoded CSR
2022/03/23 18:57:36 [INFO] signed certificate with serial number 692567500574363843249097959324089300840804006652
2022/03/23 18:57:36 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
(2)拉取二进制包,部署kube-apiserver:
①下载二进制包:kubernetes-v1.16.2-server-linux-amd64.zip的下载链接 提取码:n3as
下载好后,用xshell等远程连接工具上传到Linux上,然后解压到指定目录,接着给予运行权限:
[root@k8s-master-1 ~]# rz
[root@k8s-master-1 ~]# ll
total 124368
-rw-r--r-- 1 root root 127350862 Apr 19 18:14 kubernetes-v1.16.2-server-linux-amd64.zip
[root@k8s-master-1 ~]# unzip -d /k8s/kubernetes/bin/ kubernetes-v1.16.2-server-linux-amd64.zip
[root@k8s-master-1 ~]# cd /k8s/kubernetes/bin/
[root@k8s-master-1 bin]# ll
total 458404
-rw-r--r-- 1 root root 120672256 Mar 29 21:37 kube-apiserver
-rw-r--r-- 1 root root 110063616 Mar 29 21:37 kube-controller-manager
-rw-r--r-- 1 root root 44036096 Mar 29 21:37 kubectl
-rw-r--r-- 1 root root 113292024 Mar 29 21:37 kubelet
-rw-r--r-- 1 root root 38383616 Mar 29 21:37 kube-proxy
-rw-r--r-- 1 root root 42954752 Mar 29 21:37 kube-scheduler
[root@k8s-master-1 bin]# chmod +755 kube*
[root@k8s-master-1 bin]# ll
total 458404
-rwxr-xr-x 1 root root 120672256 Mar 29 21:37 kube-apiserver
-rwxr-xr-x 1 root root 110063616 Mar 29 21:37 kube-controller-manager
-rwxr-xr-x 1 root root 44036096 Mar 29 21:37 kubectl
-rwxr-xr-x 1 root root 113292024 Mar 29 21:37 kubelet
-rwxr-xr-x 1 root root 38383616 Mar 29 21:37 kube-proxy
-rwxr-xr-x 1 root root 42954752 Mar 29 21:37 kube-scheduler
把kubectl复制到/usr/local/bin/目录下:
[root@k8s-master-1 bin]# cp kubectl /usr/local/bin/
[root@k8s-master-1 bin]# cd /usr/local/bin/
[root@k8s-master-1 bin]# ll
total 61812
-rwxr-xr-x 1 root root 10376657 Jan 15 15:59 cfssl
-rw-r--r-- 1 root root 6595195 Jan 15 16:13 cfssl-certinfo
-rwxr-xr-x 1 root root 2277873 Jan 15 16:07 cfssljson
-rwxr-xr-x 1 root root 44036096 Mar 29 21:38 kubectl
②配置Node令牌文件token.csv:
master的apiserver启用TLS认证之后,Node工作节点的kubelet组件想要加入集群,就必须使用CA签发的有效证书,才能与apiserver进行通信;而一旦Node工作节点增多,签署证书会比较繁琐,为了简便,就产生了TLS Bootstrap机制,kubelet会以一个低权限用户自动向apiserver申请证书,kubelet的证书由apiserver动态签署。因此要提前为apiserver生成一个令牌文件,在后面的步骤中Node节点会使用到。
随机生成token字符串:
[root@k8s-master-1 kubernetes]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
e2b20b5979e898e33c644471e53ee8a2
注意:apiserver配置的token一定要和Node节点的bootstrap.kubeconfig配置保持一致!
创建token.csv:格式“token,用户,UID,用户组”
[root@k8s-master-1 kubernetes]# vim /k8s/kubernetes/cfg/token.csv
[root@k8s-master-1 kubernetes]# cat /k8s/kubernetes/cfg/token.csv
e2b20b5979e898e33c644471e53ee8a2,kubelet-bootstrap,10001,"system:node-bootstrapper"
[root@k8s-master-1 kubernetes]# cd cfg/
[root@k8s-master-1 cfg]# cat token.csv
e2b20b5979e898e33c644471e53ee8a2,kubelet-bootstrap,10001,"system:node-bootstrapper"
③创建kube-apiserver的配置文件kube-apiserver.conf:
配置项用途请查看:官方文档
[root@k8s-master-1 cfg]# vim /k8s/kubernetes/cfg/kube-apiserver.conf
[root@k8s-master-1 cfg]# cat /k8s/kubernetes/cfg/kube-apiserver.conf
KUBE_APISERVER_OPTS="--etcd-servers=https://192.168.61.161:2379,https://192.168.61.162:2379,https://192.168.61.163:2379 \
--bind-address=192.168.61.161 \
--secure-port=6443 \
--advertise-address=192.168.61.161 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--service-node-port-range=30000-32767 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/k8s/kubernetes/cfg/token.csv \
--kubelet-client-certificate=/k8s/kubernetes/ssl/kubernetes.pem \
--kubelet-client-key=/k8s/kubernetes/ssl/kubernetes-key.pem \
--tls-cert-file=/k8s/kubernetes/ssl/kubernetes.pem \
--tls-private-key-file=/k8s/kubernetes/ssl/kubernetes-key.pem \
--client-ca-file=/k8s/kubernetes/ssl/ca.pem \
--service-account-key-file=/k8s/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/k8s/etcd/ssl/ca.pem \
--etcd-certfile=/k8s/etcd/ssl/etcd.pem \
--etcd-keyfile=/k8s/etcd/ssl/etcd-key.pem \
--v=2 \
--logtostderr=false \
--log-dir=/k8s/kubernetes/logs \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/k8s/kubernetes/logs/k8s-audit.log"
[root@k8s-master-1 cfg]# ll
total 8
-rw-r--r-- 1 root root 1261 Mar 29 21:59 kube-apiserver.conf
-rw-r--r-- 1 root root 84 Mar 29 21:50 token.csv
配置参数解释:
--etcd-servers:etcd 集群地址
--bind-address:apiserver 监听的地址,一般配主机IP
--secure-port:监听的端口
--advertise-address:集群通告地址,其它Node节点通过这个地址连接 apiserver,不配置则使用 --bind-address
--service-cluster-ip-range:Service 的 虚拟IP范围,以CIDR格式标识,该IP范围不能与物理机的真实IP段有重合。
--service-node-port-range:Service 可映射的物理机端口范围,默认30000-32767
--admission-control:集群的准入控制设置,各控制模块以插件的形式依次生效,启用RBAC授权和节点自管理
--authorization-mode:授权模式,包括:AlwaysAllow,AlwaysDeny,ABAC(基于属性的访问控制),Webhook,RBAC(基于角色的访问控制),Node(专门授权由 kubelet 发出的API请求)。(默认值"AlwaysAllow")。
--enable-bootstrap-token-auth:启用TLS bootstrap功能
--token-auth-file:这个文件将被用于通过令牌认证来保护API服务的安全端口。
--v:指定日志级别,0~8,越大日志越详细
④配置kube-apiserver.service服务:
[root@k8s-master-1 ~]# vim /usr/lib/systemd/system/kube-apiserver.service
[root@k8s-master-1 ~]# cat /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-apiserver.conf
ExecStart=/k8s/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
⑤启动kube-apiserver服务组件,并设置开机自启动:
[root@k8s-master-1 ~]# systemctl daemon-reload
[root@k8s-master-1 ~]# systemctl start kube-apiserver
[root@k8s-master-1 ~]# systemctl enable kube-apiserver
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.
[root@k8s-master-1 ~]# systemctl status kube-apiserver
● kube-apiserver.service - Kubernetes API Server
Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2022-03-30 14:54:46 CST; 16s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 2278 (kube-apiserver)
CGroup: /system.slice/kube-apiserver.service
└─2278 /k8s/kubernetes/bin/kube-apiserver --etc...
Mar 30 14:54:46 k8s-master-1 systemd[1]: Started Kubernete...
Mar 30 14:54:46 k8s-master-1 systemd[1]: Starting Kubernet...
Mar 30 14:54:51 k8s-master-1 kube-apiserver[2278]: E0330 1...
Hint: Some lines were ellipsized, use -l to show in full.
查看启动日志:
[root@k8s-master-1 ~]# less /k8s/kubernetes/logs/kube-apiserver.INFO
Log file created at: 2022/03/30 14:54:47
Running on machine: k8s-master-1
Binary: Built with gc go1.13.9 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0330 14:54:47.056533 2278 flags.go:33] FLAG: --add-dir-header="false"
I0330 14:54:47.056860 2278 flags.go:33] FLAG: --address="127.0.0.1"
I0330 14:54:47.056865 2278 flags.go:33] FLAG: --admission-control="[]"
I0330 14:54:47.056902 2278 flags.go:33] FLAG: --admission-control-config-file=""
I0330 14:54:47.056905 2278 flags.go:33] FLAG: --advertise-address="192.168.61.161"
I0330 14:54:47.056907 2278 flags.go:33] FLAG: --allow-privileged="true"
I0330 14:54:47.057821 2278 flags.go:33] FLAG: --alsologtostderr="false"
I0330 14:54:47.057827 2278 flags.go:33] FLAG: --anonymous-auth="true"
I0330 14:54:47.057831 2278 flags.go:33] FLAG: --api-audiences="[]"
I0330 14:54:47.057836 2278 flags.go:33] FLAG: --apiserver-count="1"
I0330 14:54:47.057896 2278 flags.go:33] FLAG: --audit-dynamic-configuration="false"
I0330 14:54:47.057898 2278 flags.go:33] FLAG: --audit-log-batch-buffer-size="10000"
I0330 14:54:47.057900 2278 flags.go:33] FLAG: --audit-log-batch-max-size="1"
I0330 14:54:47.057901 2278 flags.go:33] FLAG: --audit-log-batch-max-wait="0s"
按q退出
⑥将kubelet-bootstrap用户绑定到系统集群角色,之后便于Node使用token请求证书:
[root@k8s-master-1 ~]# kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
成功会显示:clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
2.配置kube-controller-manager组件
(1)配置kube-controller-manager.conf文件:
[root@k8s-master-1 cfg]# vim kube-controller-manager.conf
[root@k8s-master-1 cfg]# cat kube-controller-manager.conf
KUBE_CONTROLLER_MANAGER_OPTS="--leader-elect=true \
--master=127.0.0.1:8080 \
--address=127.0.0.1 \
--allocate-node-cidrs=true \
--cluster-cidr=10.244.0.0/16 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-signing-cert-file=/k8s/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/k8s/kubernetes/ssl/ca-key.pem \
--root-ca-file=/k8s/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/k8s/kubernetes/ssl/ca-key.pem \
--experimental-cluster-signing-duration=87600h0m0s \
--v=2 \
--logtostderr=false \
--log-dir=/k8s/kubernetes/logs"
[root@k8s-master-1 ~]# cd /k8s/kubernetes/cfg/
[root@k8s-master-1 cfg]# ll
total 12
-rw-r--r-- 1 root root 1261 Mar 29 21:59 kube-apiserver.conf
-rw-r--r-- 1 root root 571 Mar 30 15:05 kube-controller-manager.conf
参数解释:
leader-elect=true #当该组件启动多个时,自动选举,默认true
master #连接本地apiserver,默认会监听本地8080端口
allocate-node-cidrs #是否分配和设置Pod的CDIR
service-cluster-ip-range #Service集群IP段
(2)创建kube-controller-manager.service服务:
[root@k8s-master-1 cfg]# vim /usr/lib/systemd/system/kube-controller-manager.service
[root@k8s-master-1 cfg]# cat /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
EnvironmentFile=/k8s/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/k8s/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
(3)启动kube-controller-manager组件,并设置开机自启动:
[root@k8s-master-1 cfg]# systemctl daemon-reload
[root@k8s-master-1 cfg]# systemctl start kube-controller-manager
[root@k8s-master-1 cfg]# systemctl enable kube-controller-manager
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
[root@k8s-master-1 cfg]# systemctl status kube-controller-manager
● kube-controller-manager.service - Kubernetes Controller Manager
Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2022-03-30 15:22:49 CST; 22s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 2393 (kube-controller)
CGroup: /system.slice/kube-controller-manager.service
└─2393 /k8s/kubernetes/bin/kube-controller-mana...
Mar 30 15:22:49 k8s-master-1 systemd[1]: Started Kubernete...
Mar 30 15:22:49 k8s-master-1 systemd[1]: Starting Kubernet...
Mar 30 15:22:49 k8s-master-1 kube-controller-manager[2393]: ...
Mar 30 15:22:50 k8s-master-1 kube-controller-manager[2393]: ...
Mar 30 15:23:00 k8s-master-1 kube-controller-manager[2393]: ...
Hint: Some lines were ellipsized, use -l to show in full.
查看启动日志:
[root@k8s-master-1 cfg]# less /k8s/kubernetes/logs/kube-controller-manager.INFO
Log file created at: 2022/03/30 15:22:49
Running on machine: k8s-master-1
Binary: Built with gc go1.13.9 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0330 15:22:49.585253 2393 flags.go:33] FLAG: --add-dir-header="false"
I0330 15:22:49.585473 2393 flags.go:33] FLAG: --address="127.0.0.1"
I0330 15:22:49.585477 2393 flags.go:33] FLAG: --allocate-node-cidrs="true"
I0330 15:22:49.585480 2393 flags.go:33] FLAG: --allow-untagged-cloud="false"
I0330 15:22:49.585481 2393 flags.go:33] FLAG: --alsologtostderr="false"
I0330 15:22:49.585483 2393 flags.go:33] FLAG: --attach-detach-reconcile-sync-period="1m0s"
I0330 15:22:49.585485 2393 flags.go:33] FLAG: --authentication-kubeconfig=""
I0330 15:22:49.585488 2393 flags.go:33] FLAG: --authentication-skip-lookup="false"
I0330 15:22:49.585489 2393 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="10s"
I0330 15:22:49.585490 2393 flags.go:33] FLAG: --authentication-tolerate-lookup-failure="false"
I0330 15:22:49.585492 2393 flags.go:33] FLAG: --authorization-always-allow-paths="[/healthz]"
I0330 15:22:49.585497 2393 flags.go:33] FLAG: --authorization-kubeconfig=""
I0330 15:22:49.585498 2393 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="10s"
I0330 15:22:49.585500 2393 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="10s"
/k8s/kubernetes/logs/kube-controller-manager.INFO
3.配置kube-scheduler组件
(1)配置kube-scheduler.conf文件:
[root@k8s-master-1 cfg]# vim /k8s/kubernetes/cfg/kube-scheduler.conf
[root@k8s-master-1 cfg]# cat /k8s/kubernetes/cfg/kube-scheduler.conf
KUBE_SCHEDULER_OPTS="--leader-elect=true \
--master=127.0.0.1:8080 \
--address=127.0.0.1 \
--v=2 \
--logtostderr=false \
--log-dir=/k8s/kubernetes/logs"
(2)配置kube-scheduler.service服务:
[root@k8s-master-1 cfg]# vim /usr/lib/systemd/system/kube-scheduler.service
[root@k8s-master-1 cfg]# cat /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
EnvironmentFile=/k8s/kubernetes/cfg/kube-scheduler.conf
ExecStart=/k8s/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
(3)启动kube-scheduler服务组件,并设置开机自启动:
[root@k8s-master-1 cfg]# systemctl daemon-reload
[root@k8s-master-1 cfg]# systemctl start kube-scheduler
[root@k8s-master-1 cfg]# systemctl enable kube-scheduler
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
[root@k8s-master-1 cfg]# systemctl status kube-scheduler
● kube-scheduler.service - Kubernetes Scheduler
Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2022-03-30 15:27:57 CST; 18s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 2454 (kube-scheduler)
CGroup: /system.slice/kube-scheduler.service
└─2454 /k8s/kubernetes/bin/kube-scheduler --lea...
Mar 30 15:27:57 k8s-master-1 systemd[1]: Started Kubernete...
Mar 30 15:27:57 k8s-master-1 systemd[1]: Starting Kubernet...
Mar 30 15:27:57 k8s-master-1 kube-scheduler[2454]: I0330 1...
Mar 30 15:27:57 k8s-master-1 kube-scheduler[2454]: I0330 1...
Hint: Some lines were ellipsized, use -l to show in full.
查看日志:
[root@k8s-master-1 cfg]# less /k8s/kubernetes/logs/kube-scheduler.INFO
Log file created at: 2022/03/30 15:27:57
Running on machine: k8s-master-1
Binary: Built with gc go1.13.9 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0330 15:27:57.866769 2454 flags.go:33] FLAG: --add-dir-header="false"
I0330 15:27:57.867105 2454 flags.go:33] FLAG: --address="127.0.0.1"
I0330 15:27:57.867111 2454 flags.go:33] FLAG: --algorithm-provider=""
I0330 15:27:57.867116 2454 flags.go:33] FLAG: --alsologtostderr="false"
I0330 15:27:57.867119 2454 flags.go:33] FLAG: --authentication-kubeconfig=""
I0330 15:27:57.867122 2454 flags.go:33] FLAG: --authentication-skip-lookup="false"
/k8s/kubernetes/logs/kube-scheduler.INFO
4.查看集群组件状态
[root@k8s-master-1 cfg]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
三、Node工作节点需要安装的组件介绍
kubelet:
一个在集群中每个Node节点上运行的代理,管理容器的生命周期,并会监控pod的运行状态,上报容器的健康情况。
kubelet在PodSpec方面工作。PodSpec 是描述 Pod 的 YAML 或 JSON 对象。kubelet 采用一组 PodSpec,这些 PodSpec 通过各种机制(主要通过 apiserver)提供,并确保这些 PodSpec 中描述的容器正在运行且正常运行。kubelet 不管理不是由 Kubernetes 创建的容器。
kube-proxy:
Kubernetes 网络代理在每个节点上运行。这反映了每个节点上的 Kubernetes API 中定义的服务,并且可以跨一组后端执行简单的 TCP、UDP 和 SCTP 流转发或轮循机制 TCP、UDP 和 SCTP 转发。服务群集 IP 和端口当前通过 Docker 链接兼容的环境变量找到,这些变量指定了服务代理打开的端口。有一个可选的插件为这些群集 IP 提供群集 DNS。用户必须使用 API 服务器 API 创建服务才能配置代理。
docker:
docker的engine引擎会负责本机的容器创建和管理工作。
四、配置Node工作节点各组件模块
1.安装Docker
前提确保自己的网络yum源能用。
[root@k8s-node-1 ~]# vim docker-install.sh
[root@k8s-node-1 ~]# cat docker-install.sh
#! /bin/bash
#脚本安装Docker
#作者:dorte
#时间:2021.9.18
#Email:1615360614qq.com
#优化环境,避免出现一些不必要的问题
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config &> /dev/null
setenforce 0
#停止防火墙,设置开机不启动防火墙
systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld
#清除规则
iptables -F
#停止NetworkManager,设置它开机不启动
systemctl stop NetworkManager &> /dev/null
systemctl disable NetworkManager &> /dev/null
#添加epel源
yum install -y epel-release
#添加docker-ce源
yum install -y yum-utils
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
cat /etc/yum.repos.d/docker-ce.repo
yum clean all && yum makecache
#同步时间
unalias cp #取消cp的别名,让其不进行确认提示,临时的,重新开机后还是会恢复
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
#配置pip镜像源,以便快速下载Python库
mkdir /.pip
cat > /.pip/pip.conf <<STOP
[global]
index-url=http://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host=mirrors.aliyun.com
STOP
#安装Docker
#卸载旧版本Docker,以免引起不兼容
yum remove docker docker-io docker-selinux python-docker-py -y
#安装Docker-CE社区版本
yum install docker-ce -y
systemctl start docker
systemctl enable docker
systemctl status docekr
docker --version
#指定docker镜像加速器,此处指定网易的
cat > /etc/docker/daemon.json <<STOP
{
"registry-mirrors":["https://hub-mirror.c.163.com"]
}
STOP
#重启相关服务,修改了脚本要执行重载操作
systemctl daemon-reload
#重启Docker,查看状态
systemctl restart docker
systemctl status docker
#查看版本信息
docker --version
#查看Docker的详细信息
docker info
[root@k8s-node-1 ~]# chmod +755 docker-install.sh
[root@k8s-node-1 ~]# ./docker-install.sh
2.配置Node节点证书
(1)在Master节点上用CA证书为Node节点配置证书签名请求文件kube-proxy-csr.json:
[root@k8s-master-1 ~]# cd /k8s/kubernetes/ssl/
[root@k8s-master-1 ssl]# vim kube-proxy-csr.json
[root@k8s-master-1 ssl]# cat kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "kubernetes",
"OU": "System"
}
]
}
(2)为 kube-proxy 生成证书和私钥:
[root@k8s-master-1 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
2022/03/30 15:37:34 [INFO] generate received request
2022/03/30 15:37:34 [INFO] received CSR
2022/03/30 15:37:34 [INFO] generating key: rsa-2048
2022/03/30 15:37:34 [INFO] encoded CSR
2022/03/30 15:37:34 [INFO] signed certificate with serial number 229259672626930233057876376397952991706752162876
2022/03/30 15:37:34 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@k8s-master-1 ssl]# ls kube-proxy*
kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem
(3)在Node节点上创建k8s所需目录:
[root@k8s-node-1 ~]# mkdir -p /k8s/kubernetes/{bin,cfg,logs,ssl}
[root@k8s-node-1 ~]# cd /k8s/kubernetes/
[root@k8s-node-1 kubernetes]# ll
total 0
drwxr-xr-x 2 root root 6 Mar 30 15:41 bin
drwxr-xr-x 2 root root 6 Mar 30 15:41 cfg
drwxr-xr-x 2 root root 6 Mar 30 15:41 logs
drwxr-xr-x 2 root root 6 Mar 30 15:41 ssl
(4) 在master节点上将证书拷贝到node节点:
[root@k8s-master-1 ssl]# scp -r /k8s/kubernetes/ssl/{ca.pem,kube-proxy.pem,kube-proxy-key.pem} root@k8s-node-1:/k8s/kubernetes/ssl/
root@k8s-node-1's password:
ca.pem 100% 1383 1.4KB/s 00:00
kube-proxy.pem 100% 1424 1.4KB/s 00:00
kube-proxy-key.pem 100% 1679 1.6KB/s 00:00
(5)将master节点上的kubelet、kube-proxy文件拷贝到node节点:
[root@k8s-master-1 ~]# scp -r /k8s/kubernetes/bin/{kubelet,kube-proxy} root@k8s-node-1:/k8s/kubernetes/bin/
root@k8s-node-1's password:
kubelet 100% 108MB 108.0MB/s 00:01
kube-proxy 100% 37MB 36.6MB/s 00:00
3.配置kubelet组件
(1)配置证书文件bootstrap.kubeconfig:
bootstrap.kubeconfig 将用于向 apiserver 请求证书,apiserver 会验证 token、证书是否有效,验证通过则自动颁发证书。
[root@k8s-node-1 ssl]# vim /k8s/kubernetes/cfg/bootstrap.kubeconfig
[root@k8s-node-1 ssl]# cat /k8s/kubernetes/cfg/bootstrap.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority: /k8s/kubernetes/ssl/ca.pem
server: https://192.168.61.161:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubelet-bootstrap
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
user:
token: e2b20b5979e898e33c644471e53ee8a2 #注意此处要与master上的token.csv文件的字段保持一致
(2)配置kubelet-config.yml文件:
为了安全性,kubelet需授权访问,禁止匿名访问,通过 kubelet-config.yml 授权 apiserver 访问 kubelet
[root@k8s-node-1 cfg]# vim /k8s/kubernetes/cfg/kubelet-config.yml
[root@k8s-node-1 cfg]# cat /k8s/kubernetes/cfg/kubelet-config.yml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /k8s/kubernetes/ssl/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthroizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
maxOpenFiles: 100000
maxPods: 110
(3)配置kubelet服务文件kubelet.conf:
[root@k8s-node-1 cfg]# vim /k8s/kubernetes/cfg/kubelet.conf
[root@k8s-node-1 cfg]# cat /k8s/kubernetes/cfg/kubelet.conf
KUBELET_OPTS="--hostname-override=k8s-node-1 \
--network-plugin=cni \
--cni-bin-dir=/opt/cni/bin \
--cni-conf-dir=/etc/cni/net.d \
--cgroups-per-qos=false \
--enforce-node-allocatable="" \
--kubeconfig=/k8s/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/k8s/kubernetes/cfg/bootstrap.kubeconfig \
--config=/k8s/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/k8s/kubernetes/ssl \
--pod-infra-container-image=kubernetes/pause:latest \
--v=2 \
--logtostderr=false \
--log-dir=/k8s/kubernetes/logs"
[root@k8s-node-1 cfg]# ll
total 12
-rw-r--r-- 1 root root 378 Mar 30 15:55 bootstrap.kubeconfig
-rw-r--r-- 1 root root 534 Mar 30 16:00 kubelet.conf
-rw-r--r-- 1 root root 610 Mar 30 15:56 kubelet-config.yml
参数介绍:
--hostname-override:当前节点注册到K8S中显示的名称,默认为主机 hostname
--network-plugin:启用 CNI 网络插件
--cni-bin-dir:CNI 插件可执行文件位置,默认在 /opt/cni/bin 下
--cni-conf-dir:CNI 插件配置文件位置,默认在 /etc/cni/net.d 下
--cgroups-per-qos:必须加上这个参数和--enforce-node-allocatable,否则报错 [Failed to start ContainerManager failed to initialize top level QOS containers.......]
--kubeconfig:会自动生成 kubelet.kubeconfig,用于连接 apiserver
--bootstrap-kubeconfig:指定 bootstrap.kubeconfig 文件
--config:kubelet 配置文件
--cert-dir:证书目录
--pod-infra-container-image:管理Pod网络的镜像,基础的 Pause 容器,默认是 k8s.gcr.io/pause:3.1
(4)配置kubelet.service服务:
[root@k8s-node-1 cfg]# vim /usr/lib/systemd/system/kubelet.service
[root@k8s-node-1 cfg]# cat /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Before=docker.service
[Service]
EnvironmentFile=/k8s/kubernetes/cfg/kubelet.conf
ExecStart=/k8s/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
(5)启动kubelet,并设置开机自启动:
[root@k8s-node-1 cfg]# systemctl daemon-reload
[root@k8s-node-1 cfg]# systemctl start kubelet
[root@k8s-node-1 cfg]# systemctl enable kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
查看启动日志:
[root@k8s-node-1 cfg]# tail -f /k8s/kubernetes/logs/kubelet.INFO
I0330 16:28:51.590365 2965 feature_gate.go:243] feature gates: &{map[]}
I0330 16:28:51.590397 2965 feature_gate.go:243] feature gates: &{map[]}
I0330 16:28:51.590463 2965 plugins.go:100] No cloud provider specified.
I0330 16:28:51.590473 2965 server.go:537] No cloud provider specified: "" from the config file: ""
I0330 16:28:51.590486 2965 bootstrap.go:119] Using bootstrap kubeconfig to generate TLS client cert, key and kubeconfig file
I0330 16:28:51.591183 2965 bootstrap.go:150] No valid private key and/or certificate found, reusing existing private key or creating a new one
I0330 16:28:51.636787 2965 csr.go:70] csr for this node already exists, reusing
I0330 16:28:51.642276 2965 csr.go:78] csr for this node is still valid
I0330 16:28:51.642287 2965 bootstrap.go:355] Waiting for client certificate to be issued
I0330 16:28:51.642428 2965 reflector.go:175] Starting reflector *v1beta1.CertificateSigningRequest (0s) from k8s.io/client-go/tools/watch/informerwatcher.go:146
(6)在master节点上给node节点授权:
kubelet 启动后,还没加入到集群中,会向 apiserver 请求证书,需手动在 k8s-master-1 上对 node 授权。通过如下命令查看是否有新的客户端请求颁发证书:
[root@k8s-master-1 cfg]# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-4cdu6Ig9ufwtoYPf8bY-L5C3R78YBTCr64NUmLgwHIE 8m7s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
颁发证书,允许它加入集群:
[root@k8s-master-1 cfg]# kubectl certificate approve node-csr-4cdu6Ig9ufwtoYPf8bY-L5C3R78YBTCr64NUmLgwHIE
certificatesigningrequest.certificates.k8s.io/node-csr-4cdu6Ig9ufwtoYPf8bY-L5C3R78YBTCr64NUmLgwHIE approved
(7)授权成功后,可在Master上查看Node节点:
[root@k8s-master-1 cfg]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-node-1 NotReady <none> 2m4s v1.18.6
此时node还处于未就绪状态,因为目前为止尚未安装CNI插件。
授权成功后,可在node节点的/k8s/kubenetes/ssl上查看到master为kubelet颁发的证书:
在 /k8s/kubenetes/cfg 下可以看到自动生成的 kubelet.kubeconfig 配置文件:
4.配置kube-proxy组件
(1)配置连接apiserver的kube-proxy.kubeconfig文件:
[root@k8s-node-1 cfg]# vim /k8s/kubernetes/cfg/kube-proxy.kubeconfig
[root@k8s-node-1 cfg]# cat /k8s/kubernetes/cfg/kube-proxy.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority: /k8s/kubernetes/ssl/ca.pem
server: https://192.168.61.161:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kube-proxy
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kube-proxy
user:
client-certificate: /k8s/kubernetes/ssl/kube-proxy.pem
client-key: /k8s/kubernetes/ssl/kube-proxy-key.pem
(2)配置kube-proxy-config.yml文件:
[root@k8s-node-1 cfg]# vim /k8s/kubernetes/cfg/kube-proxy-config.yml
[root@k8s-node-1 cfg]# vim /k8s/kubernetes/cfg/kube-proxy-config.yml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
address: 0.0.0.0
metrisBindAddress: 0.0.0.0:10249
clientConnection:
kubeconfig: /k8s/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-node-1
clusterCIDR: 10.0.0.0/24
mode: ipvs
ipvs:
scheduler: "rr"
iptables:
masqueradeAll: true
(3)配置kube-proxy.conf文件:
[root@k8s-node-1 cfg]# vim /k8s/kubernetes/cfg/kube-proxy.conf
[root@k8s-node-1 cfg]# cat /k8s/kubernetes/cfg/kube-proxy.conf
KUBE_PROXY_OPTS="--config=/k8s/kubernetes/cfg/kube-proxy-config.yml \
--v=2 \
--logtostderr=false \
--log-dir=/k8s/kubernetes/logs"
(4)配置kube-proxy.service
[root@k8s-node-1 cfg]# vim /usr/lib/systemd/system/kube-proxy.service
[root@k8s-node-1 cfg]# cat /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/k8s/kubernetes/cfg/kube-proxy.conf
ExecStart=/k8s/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
(5)启动kube-proxy,并设置开机自启动:
[root@k8s-node-1 cfg]# systemctl daemon-reload
[root@k8s-node-1 cfg]# systemctl start kube-proxy
[root@k8s-node-1 cfg]# systemctl enable kube-proxy
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
查看启动日志:
[root@k8s-node-1 cfg]# tail -f /k8s/kubernetes/logs/kube-proxy.INFO
I0330 16:49:01.143514 5973 config.go:315] Starting service config controller
I0330 16:49:01.143521 5973 shared_informer.go:223] Waiting for caches to sync for service config
I0330 16:49:01.143551 5973 config.go:133] Starting endpoints config controller
I0330 16:49:01.143562 5973 shared_informer.go:223] Waiting for caches to sync for endpoints config
I0330 16:49:01.144326 5973 reflector.go:175] Starting reflector *v1.Service (15m0s) from k8s.io/client-go/informers/factory.go:135
I0330 16:49:01.144690 5973 reflector.go:175] Starting reflector *v1.Endpoints (15m0s) from k8s.io/client-go/informers/factory.go:135
I0330 16:49:01.245140 5973 shared_informer.go:230] Caches are synced for endpoints config
I0330 16:49:01.245361 5973 proxier.go:997] Not syncing ipvs rules until Services and Endpoints have been received from master
I0330 16:49:01.245391 5973 shared_informer.go:230] Caches are synced for service config
I0330 16:49:01.246317 5973 service.go:379] Adding new service port "default/kubernetes:https" at 10.0.0.1:443/TCP
5.部署其它Node工作节点
其它节点的部署步骤和上诉过程大同小异
(1)想要效率一点,可以从node1节点直接把文件copy到其它node节点对应目录上,然后再做小修改:
[root@k8s-node-1 ~]# cd /k8s/kubernetes/cfg/
[root@k8s-node-1 cfg]# scp bootstrap.kubeconfig root@k8s-node-2:/k8s/kubernetes/cfg/
[root@k8s-node-1 cfg]# scp kubelet.conf root@k8s-node-2:/k8s/kubernetes/cfg/
[root@k8s-node-1 cfg]# scp /usr/lib/systemd/system/kubelet.service root@k8s-node-3:/usr/lib/systemd/system/
[root@k8s-node-1 cfg]# scp kube-proxy.kubeconfig root@k8s-node-2:/k8s/kubernetes/cfg/
[root@k8s-node-1 cfg]# scp kube-proxy-config.yml root@k8s-node-3:/k8s/kubernetes/cfg/
kubelet.conf要修改主机名:
kube-proxy-config.yml要修改hostnameOverride:
(2)回到master节点上去颁发证书,允许客户端加入k8s集群:
[root@k8s-master-1 cfg]# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-3mFoQwQdlrcdnTk_z8V3qopdIIFoDlS38wmk-SOyAAU 91s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
node-csr-4cdu6Ig9ufwtoYPf8bY-L5C3R78YBTCr64NUmLgwHIE 51m kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued
node-csr-J0kEI0Ca6ktTCTO7kmdJ_nOBOFlTnJ-0MuiVGt_3t5Y 91s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
[root@k8s-master-1 cfg]# kubectl certificate approve node-csr-3mFoQwQdlrcdnTk_z8V3qopdIIFoDlS38wmk-SOyAAU
certificatesigningrequest.certificates.k8s.io/node-csr-3mFoQwQdlrcdnTk_z8V3qopdIIFoDlS38wmk-SOyAAU approved
[root@k8s-master-1 cfg]# kubectl certificate approve node-csr-J0kEI0Ca6ktTCTO7kmdJ_nOBOFlTnJ-0MuiVGt_3t5Y
certificatesigningrequest.certificates.k8s.io/node-csr-J0kEI0Ca6ktTCTO7kmdJ_nOBOFlTnJ-0MuiVGt_3t5Y approved
(3)最后查看整个集群状态:
[root@k8s-master-1 cfg]# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-node-1 NotReady <none> 53m v1.18.6 192.168.61.162 <none> Red Hat Enterprise Linux Server 7.3 (Maipo) 3.10.0-514.el7.x86_64 docker://20.10.14
k8s-node-2 NotReady <none> 11m v1.18.6 192.168.61.163 <none> Red Hat Enterprise Linux Server 7.3 (Maipo) 3.10.0-514.el7.x86_64 docker://20.10.14
k8s-node-3 NotReady <none> 11m v1.18.6 192.168.61.164 <none> Red Hat Enterprise Linux Server 7.3 (Maipo) 3.10.0-514.el7.x86_64 docker://20.10.14
[root@k8s-master-1 cfg]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 170m
[root@k8s-master-1 cfg]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-1 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
etcd-0 Healthy {"health": "true"}
6.部署K8S集群网络
kubernetes会用到一个叫flannel的容器网络服务,来为集群内的容器分配唯一的虚拟IP,让不同节点上的容器能够通过内网IP进行通信。
(1)安装CNI插件之前先创建它的工作目录:
在配置kubelet服务组件时,指定了CNI的工作目录,所以这一步要创建对应的工作目录路径。
CNI:容器网络接口,由Google和Core OS主导制定的容器网络标准,它仅仅是一个接口,具体的功能由各个网络插件自己去实现。
[root@k8s-node-1 ~]# mkdir -p /opt/cni/bin /etc/cni/net.d
[root@k8s-node-2 ~]# mkdir -p /opt/cni/bin /etc/cni/net.d
[root@k8s-node-3 ~]# mkdir -p /opt/cni/bin /etc/cni/net.d
(2)下载CNI插件,上传到Linux,解压到指定目录:
这里附上我使用的CNI版本:cni-plugins-linux-amd64-v0.8.6.tgz的下载链接 提取码:h4em
[root@k8s-node-1 ~]# rz
[root@k8s-node-1 ~]# tar zxf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin/
[root@k8s-node-1 ~]# cd /opt/cni/bin/
[root@k8s-node-1 bin]# ls
bandwidth dhcp flannel host-local loopback portmap sbr tuning
bridge firewall host-device ipvlan macvlan ptp static vlan
(3)部署flannel服务:
kube-flannel.yml配置文件:kube-flannel.yml文件的下载链接 提取码:nynk
下载好后,直接上传到master节点上的/k8s/kubernetes/cfg/目录下:
[root@k8s-master-1 ~]# cd /k8s/kubernetes/cfg/
[root@k8s-master-1 cfg]# ls
busybox.yaml flanneld kube-controller-manager.conf kubernetes-dashboard.yaml token.csv
coredns.yaml kube-apiserver.conf kube-flannel.yml kube-scheduler.conf
查看kube-controller-manager.conf文件:
查看kube-flannel.yml文件:(局部截图)
Network 的地址需与 kube-controller-manager.conf 中的 --cluster-cidr=10.244.0.0/16 保持一致
(4)用kubectl apply -f 命令部署flannel:
[root@k8s-master-1 cfg]# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
Flannel会在Node节点上创建一个 Flannel的 Pod,可以查看pod的状态看flannel是否启动成功:
[root@k8s-master-1 cfg]# kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-flannel-ds-amd64-45z7d 1/1 Running 0 112s 192.168.61.164 k8s-node-3 <none> <none>
kube-flannel-ds-amd64-8xdrs 1/1 Running 0 112s 192.168.61.163 k8s-node-2 <none> <none>
kube-flannel-ds-amd64-9tm4g 1/1 Running 0 112s 192.168.61.162 k8s-node-1 <none> <none>
Flannel 部署成功后,可以查看Node节点是否就绪,STATUS显示已Ready:
[root@k8s-master-1 cfg]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-node-1 Ready <none> 3d23h v1.18.6 192.168.61.162 <none> Red Hat Enterprise Linux Server 7.3 (Maipo) 3.10.0-514.el7.x86_64 docker://20.10.14
k8s-node-2 Ready <none> 3d23h v1.18.6 192.168.61.163 <none> Red Hat Enterprise Linux Server 7.3 (Maipo) 3.10.0-514.el7.x86_64 docker://20.10.14
k8s-node-3 Ready <none> 3d23h v1.18.6 192.168.61.164 <none> Red Hat Enterprise Linux Server 7.3 (Maipo) 3.10.0-514.el7.x86_64 docker://20.10.14
(5)在Node节点上查看网卡信息,会看到多了一张flannel.1的虚拟网卡,此网卡的作用用于接收Pod的流量及转发:
(6)测试创建一个Pod,并查看状态:
[root@k8s-master-1 cfg]# kubectl create deployment web --image=nginx
deployment.apps/web created
[root@k8s-master-1 cfg]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
web-5dcb957ccc-c6w8m 1/1 Running 0 15m 10.244.1.2 k8s-node-2 <none> <none>
根据NODE这一栏显示的信息,去到对应的k8s-node-2节点查看容器,是可以查看到nginx容器的:
[root@k8s-node-2 bin]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d55521746d7c nginx "/docker-entrypoint.…" 3 minutes ago Up 3 minutes k8s_nginx_web-5dcb957ccc-c6w8m_default_b4ba6b2c-9f2a-4c9e-85fc-208dba3726b0_0
fa033fbfa0c9 kubernetes/pause:latest "/pause" 9 minutes ago Up 9 minutes k8s_POD_web-5dcb957ccc-c6w8m_default_b4ba6b2c-9f2a-4c9e-85fc-208dba3726b0_0
464b9ce8b20a 4e9f801d2217 "/opt/bin/flanneld -…" 19 minutes ago Up 19 minutes k8s_kube-flannel_kube-flannel-ds-amd64-8xdrs_kube-system_4f95ec57-fa76-4bab-b6ac-8de853ff4079_0
646f56c41d69 4e9f801d2217 "cp -f /etc/kube-fla…" 19 minutes ago Exited (0) 19 minutes ago k8s_install-cni_kube-flannel-ds-amd64-8xdrs_kube-system_4f95ec57-fa76-4bab-b6ac-8de853ff4079_0
8f89c29c95c1 kubernetes/pause:latest "/pause" 19 minutes ago Up 19 minutes k8s_POD_kube-flannel-ds-amd64-8xdrs_kube-system_4f95ec57-fa76-4bab-b6ac-8de853ff4079_0
容器Running成功后,再在k8s-node-2节点上查看网卡信息,多了一块cni0的虚拟网卡,cni0用于Pod本地通信使用:
(7)暴露端口,并进行服务访问:
7.部署集群内部DNS服务
(1)部署CoreDNS:
下载CoreDNS配置文件:coredns.yaml文件的下载连接 提取码:6sr5
下载好后,直接上传到mater节点上的/k8s/kubernetes/cfg/目录下
[root@k8s-master-1 cfg]# rz
[root@k8s-master-1 cfg]# ll
total 44
-rw-r--r-- 1 root root 5280 Apr 3 20:15 coredns.yaml
-rw-r--r-- 1 root root 233 Apr 2 21:14 flanneld
-rw-r--r-- 1 root root 1261 Mar 29 21:59 kube-apiserver.conf
-rw-r--r-- 1 root root 571 Mar 30 15:05 kube-controller-manager.conf
-rw-r--r-- 1 root root 14366 Mar 30 18:35 kube-flannel.yml
-rw-r--r-- 1 root root 163 Mar 30 15:25 kube-scheduler.conf
-rw-r--r-- 1 root root 84 Mar 29 21:50 token.csv
(2)修改coredns.yml文件,要确保clusterIP一定要与kube-config.yaml中的clusterDNS保持一致:
还有几处要修改的地方:
(4)修改完毕后,开始部署:
[root@k8s-master-1 cfg]# kubectl apply -f coredns.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
查看Pod:
[root@k8s-master-1 cfg]# kubectl get pod -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-8686db44f5-5khf9 1/1 Running 0 19m 10.244.1.5 k8s-node-2 <none> <none>
kube-flannel-ds-amd64-45z7d 1/1 Running 1 4h56m 192.168.61.164 k8s-node-3 <none> <none>
kube-flannel-ds-amd64-8xdrs 1/1 Running 2 4h56m 192.168.61.163 k8s-node-2 <none> <none>
kube-flannel-ds-amd64-9tm4g 1/1 Running 1 4h56m 192.168.61.162 k8s-node-1 <none> <none>
(5)创建busybox服务,验证CoreDNS:
[root@k8s-master-1 cfg]# vim busybox.yaml
[root@k8s-master-1 cfg]# cat busybox.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
dnsPolicy: ClusterFirst
containers:
- name: busybox
image: busybox:1.28.4
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always
创建busybox Pod:
[root@k8s-master-1 cfg]# kubectl apply -f busybox.yaml
pod/busybox created
[root@k8s-master-1 cfg]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 0 47s 10.244.0.5 k8s-node-1 <none> <none>
web-5dcb957ccc-t5xpr 1/1 Running 0 15m 10.244.2.5 k8s-node-3 <none> <none>
查看验证:
[root@k8s-node-1 system]# docker ps -a | grep busybox
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
56a8d1c13ca7 8c811b4aec35 "sleep 3600" 15 minutes ago Up 15 minutes k8s_busybox_busybox_default_b3ab6359-3e76-4933-afc7-fb108dd2e3c1_0
[root@k8s-node-1 system]# docker exec -it 56a8d1c13ca7 sh
/ #
/ # ls
bin dev etc home proc root sys tmp usr var
/ # nslookup web
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local
Name: web
Address 1: 10.0.0.42 web.default.svc.cluster.local
/ #
/ # exit
以上就是配置Master管理节点和Node工作节点组件模块的内容,在下一篇文章中我们将进行Kubernetes Dashboard的部署。