二进制安装kubernetes(v1.20.16)

序号	IP	角色	Hostname	安装组件
1	192.168.10.11	Master,Node	vm01	Apiserver,ControllerManager,Scheduler,Kubelet,Proxy,Etcd
2	192.168.10.12	Node	vm02	Kubelet,Proxy,Etcd
3	192.168.10.13	Node	vm03	Kubelet,Proxy,Etcd

2.软件版本

序号	软件名称	版本
1	Centos	7.9.2009，内核升级到5.17.1-1.el7.elrepo.x86_64
2	Docker-ce	v20.10.9
3	Etcd	v3.4.9
4	Kubernetes	v1.20.14
5	cfssl	v1.6.1

3.下载地址

序号	软件	下载地址
1	cfssl	https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64
2	cfssljson	https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64
3	cfssl-certinfo	https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64
4	etcd	https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz
5	kubernetes	https://dl.k8s.io/v1.20.14/kubernetes-server-linux-amd64.tar.gz
6	docker-ce	https://download.docker.com/linux/static/stable/x86_64/docker-20.10.9.tgz

4.初始化虚拟机

4.1安装虚拟机

在所有虚拟机上进行以下操作

4.2升级内核

#在所有虚拟机上进行操作
#更新yum源仓库
yum update -y
#导入ELRepo仓库的公共密钥
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
#安装ELRepo仓库的yum源
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
#查看可用的系统内核包
[root@vm01 ~]# yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * elrepo-kernel: ftp.yz.yamagata-u.ac.jp
Available Packages
elrepo-release.noarch                         7.0-5.el7.elrepo                elrepo-kernel
kernel-lt.x86_64                              5.4.188-1.el7.elrepo            elrepo-kernel
kernel-lt-devel.x86_64                        5.4.188-1.el7.elrepo            elrepo-kernel
kernel-lt-doc.noarch                          5.4.188-1.el7.elrepo            elrepo-kernel
kernel-lt-headers.x86_64                      5.4.188-1.el7.elrepo            elrepo-kernel
kernel-lt-tools.x86_64                        5.4.188-1.el7.elrepo            elrepo-kernel
kernel-lt-tools-libs.x86_64                   5.4.188-1.el7.elrepo            elrepo-kernel
kernel-lt-tools-libs-devel.x86_64             5.4.188-1.el7.elrepo            elrepo-kernel
kernel-ml-devel.x86_64                        5.17.1-1.el7.elrepo             elrepo-kernel
kernel-ml-doc.noarch                          5.17.1-1.el7.elrepo             elrepo-kernel
kernel-ml-headers.x86_64                      5.17.1-1.el7.elrepo             elrepo-kernel
kernel-ml-tools.x86_64                        5.17.1-1.el7.elrepo             elrepo-kernel
kernel-ml-tools-libs.x86_64                   5.17.1-1.el7.elrepo             elrepo-kernel
kernel-ml-tools-libs-devel.x86_64             5.17.1-1.el7.elrepo             elrepo-kernel
perf.x86_64                                   5.17.1-1.el7.elrepo             elrepo-kernel
python-perf.x86_64                            5.17.1-1.el7.elrepo             elrepo-kernel
#安装最新版本内核
yum --enablerepo=elrepo-kernel install -y kernel-ml
#查看系统上的所有可用内核
sudo awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
#设置默认版本,其中 0 是上面查询出来的可用内核
grub2-set-default 0
#生成 grub 配置文件
grub2-mkconfig -o /boot/grub2/grub.cfg
#重启
reboot

#删除旧内核（可选）
#查看系统中全部的内核
rpm -qa | grep kernel
#删除旧内核的 RPM 包，具体内容视上述命令的返回结果而定
yum remove kernel-3.10.0-514.el7.x86_64 \
kernel-tools-libs-3.10.0-862.11.6.el7.x86_64 \
kernel-tools-3.10.0-862.11.6.el7.x86_64 \
kernel-3.10.0-862.11.6.el7.x86_64

4.3安装模块

#在所有虚拟机上操作
[root@vm01 ~]# modprobe -- ip_vs
[root@vm01 ~]# modprobe -- ip_vs_rr
[root@vm01 ~]# modprobe -- ip_vs_wrr
[root@vm01 ~]# modprobe -- ip_vs_sh
[root@vm01 ~]# modprobe -- nf_conntrack_ipv4
modprobe: FATAL: Module nf_conntrack_ipv4 not found.
[root@vm01 ~]# lsmod | grep ip_vs
ip_vs_sh               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  0
ip_vs                 159744  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          159744  1 ip_vs
nf_defrag_ipv6         24576  2 nf_conntrack,ip_vs
libcrc32c              16384  3 nf_conntrack,xfs,ip_vs
[root@vm01 ~]# lsmod | grep nf_conntrack_ipv4

4.4系统设置

#关闭并停止防火墙，
systemctl stop firewalld && systemctl disable firewalld
#禁用SELinux，让容器可以顺利地读取主机文件系统
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0
#关闭swap
swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab

4.5设置hoss

cat >> /etc/hosts << EOF
192.168.10.11 vm01
192.168.10.12 vm02
192.168.10.13 vm03
EOF

4.6设置IPv4转发

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

4.7时间同步

yum -y install chrony
vi /etc/chrony.conf
server ntp.aliyun.com iburst
server ntp1.aliyun.com iburst
server ntp2.aliyun.com iburst
server ntp3.aliyun.com iburst

systemctl restart chronyd

[root@vm01 ~]# chronyc -a makestep
200 OK
[root@vm01 ~]# chronyc sourcestats
210 Number of sources = 2
Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
==============================================================================
203.107.6.88               26  13   35m     -0.792      1.898    -24us  1486us
120.25.115.20              23  16   36m     +0.055      0.709   -251us   545us
[root@vm01 ~]# chronyc sources -v
210 Number of sources = 2

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* 203.107.6.88                  2   8   377   150  +1569us[+1671us] +/-   22ms
^+ 120.25.115.20                 2   8   377    86   -772us[ -772us] +/-   25ms

4.8安装依赖软件包

yum install -y ipvsadm ipset sysstat conntrack libseccomp wget git

5.SSH免密登录

#此步操作在Master主机上进行
[root@vm01 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:nmsw1JMs6U2M+TE0fh+ZmQrI2doLV5kbpO7X0gutmE4 root@vm01
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|        o .      |
|     . & = o =   |
|      X # * *    |
|     o OSB = .   |
|      *.*.o..    |
|       *E..o.    |
|       .++ooo    |
|       o=.....   |
+----[SHA256]-----+

#配置公钥到其他节点，输入对方密码即可完成从master到vm02的免密访问
[root@vm01 ~]# ssh-copy-id vm02
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'vm02 (192.168.10.12)' can't be established.
ECDSA key fingerprint is SHA256:pFfmADyl1dFq2Uadp/YwSEe+yW29sxkfzoQD/y6jvts.
ECDSA key fingerprint is MD5:27:53:f0:aa:b8:6c:2c:2e:b7:e5:ef:c7:fb:32:10:6f.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@vm02's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'vm02'"
and check to make sure that only the key(s) you wanted were added.

[root@vm01 ~]# ssh-copy-id vm03
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'vm03 (192.168.10.13)' can't be established.
ECDSA key fingerprint is SHA256:pFfmADyl1dFq2Uadp/YwSEe+yW29sxkfzoQD/y6jvts.
ECDSA key fingerprint is MD5:27:53:f0:aa:b8:6c:2c:2e:b7:e5:ef:c7:fb:32:10:6f.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@vm03's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'vm03'"
and check to make sure that only the key(s) you wanted were added.

6.创建相关目录

#在所有虚拟机上进行操作
mkdir -p /opt/TLS/{download,etcd,k8s}
mkdir -p /opt/TLS/etcd/{cfg,bin,ssl}
mkdir -p /opt/TLS/k8s/{cfg,bin,ssl}

7.下载软件

#以下操作只在master上进行
#进入到下载目录
cd /opt/TLS/download
#下载并解压cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64
chmod +x cfssl*
[root@vm03 download]# ll
total 40232
-rwxr-xr-x 1 root root 16659824 Dec  7 15:36 cfssl_1.6.1_linux_amd64
-rwxr-xr-x 1 root root 13502544 Dec  7 15:35 cfssl-certinfo_1.6.1_linux_amd64
-rwxr-xr-x 1 root root 11029744 Dec  7 15:35 cfssljson_1.6.1_linux_amd64

#下载并解压etcd
wget https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz
tar -xvf etcd-v3.4.9-linux-amd64.tar.gz
chmod +x etcd-v3.4.9-linux-amd64/etcd*
[root@vm03 download]# ll etcd-v3.4.9-linux-amd64/
total 40540
drwxr-xr-x 14 630384594 600260513     4096 May 22  2020 Documentation
-rwxr-xr-x  1 630384594 600260513 23827424 May 22  2020 etcd
-rwxr-xr-x  1 630384594 600260513 17612384 May 22  2020 etcdctl
-rw-r--r--  1 630384594 600260513    43094 May 22  2020 README-etcdctl.md
-rw-r--r--  1 630384594 600260513     8431 May 22  2020 README.md
-rw-r--r--  1 630384594 600260513     7855 May 22  2020 READMEv2-etcdctl.md

#下载并解压kubernetes
wget https://dl.k8s.io/v1.20.14/kubernetes-server-linux-amd64.tar.gz
tar zxvf kubernetes-server-linux-amd64.tar.gz
chmod +x kubernetes/server/bin/{kubectl,kubelet,kube-apiserver,kube-controller-manager,kube-scheduler,kube-proxy}
[root@vm03 download]# ll kubernetes/server/bin/
total 1134068
-rwxr-xr-x 1 root root  57724928 Feb 16 20:49 apiextensions-apiserver
-rwxr-xr-x 1 root root  45211648 Feb 16 20:49 kubeadm
-rwxr-xr-x 1 root root  51773440 Feb 16 20:49 kube-aggregator
-rwxr-xr-x 1 root root 131301376 Feb 16 20:49 kube-apiserver
-rw-r--r-- 1 root root         8 Feb 16 20:48 kube-apiserver.docker_tag
-rw------- 1 root root 136526848 Feb 16 20:48 kube-apiserver.tar
-rwxr-xr-x 1 root root 121110528 Feb 16 20:49 kube-controller-manager
-rw-r--r-- 1 root root         8 Feb 16 20:48 kube-controller-manager.docker_tag
-rw------- 1 root root 126336000 Feb 16 20:48 kube-controller-manager.tar
-rwxr-xr-x 1 root root  46592000 Feb 16 20:49 kubectl
-rwxr-xr-x 1 root root  54333584 Feb 16 20:49 kubectl-convert
-rwxr-xr-x 1 root root 124521440 Feb 16 20:49 kubelet
-rwxr-xr-x 1 root root   1507328 Feb 16 20:49 kube-log-runner
-rwxr-xr-x 1 root root  44163072 Feb 16 20:49 kube-proxy
-rw-r--r-- 1 root root         8 Feb 16 20:48 kube-proxy.docker_tag
-rw------- 1 root root 114255872 Feb 16 20:48 kube-proxy.tar
-rwxr-xr-x 1 root root  49618944 Feb 16 20:49 kube-scheduler
-rw-r--r-- 1 root root         8 Feb 16 20:48 kube-scheduler.docker_tag
-rw------- 1 root root  54844416 Feb 16 20:48 kube-scheduler.tar
-rwxr-xr-x 1 root root   1437696 Feb 16 20:49 mounter

#下载并解压docker-ce
wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.9.tgz
tar -xvf docker-20.10.9.tgz
chmod +x docker/*
[root@vm03 download]# ll docker
total 200840
-rwxr-xr-x 1 1000 1000 33908392 Oct  5 00:08 containerd
-rwxr-xr-x 1 1000 1000  6508544 Oct  5 00:08 containerd-shim
-rwxr-xr-x 1 1000 1000  8609792 Oct  5 00:08 containerd-shim-runc-v2
-rwxr-xr-x 1 1000 1000 21131264 Oct  5 00:08 ctr
-rwxr-xr-x 1 1000 1000 52883616 Oct  5 00:08 docker
-rwxr-xr-x 1 1000 1000 64758736 Oct  5 00:08 dockerd
-rwxr-xr-x 1 1000 1000   708616 Oct  5 00:08 docker-init
-rwxr-xr-x 1 1000 1000  2784145 Oct  5 00:08 docker-proxy
-rwxr-xr-x 1 1000 1000 14352296 Oct  5 00:08 runc

8.准备cfss工具

cfssl是一个开源的证书管理工具，使用json文件生成证书，相比openssl更方便使用。

#只在master上操作
cd /opt/TLS/download
cp cfssl_1.6.1_linux_amd64 /usr/local/bin/cfssl
cp cfssljson_1.6.1_linux_amd64 /usr/local/bin/cfssljson
cp cfssl-certinfo_1.6.1_linux_amd64 /usr/local/bin/cfssl-certinfo
[root@vm03 download]# ll /usr/local/bin/cfssl*
-rwxr-xr-x 1 root root 16659824 Apr  4 08:46 /usr/local/bin/cfssl
-rwxr-xr-x 1 root root 13502544 Apr  4 08:46 /usr/local/bin/cfssl-certinfo
-rwxr-xr-x 1 root root 11029744 Apr  4 08:46 /usr/local/bin/cfssljson

9.生成etcd证书

9.1自签CA申请文件

cd /opt/TLS/etcd/ssl
cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json << EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

9.2生成自签CA证书

[root@vm03 ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2022/04/04 08:51:25 [INFO] generating a new CA key and certificate from CSR
2022/04/04 08:51:25 [INFO] generate received request
2022/04/04 08:51:25 [INFO] received CSR
2022/04/04 08:51:25 [INFO] generating key: rsa-2048
2022/04/04 08:51:26 [INFO] encoded CSR
2022/04/04 08:51:26 [INFO] signed certificate with serial number 464748957865402020542542705181876295838207954582
[root@vm03 ssl]# ll
total 20
-rw-r--r-- 1 root root  287 Apr  4 08:51 ca-config.json
-rw-r--r-- 1 root root  956 Apr  4 08:51 ca.csr
-rw-r--r-- 1 root root  209 Apr  4 08:51 ca-csr.json
-rw------- 1 root root 1679 Apr  4 08:51 ca-key.pem
-rw-r--r-- 1 root root 1216 Apr  4 08:51 ca.pem
#上述操作，会生成ca.pem和ca-key.pem两个文件

9.3创建etcd证书申请文件

cat > server-csr.json << EOF
{
    "CN": "etcd",
    "hosts": [
    "192.168.10.11",
    "192.168.10.12",
    "192.168.10.13"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF
#上述文件hosts字段中IP为所有etcd节点的集群内部通信IP，一个都不能少！为了方便后期扩容可以多写几个预留的IP。

9.4签发Etcd HTTPS证书

[root@vm03 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2022/04/04 08:55:55 [INFO] generate received request
2022/04/04 08:55:55 [INFO] received CSR
2022/04/04 08:55:55 [INFO] generating key: rsa-2048
2022/04/04 08:55:55 [INFO] encoded CSR
2022/04/04 08:55:55 [INFO] signed certificate with serial number 177379691802225269854687255587397345225756828558
[root@vm03 ssl]# ll
total 36
-rw-r--r-- 1 root root  287 Apr  4 08:51 ca-config.json
-rw-r--r-- 1 root root  956 Apr  4 08:51 ca.csr
-rw-r--r-- 1 root root  209 Apr  4 08:51 ca-csr.json
-rw------- 1 root root 1679 Apr  4 08:51 ca-key.pem
-rw-r--r-- 1 root root 1216 Apr  4 08:51 ca.pem
-rw-r--r-- 1 root root 1013 Apr  4 08:55 server.csr
-rw-r--r-- 1 root root  290 Apr  4 08:55 server-csr.json
-rw------- 1 root root 1675 Apr  4 08:55 server-key.pem
-rw-r--r-- 1 root root 1338 Apr  4 08:55 server.pem
#上述操作会生成server.pem和server-key.pem两个文件

10.部署Etcd集群

10.1生成配置文件

#这里为了方便操作，同时生成了3个etcd虚拟机上的配置文件，然后将各自的配置文件分发至不同的虚拟机，减少了修改的操作。
cd /opt/TLS/etcd/cfg

#-------------------------------------
#生成vm01虚拟机上对应的配置文件
#-------------------------------------
cat > etcd01.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.10.11:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.10.11:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.11:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.11:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.10.11:2380,etcd-2=https://192.168.10.12:2380,etcd-3=https://192.168.10.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

#-------------------------------------
#生成vm02虚拟机上对应的配置文件
#-------------------------------------
cat > etcd02.conf << EOF
#[Member]
ETCD_NAME="etcd-2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.10.12:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.10.12:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.12:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.12:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.10.11:2380,etcd-2=https://192.168.10.12:2380,etcd-3=https://192.168.10.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

#-------------------------------------
#生成vm03虚拟机上对应的配置文件
#-------------------------------------
cat > etcd03.conf << EOF
#[Member]
ETCD_NAME="etcd-3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.10.13:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.10.13:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.13:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.13:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.10.11:2380,etcd-2=https://192.168.10.12:2380,etcd-3=https://192.168.10.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

#查看已生成的配置文件清单列表
[root@vm03 cfg]# ll
total 16
-rw-r--r-- 1 root root 509 Apr  4 09:05 etcd01.conf
-rw-r--r-- 1 root root 509 Apr  4 09:05 etcd02.conf
-rw-r--r-- 1 root root 509 Apr  4 09:05 etcd03.conf

#---------------------------备注说明-------------------------------
# • ETCD_NAME：节点名称，集群中唯一
# • ETCD_DATA_DIR：数据目录
# • ETCD_LISTEN_PEER_URLS：集群通信监听地址
# • ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
# • ETCD_INITIAL_ADVERTISE_PEERURLS：集群通告地址
# • ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
# • ETCD_INITIAL_CLUSTER：集群节点地址
# • ETCD_INITIALCLUSTER_TOKEN：集群Token
# • ETCD_INITIALCLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群
#-----------------------------------------------------------------

10.2生成etcd管理文件

cd /opt/TLS/etcd/cfg

cat > etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

#查看已生成的文件列表清单
[root@vm03 cfg]# ll
total 16
-rw-r--r-- 1 root root 509 Apr  4 09:05 etcd01.conf
-rw-r--r-- 1 root root 509 Apr  4 09:05 etcd02.conf
-rw-r--r-- 1 root root 509 Apr  4 09:05 etcd03.conf
-rw-r--r-- 1 root root 535 Apr  4 09:05 etcd.service

10.3分发文件

#创建etcd运行时所需的目录
mkdir -p /var/lib/etcd/default.etcd
ssh vm02 "mkdir -p /var/lib/etcd/default.etcd"
ssh vm03 "mkdir -p /var/lib/etcd/default.etcd"

#创建ecd配置文件目录
mkdir -p /opt/etcd/{bin,cfg,ssl}
ssh vm02 "mkdir -p /opt/etcd/{bin,cfg,ssl}"
ssh vm03 "mkdir -p /opt/etcd/{bin,cfg,ssl}"

#分发etcd可执行文件
scp -r /opt/TLS/download/etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
scp -r /opt/TLS/download/etcd-v3.4.9-linux-amd64/{etcd,etcdctl} vm02:/opt/etcd/bin/
scp -r /opt/TLS/download/etcd-v3.4.9-linux-amd64/{etcd,etcdctl} vm03:/opt/etcd/bin/

#分发etcd配置文件
scp -r /opt/TLS/etcd/cfg/etcd01.conf /opt/etcd/cfg/etcd.conf
scp -r /opt/TLS/etcd/cfg/etcd02.conf vm02:/opt/etcd/cfg/etcd.conf
scp -r /opt/TLS/etcd/cfg/etcd03.conf vm03:/opt/etcd/cfg/etcd.conf

#分发etcd管理文件
scp -r /opt/TLS/etcd/cfg/etcd.service /usr/lib/systemd/system/etcd.service
scp -r /opt/TLS/etcd/cfg/etcd.service vm02:/usr/lib/systemd/system/etcd.service
scp -r /opt/TLS/etcd/cfg/etcd.service vm03:/usr/lib/systemd/system/etcd.service

#分发etcd证书文件
scp -r /opt/TLS/etcd/ssl/*pem /opt/etcd/ssl
scp -r /opt/TLS/etcd/ssl/*pem vm02:/opt/etcd/ssl
scp -r /opt/TLS/etcd/ssl/*pem vm03:/opt/etcd/ssl

10.4核对文件

#核对etcd可执行文件
[root@vm01 cfg]# ls -l /opt/etcd/bin/
total 40472
-rwxr-xr-x 1 root root 23827424 Apr  3 12:38 etcd
-rwxr-xr-x 1 root root 17612384 Apr  3 12:38 etcdctl
[root@vm01 cfg]# ssh vm02 "ls -l /opt/etcd/bin/"
total 40472
-rwxr-xr-x 1 root root 23827424 Apr  3 12:38 etcd
-rwxr-xr-x 1 root root 17612384 Apr  3 12:38 etcdctl
[root@vm01 cfg]# ssh vm03 "ls -l /opt/etcd/bin/"
total 40472
-rwxr-xr-x 1 root root 23827424 Apr  3 12:38 etcd
-rwxr-xr-x 1 root root 17612384 Apr  3 12:38 etcdctl

#核对etcd配置文件
[root@vm01 cfg]# ls -l /opt/etcd/cfg/
total 4
-rw-r--r-- 1 root root 509 Apr  3 12:38 etcd.conf
[root@vm01 cfg]# ssh vm02 "ls -l /opt/etcd/cfg/"
total 4
-rw-r--r-- 1 root root 509 Apr  3 12:38 etcd.conf
[root@vm01 cfg]# ssh vm03 "ls -l /opt/etcd/cfg/"
total 4
-rw-r--r-- 1 root root 509 Apr  4 09:16 etcd.conf

#核对etcd管理文件
[root@vm01 cfg]# ls -l /usr/lib/systemd/system/etcd*
-rw-r--r-- 1 root root 535 Apr  3 12:39 /usr/lib/systemd/system/etcd.service
[root@vm01 cfg]# ssh vm02 "ls -l /usr/lib/systemd/system/etcd*"
-rw-r--r-- 1 root root 535 Apr  3 12:39 /usr/lib/systemd/system/etcd.service
[root@vm01 cfg]# ssh vm03 "ls -l /usr/lib/systemd/system/etcd*"
-rw-r--r-- 1 root root 535 Apr  4 09:17 /usr/lib/systemd/system/etcd.service

#核对etcd证书文件
[root@vm01 cfg]# ls -l /opt/etcd/ssl
total 16
-rw------- 1 root root 1679 Apr  3 12:39 ca-key.pem
-rw-r--r-- 1 root root 1216 Apr  3 12:39 ca.pem
-rw------- 1 root root 1675 Apr  3 12:39 server-key.pem
-rw-r--r-- 1 root root 1338 Apr  3 12:39 server.pem
[root@vm01 cfg]# ssh vm02 "ls -l /opt/etcd/ssl"
total 16
-rw------- 1 root root 1679 Apr  3 12:39 ca-key.pem
-rw-r--r-- 1 root root 1216 Apr  3 12:39 ca.pem
-rw------- 1 root root 1675 Apr  3 12:39 server-key.pem
-rw-r--r-- 1 root root 1338 Apr  3 12:39 server.pem
[root@vm01 cfg]# ssh vm03 "ls -l /opt/etcd/ssl"
total 16
-rw------- 1 root root 1679 Apr  4 09:17 ca-key.pem
-rw-r--r-- 1 root root 1216 Apr  4 09:17 ca.pem
-rw------- 1 root root 1675 Apr  4 09:17 server-key.pem
-rw-r--r-- 1 root root 1338 Apr  4 09:17 server.pem

10.5启动etcd集群

#按顺序分别在vm01、vm02和vm03这3台虚拟机上执行以下命令，其中在vm01上执行命令时会有等待现象，主要是等待其他机器的状态

#在vm01上执行启动命令，并设置开机启动，同时查看etcd状态
[root@vm01 cfg]# systemctl daemon-reload && systemctl start etcd && systemctl enable etcd && systemctl status etcd
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-04-03 12:52:39 CST; 83ms ago
 Main PID: 1281 (etcd)
   CGroup: /system.slice/etcd.service
           └─1281 /opt/etcd/bin/etcd --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem -...

Apr 03 12:52:39 vm01 etcd[1281]: {"level":"info","ts":"2022-04-03T12:52:39.282+0800","caller":"raft/node.go:325","msg":"raft.node: 6571fb7574e87dba elected leader 6571fb7574e87dba at term 4"}
Apr 03 12:52:39 vm01 etcd[1281]: {"level":"info","ts":"2022-04-03T12:52:39.290+0800","caller":"etcdserver/server.go:2036","msg":"published local member to cluster through raft","local-member-id":"6571fb...
Apr 03 12:52:39 vm01 systemd[1]: Started Etcd Server.
Apr 03 12:52:39 vm01 etcd[1281]: {"level":"info","ts":"2022-04-03T12:52:39.299+0800","caller":"embed/serve.go:191","msg":"serving client traffic securely","address":"192.168.10.11:2379"}
Apr 03 12:52:39 vm01 etcd[1281]: {"level":"warn","ts":"2022-04-03T12:52:39.338+0800","caller":"etcdserver/cluster_util.go:315","msg":"failed to reach the peer URL","address":"https://192.168.10.13:2380/...
Apr 03 12:52:39 vm01 etcd[1281]: {"level":"warn","ts":"2022-04-03T12:52:39.338+0800","caller":"etcdserver/cluster_util.go:168","msg":"failed to get version","remote-member-id":"d1fbb74bc6...ction refused"}
Apr 03 12:52:39 vm01 etcd[1281]: {"level":"info","ts":"2022-04-03T12:52:39.338+0800","caller":"etcdserver/server.go:2527","msg":"setting up initial cluster version","cluster-version":"3.0"}
Apr 03 12:52:39 vm01 etcd[1281]: {"level":"info","ts":"2022-04-03T12:52:39.341+0800","caller":"membership/cluster.go:558","msg":"set initial cluster version","cluster-id":"a967fee455377b3...version":"3.0"}
Apr 03 12:52:39 vm01 etcd[1281]: {"level":"info","ts":"2022-04-03T12:52:39.341+0800","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.0"}
Apr 03 12:52:39 vm01 etcd[1281]: {"level":"info","ts":"2022-04-03T12:52:39.341+0800","caller":"etcdserver/server.go:2559","msg":"cluster version is updated","cluster-version":"3.0"}
Hint: Some lines were ellipsized, use -l to show in full.

#在vm02上执行启动命令，并设置开机启动，同时查看etcd状态
[root@vm02 ~]# systemctl daemon-reload && systemctl start etcd && systemctl enable etcd && systemctl status etcd
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-04-03 12:52:41 CST; 76ms ago
 Main PID: 1188 (etcd)
   CGroup: /system.slice/etcd.service
           └─1188 /opt/etcd/bin/etcd --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem -...

Apr 03 12:52:41 vm02 etcd[1188]: {"level":"info","ts":"2022-04-03T12:52:41.311+0800","caller":"raft/raft.go:811","msg":"9b449b0ff1d4c375 [logterm: 1, index: 3] sent MsgVote request to d1f...e5c at term 2"}
Apr 03 12:52:41 vm02 etcd[1188]: {"level":"info","ts":"2022-04-03T12:52:41.582+0800","caller":"raft/raft.go:859","msg":"9b449b0ff1d4c375 [term: 2] received a MsgVote message with higher t...dba [term: 4]"}
Apr 03 12:52:41 vm02 etcd[1188]: {"level":"info","ts":"2022-04-03T12:52:41.582+0800","caller":"raft/raft.go:700","msg":"9b449b0ff1d4c375 became follower at term 4"}
Apr 03 12:52:41 vm02 etcd[1188]: {"level":"info","ts":"2022-04-03T12:52:41.583+0800","caller":"raft/raft.go:960","msg":"9b449b0ff1d4c375 [logterm: 1, index: 3, vote: 0] cast MsgVote for 6... 3] at term 4"}
Apr 03 12:52:41 vm02 etcd[1188]: {"level":"info","ts":"2022-04-03T12:52:41.588+0800","caller":"raft/node.go:325","msg":"raft.node: 9b449b0ff1d4c375 elected leader 6571fb7574e87dba at term 4"}
Apr 03 12:52:41 vm02 etcd[1188]: {"level":"info","ts":"2022-04-03T12:52:41.601+0800","caller":"etcdserver/server.go:2036","msg":"published local member to cluster through raft","local-member-id":"9b449b...
Apr 03 12:52:41 vm02 systemd[1]: Started Etcd Server.
Apr 03 12:52:41 vm02 etcd[1188]: {"level":"info","ts":"2022-04-03T12:52:41.610+0800","caller":"embed/serve.go:191","msg":"serving client traffic securely","address":"192.168.10.12:2379"}
Apr 03 12:52:41 vm02 etcd[1188]: {"level":"info","ts":"2022-04-03T12:52:41.644+0800","caller":"membership/cluster.go:558","msg":"set initial cluster version","cluster-id":"a967fee455377b3...version":"3.0"}
Apr 03 12:52:41 vm02 etcd[1188]: {"level":"info","ts":"2022-04-03T12:52:41.645+0800","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.0"}
Hint: Some lines were ellipsized, use -l to show in full.

#在vm03上执行启动命令，并设置开机启动，同时查看etcd状态
[root@vm03 ~]# systemctl daemon-reload && systemctl start etcd && systemctl enable etcd && systemctl status etcd
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-04-04 09:29:12 CST; 90ms ago
 Main PID: 1160 (etcd)
   CGroup: /system.slice/etcd.service
           └─1160 /opt/etcd/bin/etcd --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem -...

Apr 04 09:29:12 vm03 etcd[1160]: {"level":"info","ts":"2022-04-04T09:29:12.907+0800","caller":"membership/cluster.go:558","msg":"set initial cluster version","cluster-id":"a967fee455377b3...version":"3.0"}
Apr 04 09:29:12 vm03 etcd[1160]: {"level":"info","ts":"2022-04-04T09:29:12.907+0800","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.0"}
Apr 04 09:29:12 vm03 etcd[1160]: {"level":"info","ts":"2022-04-04T09:29:12.907+0800","caller":"etcdserver/server.go:2036","msg":"published local member to cluster through raft","local-member-id":"d1fbb7...
Apr 04 09:29:12 vm03 systemd[1]: Started Etcd Server.
Apr 04 09:29:12 vm03 etcd[1160]: {"level":"info","ts":"2022-04-04T09:29:12.915+0800","caller":"embed/serve.go:191","msg":"serving client traffic securely","address":"192.168.10.13:2379"}
Apr 04 09:29:12 vm03 etcd[1160]: {"level":"info","ts":"2022-04-04T09:29:12.916+0800","caller":"etcdserver/server.go:715","msg":"initialized peer connections; fast-forwarding election ticks","local-membe...
Apr 04 09:29:12 vm03 etcd[1160]: {"level":"info","ts":"2022-04-04T09:29:12.932+0800","caller":"rafthttp/stream.go:250","msg":"set message encoder","from":"d1fbb74bc6a61e5c","to":"d1fbb74b...tream Message"}
Apr 04 09:29:12 vm03 etcd[1160]: {"level":"warn","ts":"2022-04-04T09:29:12.933+0800","caller":"rafthttp/stream.go:277","msg":"established TCP streaming connection with remote peer","strea...1fb7574e87dba"}
Apr 04 09:29:12 vm03 etcd[1160]: {"level":"info","ts":"2022-04-04T09:29:12.967+0800","caller":"rafthttp/stream.go:250","msg":"set message encoder","from":"d1fbb74bc6a61e5c","to":"d1fbb74b...eam MsgApp v2"}
Apr 04 09:29:12 vm03 etcd[1160]: {"level":"warn","ts":"2022-04-04T09:29:12.967+0800","caller":"rafthttp/stream.go:277","msg":"established TCP streaming connection with remote peer","strea...1fb7574e87dba"}
Hint: Some lines were ellipsized, use -l to show in full.

10.6查看etcd集群状态

#在任意一台集群上执行以下命令，这里选中了vm01

ETCDCTL_API=3 /opt/etcd/bin/etcdctl \
--cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/server.pem \
--key=/opt/etcd/ssl/server-key.pem \
--write-out=table \
--endpoints="https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379" endpoint health

#返回结果
+----------------------------+--------+-------------+-------+
|          ENDPOINT          | HEALTH |    TOOK     | ERROR |
+----------------------------+--------+-------------+-------+
| https://192.168.10.11:2379 |   true | 10.702229ms |       |
| https://192.168.10.13:2379 |   true |  18.81801ms |       |
| https://192.168.10.12:2379 |   true | 18.017598ms |       |
+----------------------------+--------+-------------+-------+


ETCDCTL_API=3 /opt/etcd/bin/etcdctl \
--cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/server.pem \
--key=/opt/etcd/ssl/server-key.pem \
--write-out=table \
--endpoints="https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379" endpoint status

#返回结果
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.10.11:2379 | 6571fb7574e87dba |   3.4.9 |   20 kB |      true |      false |         4 |          9 |                  9 |        |
| https://192.168.10.12:2379 | 9b449b0ff1d4c375 |   3.4.9 |   25 kB |     false |      false |         4 |          9 |                  9 |        |
| https://192.168.10.13:2379 | d1fbb74bc6a61e5c |   3.4.9 |   25 kB |     false |      false |         4 |          9 |                  9 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

至此，etcd集群已搭建完成，从上述表格来看，vm01(192.168.10.11)作为了主节点。如有问题请使用“tail -fn 500 /var/log/message”来查看系统日志进行分析。

11.安装docker-ce

11.1创建docker管理文件

#在vm01上进行操作,为了方便操作，将可执行文件和配置文件进行了分离
#可执行文件放在/opt/TLS/download/docker/bin下
#配置文件放在/opt/TLS/download/docker/cfg下

cd /opt/TLS/download
mkdir -p bin
mv docker/* bin
mv bin docker
mkdir -p docker/cfg
cd /opt/TLS/download/docker/cfg

#创建配置文件
cd /opt/TLS/download/docker/cfg
cat > docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/local/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF

tee daemon.json << 'EOF'
{
  "registry-mirrors": ["https://ung2thfc.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m"
  },
  "storage-driver": "overlay2"
}
EOF

#查看文件目录结构
[root@vm01 docker]# cd /opt/TLS/download/docker/
[root@vm01 docker]# tree ./
./
├── bin
│   ├── containerd
│   ├── containerd-shim
│   ├── containerd-shim-runc-v2
│   ├── ctr
│   ├── docker
│   ├── dockerd
│   ├── docker-init
│   ├── docker-proxy
│   └── runc
└── cfg
    ├── daemon.json
    └── docker.service

11.2分发文件

#创建docker目录
mkdir -p /etc/docker
ssh vm02 "mkdir -p /etc/docker"
ssh vm03 "mkdir -p /etc/docker"

#分发docker管理文件
scp /opt/TLS/download/docker/cfg/docker.service /usr/lib/systemd/system/docker.service
scp /opt/TLS/download/docker/cfg/docker.service vm02:/usr/lib/systemd/system/docker.service
scp /opt/TLS/download/docker/cfg/docker.service vm03:/usr/lib/systemd/system/docker.service

#分发docker配置文件
scp /opt/TLS/download/docker/cfg/daemon.json /etc/docker/daemon.json
scp /opt/TLS/download/docker/cfg/daemon.json vm02:/etc/docker/daemon.json
scp /opt/TLS/download/docker/cfg/daemon.json vm03:/etc/docker/daemon.json

#分发docker可执行文件
scp /opt/TLS/download/docker/bin/* /usr/local/bin
scp /opt/TLS/download/docker/bin/* vm02:/usr/local/bin
scp /opt/TLS/download/docker/bin/* vm03:/usr/local/bin

11.3核对文件

#核对docker管理文件
[root@vm01 docker]# ls -l /usr/lib/systemd/system/docker.service
-rw-r--r-- 1 root root 456 Apr  3 13:17 /usr/lib/systemd/system/docker.service
[root@vm01 docker]# ssh vm02 "ls -l /usr/lib/systemd/system/docker.service"
-rw-r--r-- 1 root root 456 Apr  3 13:17 /usr/lib/systemd/system/docker.service
[root@vm01 docker]# ssh vm03 "ls -l /usr/lib/systemd/system/docker.service"
-rw-r--r-- 1 root root 456 Apr  4 09:52 /usr/lib/systemd/system/docker.service

#核对docker配置文件
[root@vm01 docker]# ls -l /etc/docker/daemon.json
-rw-r--r-- 1 root root 219 Apr  3 13:17 /etc/docker/daemon.json
[root@vm01 docker]# ssh vm02 "ls -l /etc/docker/daemon.json"
-rw-r--r-- 1 root root 219 Apr  3 13:18 /etc/docker/daemon.json
[root@vm01 docker]# ssh vm03 "ls -l /etc/docker/daemon.json"
-rw-r--r-- 1 root root 219 Apr  4 09:52 /etc/docker/daemon.json

#核对docker可执行文件
[root@vm01 docker]# ls -l /usr/local/bin/
total 241072
-rwxr-xr-x 1 root root 16659824 Apr  3 12:34 cfssl
-rwxr-xr-x 1 root root 13502544 Apr  3 12:34 cfssl-certinfo
-rwxr-xr-x 1 root root 11029744 Apr  3 12:34 cfssljson
-rwxr-xr-x 1 root root 33908392 Apr  3 13:19 containerd
-rwxr-xr-x 1 root root  6508544 Apr  3 13:19 containerd-shim
-rwxr-xr-x 1 root root  8609792 Apr  3 13:19 containerd-shim-runc-v2
-rwxr-xr-x 1 root root 21131264 Apr  3 13:19 ctr
-rwxr-xr-x 1 root root 52883616 Apr  3 13:19 docker
-rwxr-xr-x 1 root root 64758736 Apr  3 13:19 dockerd
-rwxr-xr-x 1 root root   708616 Apr  3 13:19 docker-init
-rwxr-xr-x 1 root root  2784145 Apr  3 13:19 docker-proxy
-rwxr-xr-x 1 root root 14352296 Apr  3 13:19 runc
[root@vm01 docker]# ssh vm02 "ls -l /usr/local/bin/"
total 200840
-rwxr-xr-x 1 root root 33908392 Apr  3 13:19 containerd
-rwxr-xr-x 1 root root  6508544 Apr  3 13:19 containerd-shim
-rwxr-xr-x 1 root root  8609792 Apr  3 13:19 containerd-shim-runc-v2
-rwxr-xr-x 1 root root 21131264 Apr  3 13:19 ctr
-rwxr-xr-x 1 root root 52883616 Apr  3 13:19 docker
-rwxr-xr-x 1 root root 64758736 Apr  3 13:19 dockerd
-rwxr-xr-x 1 root root   708616 Apr  3 13:19 docker-init
-rwxr-xr-x 1 root root  2784145 Apr  3 13:19 docker-proxy
-rwxr-xr-x 1 root root 14352296 Apr  3 13:19 runc
[root@vm01 docker]# ssh vm03 "ls -l /usr/local/bin/"
total 200840
-rwxr-xr-x 1 root root 33908392 Apr  4 09:54 containerd
-rwxr-xr-x 1 root root  6508544 Apr  4 09:54 containerd-shim
-rwxr-xr-x 1 root root  8609792 Apr  4 09:54 containerd-shim-runc-v2
-rwxr-xr-x 1 root root 21131264 Apr  4 09:54 ctr
-rwxr-xr-x 1 root root 52883616 Apr  4 09:54 docker
-rwxr-xr-x 1 root root 64758736 Apr  4 09:54 dockerd
-rwxr-xr-x 1 root root   708616 Apr  4 09:54 docker-init
-rwxr-xr-x 1 root root  2784145 Apr  4 09:54 docker-proxy
-rwxr-xr-x 1 root root 14352296 Apr  4 09:54 runc

11.4启动docker

#在vm01上执行启动命令，设置开启启动，并查看状态
[root@vm01 docker]# systemctl daemon-reload && systemctl start docker && systemctl enable docker && systemctl status docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-04-03 13:26:46 CST; 72ms ago
     Docs: https://docs.docker.com
 Main PID: 1466 (dockerd)
   CGroup: /system.slice/docker.service
           ├─1466 /usr/local/bin/dockerd
           └─1471 containerd --config /var/run/docker/containerd/containerd.toml --log-level info

Apr 03 13:26:46 vm01 dockerd[1466]: time="2022-04-03T13:26:46.552291845+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Apr 03 13:26:46 vm01 dockerd[1466]: time="2022-04-03T13:26:46.577035980+08:00" level=warning msg="Your kernel does not support cgroup blkio weight"
Apr 03 13:26:46 vm01 dockerd[1466]: time="2022-04-03T13:26:46.577384262+08:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Apr 03 13:26:46 vm01 dockerd[1466]: time="2022-04-03T13:26:46.577753307+08:00" level=info msg="Loading containers: start."
Apr 03 13:26:46 vm01 dockerd[1466]: time="2022-04-03T13:26:46.654683641+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip ca...ed IP address"
Apr 03 13:26:46 vm01 dockerd[1466]: time="2022-04-03T13:26:46.696405877+08:00" level=info msg="Loading containers: done."
Apr 03 13:26:46 vm01 dockerd[1466]: time="2022-04-03T13:26:46.705318380+08:00" level=info msg="Docker daemon" commit=79ea9d3 graphdriver(s)=overlay2 version=20.10.9
Apr 03 13:26:46 vm01 dockerd[1466]: time="2022-04-03T13:26:46.705785575+08:00" level=info msg="Daemon has completed initialization"
Apr 03 13:26:46 vm01 systemd[1]: Started Docker Application Container Engine.
Apr 03 13:26:46 vm01 dockerd[1466]: time="2022-04-03T13:26:46.739607525+08:00" level=info msg="API listen on /var/run/docker.sock"
Hint: Some lines were ellipsized, use -l to show in full.


#在vm02上执行启动命令，设置开启启动，并查看状态
[root@vm02 ~]# systemctl daemon-reload && systemctl start docker && systemctl enable docker && systemctl status docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-04-03 13:26:53 CST; 84ms ago
     Docs: https://docs.docker.com
 Main PID: 1301 (dockerd)
   CGroup: /system.slice/docker.service
           ├─1301 /usr/local/bin/dockerd
           └─1307 containerd --config /var/run/docker/containerd/containerd.toml --log-level info

Apr 03 13:26:53 vm02 dockerd[1301]: time="2022-04-03T13:26:53.245105288+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Apr 03 13:26:53 vm02 dockerd[1301]: time="2022-04-03T13:26:53.267932539+08:00" level=warning msg="Your kernel does not support cgroup blkio weight"
Apr 03 13:26:53 vm02 dockerd[1301]: time="2022-04-03T13:26:53.268280419+08:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Apr 03 13:26:53 vm02 dockerd[1301]: time="2022-04-03T13:26:53.268627605+08:00" level=info msg="Loading containers: start."
Apr 03 13:26:53 vm02 dockerd[1301]: time="2022-04-03T13:26:53.356983369+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip ca...ed IP address"
Apr 03 13:26:53 vm02 dockerd[1301]: time="2022-04-03T13:26:53.402881653+08:00" level=info msg="Loading containers: done."
Apr 03 13:26:53 vm02 dockerd[1301]: time="2022-04-03T13:26:53.417527585+08:00" level=info msg="Docker daemon" commit=79ea9d3 graphdriver(s)=overlay2 version=20.10.9
Apr 03 13:26:53 vm02 dockerd[1301]: time="2022-04-03T13:26:53.417931806+08:00" level=info msg="Daemon has completed initialization"
Apr 03 13:26:53 vm02 systemd[1]: Started Docker Application Container Engine.
Apr 03 13:26:53 vm02 dockerd[1301]: time="2022-04-03T13:26:53.482157061+08:00" level=info msg="API listen on /var/run/docker.sock"
Hint: Some lines were ellipsized, use -l to show in full.


#在vm03上执行启动命令，设置开启启动，并查看状态
[root@vm03 ~]# systemctl daemon-reload && systemctl start docker && systemctl enable docker && systemctl status docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-04-04 10:00:48 CST; 79ms ago
     Docs: https://docs.docker.com
 Main PID: 1260 (dockerd)
   CGroup: /system.slice/docker.service
           ├─1260 /usr/local/bin/dockerd
           └─1266 containerd --config /var/run/docker/containerd/containerd.toml --log-level info

Apr 04 10:00:48 vm03 dockerd[1260]: time="2022-04-04T10:00:48.741931283+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Apr 04 10:00:48 vm03 dockerd[1260]: time="2022-04-04T10:00:48.762734549+08:00" level=warning msg="Your kernel does not support cgroup blkio weight"
Apr 04 10:00:48 vm03 dockerd[1260]: time="2022-04-04T10:00:48.763052152+08:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Apr 04 10:00:48 vm03 dockerd[1260]: time="2022-04-04T10:00:48.763369435+08:00" level=info msg="Loading containers: start."
Apr 04 10:00:48 vm03 dockerd[1260]: time="2022-04-04T10:00:48.843920653+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip ca...ed IP address"
Apr 04 10:00:48 vm03 dockerd[1260]: time="2022-04-04T10:00:48.896461096+08:00" level=info msg="Loading containers: done."
Apr 04 10:00:48 vm03 dockerd[1260]: time="2022-04-04T10:00:48.910089764+08:00" level=info msg="Docker daemon" commit=79ea9d3 graphdriver(s)=overlay2 version=20.10.9
Apr 04 10:00:48 vm03 dockerd[1260]: time="2022-04-04T10:00:48.910487468+08:00" level=info msg="Daemon has completed initialization"
Apr 04 10:00:48 vm03 systemd[1]: Started Docker Application Container Engine.
Apr 04 10:00:48 vm03 dockerd[1260]: time="2022-04-04T10:00:48.942539314+08:00" level=info msg="API listen on /var/run/docker.sock"
Hint: Some lines were ellipsized, use -l to show in full.

#在vm01、vm02、vm03上执行“docker info”命令，看到如下信息即可
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.9
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8
 runc version: v1.0.2-0-g52b36a2d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.17.1-1.el7.elrepo.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.907GiB
 Name: vm01
 ID: 3WPS:FK3T:D5HX:ZSNS:D6NE:NGNQ:TWTO:OE6B:HQYG:SXAQ:6J2V:PA6K
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://ung2thfc.mirror.aliyuncs.com/
 Live Restore Enabled: false
 Product License: Community Engine

至此，所有节点上的docker已部署完成。

12.部署Master

12.1自签CA证书

12.1.1生成CA证书配置

cd /opt/TLS/k8s/ssl
cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json << EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

12.1.2生成CA证书

#生成CA证书文件
[root@vm01 ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2022/04/03 13:38:51 [INFO] generating a new CA key and certificate from CSR
2022/04/03 13:38:51 [INFO] generate received request
2022/04/03 13:38:51 [INFO] received CSR
2022/04/03 13:38:51 [INFO] generating key: rsa-2048
2022/04/03 13:38:51 [INFO] encoded CSR
2022/04/03 13:38:51 [INFO] signed certificate with serial number 652185253661746806409242928399719456314448070149

#查看已生成的证书文件
[root@vm01 ssl]# ll
total 20
-rw-r--r-- 1 root root  294 Apr  3 13:37 ca-config.json
-rw-r--r-- 1 root root 1001 Apr  3 13:38 ca.csr
-rw-r--r-- 1 root root  264 Apr  3 13:37 ca-csr.json
-rw------- 1 root root 1675 Apr  3 13:38 ca-key.pem
-rw-r--r-- 1 root root 1310 Apr  3 13:38 ca.pem

#这里生成了ca.pem和ca-key.pem两个文件

12.2部署Apiserver

12.2.1创建证书申请文件

cd /opt/TLS/k8s/ssl
cat > server-csr.json << EOF
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "192.168.10.11",
      "192.168.10.12",
      "192.168.10.13",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

#上述文件hosts字段中IP为所有Master IP，一个都不能少！为了方便后期扩容可以多写几个预留的IP

12.2.2签发apiserver 证书

[root@vm01 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
2022/04/03 13:55:17 [INFO] generate received request
2022/04/03 13:55:17 [INFO] received CSR
2022/04/03 13:55:17 [INFO] generating key: rsa-2048
2022/04/03 13:55:17 [INFO] encoded CSR
2022/04/03 13:55:17 [INFO] signed certificate with serial number 427283511171072372380793803662853692846755378337

#查看已生成的证书文件
[root@vm01 ssl]# ll
total 36
-rw-r--r-- 1 root root  294 Apr  3 13:37 ca-config.json
-rw-r--r-- 1 root root 1001 Apr  3 13:38 ca.csr
-rw-r--r-- 1 root root  264 Apr  3 13:37 ca-csr.json
-rw------- 1 root root 1675 Apr  3 13:38 ca-key.pem
-rw-r--r-- 1 root root 1310 Apr  3 13:38 ca.pem
-rw-r--r-- 1 root root 1261 Apr  3 13:55 server.csr
-rw-r--r-- 1 root root  557 Apr  3 13:55 server-csr.json
-rw------- 1 root root 1675 Apr  3 13:55 server-key.pem
-rw-r--r-- 1 root root 1627 Apr  3 13:55 server.pem

#这里生成了server.pem和server-key.pem两个文件

12.2.3创建配置文件

cd /opt/TLS/k8s/cfg

cat > kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--insecure-port=0 \\
--etcd-servers=https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379 \\
--bind-address=192.168.10.11 \\
--secure-port=6443 \\
--advertise-address=192.168.10.11 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname,InternalDNS,ExternalDNS \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--service-account-issuer=api \\
--service-account-signing-key-file=/opt/kubernetes/ssl/server-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \\
--proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \\
--requestheader-allowed-names=kubernetes \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--enable-aggregator-routing=true \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF


# 上面两个\\ 第一个是转义符，第二个是换行符，使用转义符是为了使用EOF保留换行符。
# • --logtostderr：启用日志
# • ---v：日志等级
# • --log-dir：日志目录
# • --etcd-servers：etcd集群地址
# • --bind-address：监听地址
# • --secure-port：https安全端口
# • --advertise-address：集群通告地址
# • --allow-privileged：启用授权
# • --service-cluster-ip-range：Service虚拟IP地址段
# • --enable-admission-plugins：准入控制模块
# • --authorization-mode：认证授权，启用RBAC授权和节点自管理
# • --enable-bootstrap-token-auth：启用TLS bootstrap机制
# • --token-auth-file：bootstrap token文件
# • --service-node-port-range：Service nodeport类型默认分配端口范围
# • --kubelet-client-xxx：apiserver访问kubelet客户端证书
# • --tls-xxx-file：apiserver https证书
# • 1.20以上版本必须加的参数：--service-account-issuer，--service-account-signing-key-file
# • --etcd-xxxfile：连接Etcd集群证书
# • --audit-log-xxx：审计日志
# • 启动聚合层相关配置：
# • --requestheader-client-ca-file，--proxy-client-cert-file，--proxy-client-key-file，
# • --requestheader-allowed-names，--requestheader-extra-headers-prefix，
# • --requestheader-group-headers，--requestheader-username-headers，
# • --enable-aggregator-routing

12.2.4启用 TLS Bootstrapping 机制

TLS Bootstraping：Master apiserver启用TLS认证后，Node节点kubelet和kube-proxy要与kube-apiserver进行通信，必须使用CA签发的有效证书才可以，当Node节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。为了简化流程，Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式，目前主要用于kubelet，kube-proxy还是由我们统一颁发一个证书。

#创建token文件
cat > token.csv << EOF
c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

# 格式：token，用户名，UID，用户组
# token也可自行生成替换：
# head -c 16 /dev/urandom | od -An -t x | tr -d ' '

12.2.5创建管理文件

cat > kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

#查看上述命令生成的相关文件
[root@vm01 cfg]# ll
total 12
-rw-r--r-- 1 root root 1815 Apr  3 13:57 kube-apiserver.conf
-rw-r--r-- 1 root root  286 Apr  3 14:06 kube-apiserver.service
-rw-r--r-- 1 root root   84 Apr  3 13:57 token.csv

12.2.7分发文件

#创建kubernetes目录
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} 

#拷贝证书文件
scp -r /opt/TLS/k8s/ssl/*pem /opt/kubernetes/ssl/

#拷贝配置文件
scp -r /opt/TLS/k8s/cfg/token.csv /opt/kubernetes/cfg/
scp /opt/TLS/k8s/cfg/kube-apiserver.conf /opt/kubernetes/cfg/kube-apiserver.conf

#拷贝管理文件
scp /opt/TLS/k8s/cfg/kube-apiserver.service /usr/lib/systemd/system/kube-apiserver.service

#拷贝可执行文件
scp /opt/TLS/download/kubernetes/server/bin/{kube-apiserver,kube-scheduler,kube-controller-manager} /opt/kubernetes/bin
scp /opt/TLS/download/kubernetes/server/bin/kubectl /usr/local/bin/

12.2.8核对文件

#核对证书文件
[root@vm01 cfg]# ll /opt/kubernetes/ssl/
total 16
-rw------- 1 root root 1675 Apr  3 14:11 ca-key.pem
-rw-r--r-- 1 root root 1310 Apr  3 14:11 ca.pem
-rw------- 1 root root 1675 Apr  3 14:11 server-key.pem
-rw-r--r-- 1 root root 1627 Apr  3 14:11 server.pem

#核对配置文件
[root@vm01 cfg]# ll /opt/kubernetes/cfg/token.csv
-rw-r--r-- 1 root root 84 Apr  3 14:11 /opt/kubernetes/cfg/token.csv

[root@vm01 cfg]# ll /opt/kubernetes/cfg/kube-apiserver.conf
-rw-r--r-- 1 root root 1815 Apr  3 14:12 /opt/kubernetes/cfg/kube-apiserver.conf

#核对管理文件
[root@vm01 cfg]# ll /usr/lib/systemd/system/kube-apiserver.service
-rw-r--r-- 1 root root 286 Apr  3 14:11 /usr/lib/systemd/system/kube-apiserver.service

#核对可执行文件
[root@vm01 cfg]# ll /opt/kubernetes/bin/{kube-apiserver,kube-scheduler,kube-controller-manager}
-rwxr-xr-x 1 root root 131301376 Apr  3 14:12 /opt/kubernetes/bin/kube-apiserver
-rwxr-xr-x 1 root root 121110528 Apr  3 14:12 /opt/kubernetes/bin/kube-controller-manager
-rwxr-xr-x 1 root root  49618944 Apr  3 14:12 /opt/kubernetes/bin/kube-scheduler

[root@vm01 cfg]# ll /usr/local/bin/kubectl
-rwxr-xr-x 1 root root 46592000 Apr  3 14:12 /usr/local/bin/kubectl

12.2.9启动kube-apiserver

[root@vm01 cfg]# systemctl daemon-reload && systemctl start kube-apiserver && systemctl enable kube-apiserver && systemctl status kube-apiserver
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.
● kube-apiserver.service - Kubernetes API Server
   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-04-03 14:14:54 CST; 111ms ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 11765 (kube-apiserver)
   CGroup: /system.slice/kube-apiserver.service
           └─11765 /opt/kubernetes/bin/kube-apiserver --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --insecure-port=0 --etcd-servers=https://192.168.10.11:2379,https://192.168.10.12:2379,http...

Apr 03 14:14:54 vm01 systemd[1]: Started Kubernetes API Server.

12.3部署ControllerManager

12.3.1创建配置文件

cd /opt/TLS/k8s/cfg
cat > kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--cluster-signing-duration=87600h0m0s"
EOF

# • --kubeconfig：连接apiserver配置文件
# • --leader-elect：当该组件启动多个时，自动选举（HA）
# • --cluster-signing-cert-file/--cluster-signing-key-file：自动为kubelet颁发证书的CA，与apiserver保持一致

12.3.2生成证书配置文件

cd /opt/TLS/k8s/ssl
cat > kube-controller-manager-csr.json << EOF
{
  "CN": "system:kube-controller-manager",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing", 
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

12.3.3生成证书文件

[root@vm01 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
2022/04/03 14:19:13 [INFO] generate received request
2022/04/03 14:19:13 [INFO] received CSR
2022/04/03 14:19:13 [INFO] generating key: rsa-2048
2022/04/03 14:19:13 [INFO] encoded CSR
2022/04/03 14:19:13 [INFO] signed certificate with serial number 207379066893533311974100622812990123367796996104
2022/04/03 14:19:13 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

[root@vm01 ssl]# ll kube-controller-manager*
-rw-r--r-- 1 root root 1045 Apr  3 14:19 kube-controller-manager.csr
-rw-r--r-- 1 root root  255 Apr  3 14:18 kube-controller-manager-csr.json
-rw------- 1 root root 1679 Apr  3 14:19 kube-controller-manager-key.pem
-rw-r--r-- 1 root root 1436 Apr  3 14:19 kube-controller-manager.pem
#这里生成了kube-controller-manager.pem和kube-controller-manager-key.pem文件

12.3.4生成kubeconfig文件

# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.11:6443 \
  --kubeconfig=/opt/TLS/k8s/cfg/kube-controller-manager.kubeconfig
  
# 设置客户端认证参数
kubectl config set-credentials kube-controller-manager \
  --client-certificate=./kube-controller-manager.pem \
  --client-key=./kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=/opt/TLS/k8s/cfg/kube-controller-manager.kubeconfig
  
# 设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-controller-manager \
  --kubeconfig=/opt/TLS/k8s/cfg/kube-controller-manager.kubeconfig
  
# 设置默认上下文
kubectl config use-context default --kubeconfig=/opt/TLS/k8s/cfg/kube-controller-manager.kubeconfig

12.3.5生成管理文件

cd /opt/TLS/k8s/cfg

cat > kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

12.3.6分发文件

#分发证书文件
scp -r /opt/TLS/k8s/ssl/kube-controller-manager*.pem /opt/kubernetes/ssl/

#分发配置文件
scp -r /opt/TLS/k8s/cfg/kube-controller-manager.conf /opt/kubernetes/cfg/

#分发管理文件
scp /opt/TLS/k8s/cfg/kube-controller-manager.service /usr/lib/systemd/system/kube-controller-manager.service

#分发kubeconfig文件
scp /opt/TLS/k8s/cfg/kube-controller-manager.kubeconfig /opt/kubernetes/cfg/kube-controller-manager.kubeconfig

12.3.7核对文件

#核对证书文件
[root@vm01 cfg]# ll /opt/kubernetes/ssl/kube-controller-manager*.pem
-rw------- 1 root root 1679 Apr  3 14:30 /opt/kubernetes/ssl/kube-controller-manager-key.pem
-rw-r--r-- 1 root root 1436 Apr  3 14:30 /opt/kubernetes/ssl/kube-controller-manager.pem

#核对配置文件
[root@vm01 cfg]# ll /opt/kubernetes/cfg/kube-controller-manager.conf
-rw-r--r-- 1 root root 582 Apr  3 14:30 /opt/kubernetes/cfg/kube-controller-manager.conf

#核对管理文件
[root@vm01 cfg]# ll /usr/lib/systemd/system/kube-controller-manager.service
-rw-r--r-- 1 root root 321 Apr  3 14:30 /usr/lib/systemd/system/kube-controller-manager.service

#核对kubeconfig文件
[root@vm01 cfg]# ll /opt/kubernetes/cfg/kube-controller-manager.kubeconfig
-rw------- 1 root root 6279 Apr  3 14:30 /opt/kubernetes/cfg/kube-controller-manager.kubeconfig

12.3.8启动ControllerManager

[root@vm01 cfg]# systemctl daemon-reload && systemctl start kube-controller-manager && systemctl enable kube-controller-manager && systemctl status kube-controller-manager
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
● kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-04-03 14:33:09 CST; 111ms ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 11872 (kube-controller)
   CGroup: /system.slice/kube-controller-manager.service
           └─11872 /opt/kubernetes/bin/kube-controller-manager --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --leader-elect=true --kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubec...

Apr 03 14:33:09 vm01 systemd[1]: Started Kubernetes Controller Manager.

12.4部署Scheduler

12.4.1生成配置文件

cd /opt/TLS/k8s/cfg/
cat > kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect \\
--kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\
--bind-address=127.0.0.1"
EOF

12.4.2生成证书配置文件

cd /opt/TLS/k8s/ssl
cat > kube-scheduler-csr.json << EOF
{
  "CN": "system:kube-scheduler",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

12.4.3生成证书文件

[root@vm01 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
2022/04/03 14:37:29 [INFO] generate received request
2022/04/03 14:37:29 [INFO] received CSR
2022/04/03 14:37:29 [INFO] generating key: rsa-2048
2022/04/03 14:37:29 [INFO] encoded CSR
2022/04/03 14:37:29 [INFO] signed certificate with serial number 270861181620040490080757616258059917703352589307
2022/04/03 14:37:29 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

#查看已生成的证书文件
[root@vm01 ssl]# ll kube-scheduler*
-rw-r--r-- 1 root root 1029 Apr  3 14:37 kube-scheduler.csr
-rw-r--r-- 1 root root  245 Apr  3 14:37 kube-scheduler-csr.json
-rw------- 1 root root 1675 Apr  3 14:37 kube-scheduler-key.pem
-rw-r--r-- 1 root root 1424 Apr  3 14:37 kube-scheduler.pem
#这里生成了kube-scheduler.pem和kube-scheduler-key.pem文件

12.4.4生成kubeconfig文件

# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.11:6443 \
  --kubeconfig=/opt/TLS/k8s/cfg/kube-scheduler.kubeconfig
  
# 设置客户端认证参数
kubectl config set-credentials kube-scheduler \
  --client-certificate=./kube-scheduler.pem \
  --client-key=./kube-scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=/opt/TLS/k8s/cfg/kube-scheduler.kubeconfig
  
# 设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-scheduler \
  --kubeconfig=/opt/TLS/k8s/cfg/kube-scheduler.kubeconfig
  
# 设置默认上下文
kubectl config use-context default --kubeconfig=/opt/TLS/k8s/cfg/kube-scheduler.kubeconfig

12.4.5生成管理文件

cd /opt/TLS/k8s/cfg
cat > kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

12.4.6分发文件

#分发配置文件
scp /opt/TLS/k8s/cfg/kube-scheduler.conf  /opt/kubernetes/cfg/kube-scheduler.conf

#分发证书文件
scp /opt/TLS/k8s/ssl/kube-scheduler*.pem /opt/kubernetes/ssl/

#分发kubeconfig文件
scp /opt/TLS/k8s/cfg/kube-scheduler.kubeconfig /opt/kubernetes/cfg/kube-scheduler.kubeconfig

#分发管理文件
scp /opt/TLS/k8s/cfg/kube-scheduler.service /usr/lib/systemd/system/kube-scheduler.service

12.4.7核对文件

#核对配置文件
[root@vm01 cfg]# ll /opt/kubernetes/cfg/kube-scheduler.conf
-rw-r--r-- 1 root root 188 Apr  3 14:44 /opt/kubernetes/cfg/kube-scheduler.conf

#核对证书文件
[root@vm01 cfg]# ll /opt/kubernetes/ssl/kube-scheduler*.pem
-rw------- 1 root root 1675 Apr  3 14:45 /opt/kubernetes/ssl/kube-scheduler-key.pem
-rw-r--r-- 1 root root 1424 Apr  3 14:45 /opt/kubernetes/ssl/kube-scheduler.pem

#核对kubeconfig文件
[root@vm01 cfg]# ll /opt/kubernetes/cfg/kube-scheduler.kubeconfig
-rw------- 1 root root 6241 Apr  3 14:45 /opt/kubernetes/cfg/kube-scheduler.kubeconfig

#核对管理文件
[root@vm01 cfg]# ll /usr/lib/systemd/system/kube-scheduler.service
-rw-r--r-- 1 root root 285 Apr  3 14:45 /usr/lib/systemd/system/kube-scheduler.service

12.4.8启动scheduler

[root@vm01 cfg]# systemctl daemon-reload && systemctl start kube-scheduler && systemctl enable kube-scheduler && systemctl status kube-scheduler
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
● kube-scheduler.service - Kubernetes Scheduler
   Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-04-03 14:48:19 CST; 113ms ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 11972 (kube-scheduler)
   CGroup: /system.slice/kube-scheduler.service
           └─11972 /opt/kubernetes/bin/kube-scheduler --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --leader-elect --kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig --bind-address=12...

Apr 03 14:48:19 vm01 systemd[1]: Started Kubernetes Scheduler.
Apr 03 14:48:19 vm01 kube-scheduler[11972]: Flag --logtostderr has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig...k8s-components
Apr 03 14:48:19 vm01 kube-scheduler[11972]: Flag --log-dir has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-ins...k8s-components
Hint: Some lines were ellipsized, use -l to show in full.

至此，Master节点上的三个组件（Apiserver、ControllerManager、Scheduler）已部署并启动成功，下面来检查一下所有组件的状态吧。

13.检查集群组件状态

13.1生成连接集群证书配置

cd /opt/TLS/k8s/ssl
cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

13.2生成连接证书

[root@vm01 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
2022/04/03 14:52:53 [INFO] generate received request
2022/04/03 14:52:53 [INFO] received CSR
2022/04/03 14:52:53 [INFO] generating key: rsa-2048
2022/04/03 14:52:53 [INFO] encoded CSR
2022/04/03 14:52:53 [INFO] signed certificate with serial number 544157816284296715610790502652620056833806648888
2022/04/03 14:52:53 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

#查看已生成的证书
[root@vm01 ssl]# ll admin*
-rw-r--r-- 1 root root 1009 Apr  3 14:52 admin.csr
-rw-r--r-- 1 root root  229 Apr  3 14:52 admin-csr.json
-rw------- 1 root root 1679 Apr  3 14:52 admin-key.pem
-rw-r--r-- 1 root root 1399 Apr  3 14:52 admin.pem

13.3生成kubeconfig文件

cd /opt/TLS/k8s/cfg

# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.11:6443 \
  --kubeconfig=/opt/TLS/k8s/cfg/config

# 设置客户端认证参数
kubectl config set-credentials cluster-admin \
  --client-certificate=/opt/TLS/k8s/ssl/admin.pem \
  --client-key=/opt/TLS/k8s/ssl/admin-key.pem \
  --embed-certs=true \
  --kubeconfig=/opt/TLS/k8s/cfg/config

#设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user=cluster-admin \
  --kubeconfig=/opt/TLS/k8s/cfg/config

#设置默认上下文
kubectl config use-context default --kubeconfig=/opt/TLS/k8s/cfg/config

13.4分发文件

mkdir /root/.kube
scp /opt/TLS/k8s/cfg/config /root/.kube/config

13.5查看集群组件状态

#通过kubectl工具查看当前集群组件状态
[root@vm01 cfg]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
etcd-0               Healthy   {"health":"true"}
controller-manager   Healthy   ok
etcd-2               Healthy   {"health":"true"}
etcd-1               Healthy   {"health":"true"}

#输出以上信息说明Master节点组件运行正常

14.授权用户允许请求证书

kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created

15.部署WorkNode节点

因为本机资源的限制，我们可以让Master Node上兼任Worker Node角色

15.1创建工作目录

mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
ssh vm02 "mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}"
ssh vm03 "mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}"

15.2分发文件

scp -r /opt/TLS/download/kubernetes/server/bin/{kubelet,kube-proxy} /opt/kubernetes/bin
scp /opt/TLS/download/kubernetes/server/bin/kubelet /usr/local/bin

15.3核对文件

[root@vm01 cfg]# ll /opt/kubernetes/bin/{kubelet,kube-proxy}
-rwxr-xr-x 1 root root 124521440 Apr  3 15:09 /opt/kubernetes/bin/kubelet
-rwxr-xr-x 1 root root  44163072 Apr  3 15:09 /opt/kubernetes/bin/kube-proxy

[root@vm01 cfg]# ll /usr/local/bin/kubelet
-rwxr-xr-x 1 root root 124521440 Apr  3 15:10 /usr/local/bin/kubelet

15.4部署kubelet

15.4.1创建配置文件

这里为了方便，一次性创建了3台虚拟机上的kubelet配置文件，然后将对应的配置文件分发至不同的机器即可

cd /opt/TLS/k8s/cfg/
cat > kubelet01.conf << EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--hostname-override=vm01 \\
--network-plugin=cni \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=ibmcom/pause-amd64:3.1"
EOF

cat > kubelet02.conf << EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--hostname-override=vm02 \\
--network-plugin=cni \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=ibmcom/pause-amd64:3.1"
EOF

cat > kubelet03.conf << EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--hostname-override=vm03 \\
--network-plugin=cni \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=ibmcom/pause-amd64:3.1"
EOF

# • --hostname-override：显示名称，集群中唯一
# • --network-plugin：启用CNI
# • --kubeconfig：空路径，会自动生成，后面用于连接apiserver
# • --bootstrap-kubeconfig：首次启动向apiserver申请证书
# • --config：配置参数文件
# • --cert-dir：kubelet证书生成目录
# • --pod-infra-container-image：管理Pod网络容器的镜像

15.4.2配置参数文件

cat > kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: systemd
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local 
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/kubernetes/ssl/ca.pem 
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF

15.4.3创建管理文件

cat > kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

15.4.4创建kubeconfig文件

# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.11:6443 \
  --kubeconfig=/opt/TLS/k8s/cfg/bootstrap.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials "kubelet-bootstrap" \
  --token=c47ffb939f5ca36231d9e3121a252940 \
  --kubeconfig=/opt/TLS/k8s/cfg/bootstrap.kubeconfig

# 设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user="kubelet-bootstrap" \
  --kubeconfig=/opt/TLS/k8s/cfg/bootstrap.kubeconfig

# 设置默认上下文
kubectl config use-context default --kubeconfig=/opt/TLS/k8s/cfg/bootstrap.kubeconfig

15.4.5分发文件

#分发配置文件
scp /opt/TLS/k8s/cfg/kubelet01.conf /opt/kubernetes/cfg/kubelet.conf

#分发参数文件
scp /opt/TLS/k8s/cfg/kubelet-config.yml /opt/kubernetes/cfg/kubelet-config.yml

#分发kubeconfig文件
scp /opt/TLS/k8s/cfg/bootstrap.kubeconfig /opt/kubernetes/cfg/bootstrap.kubeconfig

#分发管理文件
scp /opt/TLS/k8s/cfg/kubelet.service /usr/lib/systemd/system/kubelet.service

15.4.6核对文件

#核对配置文件
[root@vm01 cfg]# ll /opt/kubernetes/cfg/kubelet.conf
-rw-r--r-- 1 root root 382 Apr  3 15:19 /opt/kubernetes/cfg/kubelet.conf

#核对参数文件
[root@vm01 cfg]# ll /opt/kubernetes/cfg/kubelet-config.yml
-rw-r--r-- 1 root root 610 Apr  3 15:19 /opt/kubernetes/cfg/kubelet-config.yml

#核对kubeconfig文件
[root@vm01 cfg]# ll /opt/kubernetes/cfg/bootstrap.kubeconfig
-rw------- 1 root root 2103 Apr  3 15:19 /opt/kubernetes/cfg/bootstrap.kubeconfig

#核对管理文件
[root@vm01 cfg]# ll /usr/lib/systemd/system/kubelet.service
-rw-r--r-- 1 root root 246 Apr  3 15:19 /usr/lib/systemd/system/kubelet.service

15.4.7启动kubelet

[root@vm01 cfg]# systemctl daemon-reload && systemctl start kubelet && systemctl enable kubelet && systemctl status kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-04-03 15:22:33 CST; 113ms ago
 Main PID: 12121 (kubelet)
   CGroup: /system.slice/kubelet.service
           └─12121 /opt/kubernetes/bin/kubelet --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --hostname-override=vm01 --network-plugin=cni --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig ...

Apr 03 15:22:33 vm01 systemd[1]: Started Kubernetes Kubelet.

15.4.8批准kubelet证书申请

#查看kubelet证书请求
[root@vm01 cfg]# kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           REQUESTEDDURATION   CONDITION
node-csr-6mDDHTg4HuOsVY_7oJRUqtS-6YQFe7JytpYdbRs9kek   57s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Pending

#批准申请
[root@vm01 cfg]# kubectl certificate approve node-csr-6mDDHTg4HuOsVY_7oJRUqtS-6YQFe7JytpYdbRs9kek
certificatesigningrequest.certificates.k8s.io/node-csr-6mDDHTg4HuOsVY_7oJRUqtS-6YQFe7JytpYdbRs9kek approved

#查看证书请求状态
[root@vm01 cfg]# kubectl get csr
NAME                                                   AGE    SIGNERNAME                                    REQUESTOR           REQUESTEDDURATION   CONDITION
node-csr-6mDDHTg4HuOsVY_7oJRUqtS-6YQFe7JytpYdbRs9kek   111s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued
#查看集群节点
[root@vm01 cfg]# kubectl get nodes
NAME   STATUS     ROLES    AGE   VERSION
vm01   NotReady   <none>   32s   v1.23.4

# 由于网络插件还没有部署，节点会没有准备就绪 NotReady

15.5部署kube-proxy

15.5.1创建配置文件

cd /opt/TLS/k8s/cfg/
cat > kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"
EOF

15.5.2创建参数文件

cat > kube-proxy-config01.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: vm01
clusterCIDR: 10.244.0.0/16
mode: ipvs
ipvs:
  scheduler: "rr"
iptables:
  masqueradeAll: true
EOF

cat > kube-proxy-config02.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: vm02
clusterCIDR: 10.244.0.0/16
mode: ipvs
ipvs:
  scheduler: "rr"
iptables:
  masqueradeAll: true
EOF

cat > kube-proxy-config03.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: vm03
clusterCIDR: 10.244.0.0/16
mode: ipvs
ipvs:
  scheduler: "rr"
iptables:
  masqueradeAll: true
EOF

15.5.3生成证书配置文件

cd /opt/TLS/k8s/ssl
cat > kube-proxy-csr.json << EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

15.5.4生成证书文件

[root@vm01 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
2022/04/03 15:30:38 [INFO] generate received request
2022/04/03 15:30:38 [INFO] received CSR
2022/04/03 15:30:38 [INFO] generating key: rsa-2048
2022/04/03 15:30:38 [INFO] encoded CSR
2022/04/03 15:30:38 [INFO] signed certificate with serial number 117156627576808648708142496682355499174590336333
2022/04/03 15:30:38 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

#查看已生成的证书
[root@vm01 ssl]# ll  kube-proxy*
-rw-r--r-- 1 root root 1009 Apr  3 15:30 kube-proxy.csr
-rw-r--r-- 1 root root  230 Apr  3 15:30 kube-proxy-csr.json
-rw------- 1 root root 1679 Apr  3 15:30 kube-proxy-key.pem
-rw-r--r-- 1 root root 1403 Apr  3 15:30 kube-proxy.pem

15.5.5生成kubeconfig文件

# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.11:6443 \
  --kubeconfig=/opt/TLS/k8s/cfg/kube-proxy.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=/opt/TLS/k8s/ssl/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=/opt/TLS/k8s/cfg/kube-proxy.kubeconfig

# 设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=/opt/TLS/k8s/cfg/kube-proxy.kubeconfig

# 设置默认上下文
kubectl config use-context default --kubeconfig=/opt/TLS/k8s/cfg/kube-proxy.kubeconfig

15.5.6生成管理文件

cd /opt/TLS/k8s/cfg
cat > kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

15.5.7分发文件

scp /opt/TLS/k8s/ssl/kube-proxy*.pem /opt/kubernetes/ssl
scp /opt/TLS/k8s/cfg/kube-proxy.conf /opt/kubernetes/cfg/kube-proxy.conf
scp /opt/TLS/k8s/cfg/kube-proxy-config01.yml /opt/kubernetes/cfg/kube-proxy-config.yml
scp /opt/TLS/k8s/cfg/kube-proxy.kubeconfig /opt/kubernetes/cfg/kube-proxy.kubeconfig
scp /opt/TLS/k8s/cfg/kube-proxy.service /usr/lib/systemd/system/kube-proxy.service

15.5.8核对文件

[root@vm01 cfg]# ll /opt/kubernetes/ssl/kube-proxy*.pem
-rw------- 1 root root 1679 Apr  3 15:35 /opt/kubernetes/ssl/kube-proxy-key.pem
-rw-r--r-- 1 root root 1403 Apr  3 15:35 /opt/kubernetes/ssl/kube-proxy.pem

[root@vm01 cfg]# ll /opt/kubernetes/cfg/kube-proxy.conf
-rw-r--r-- 1 root root 132 Apr  3 15:35 /opt/kubernetes/cfg/kube-proxy.conf

[root@vm01 cfg]# ll /opt/kubernetes/cfg/kube-proxy-config.yml
-rw-r--r-- 1 root root 320 Apr  3 15:35 /opt/kubernetes/cfg/kube-proxy-config.yml

[root@vm01 cfg]# ll /opt/kubernetes/cfg/kube-proxy.kubeconfig
-rw------- 1 root root 6209 Apr  3 15:35 /opt/kubernetes/cfg/kube-proxy.kubeconfig

[root@vm01 cfg]# ll /usr/lib/systemd/system/kube-proxy.service
-rw-r--r-- 1 root root 253 Apr  3 15:35 /usr/lib/systemd/system/kube-proxy.service

15.5.9启动kube-proxy

[root@vm01 cfg]# systemctl daemon-reload && systemctl start kube-proxy && systemctl enable kube-proxy && systemctl status kube-proxy
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
● kube-proxy.service - Kubernetes Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-04-03 15:36:32 CST; 118ms ago
 Main PID: 13681 (kube-proxy)
   CGroup: /system.slice/kube-proxy.service
           ├─13681 /opt/kubernetes/bin/kube-proxy --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --config=/opt/kubernetes/cfg/kube-proxy-config.yml
           └─13708 modprobe -- ip_vs_sh

Apr 03 15:36:32 vm01 systemd[1]: Started Kubernetes Proxy.
Apr 03 15:36:32 vm01 kube-proxy[13681]: Flag --logtostderr has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-ins...k8s-components
Apr 03 15:36:32 vm01 kube-proxy[13681]: Flag --log-dir has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrum...k8s-components
Hint: Some lines were ellipsized, use -l to show in full.

16.新增其他WorkNode

16.2新增vm02

16.2.1分发文件

#此操作在Master（vm01）上进行

#分发kubernetes工作目录
scp -r /opt/kubernetes root@192.168.10.12:/opt/

#分发kubelet,kube-proxy的管理文件
scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.10.12:/usr/lib/systemd/system

#分发证书文件
scp /opt/kubernetes/ssl/ca.pem root@192.168.10.12:/opt/kubernetes/ssl

#替换kubelet.conf文件
scp /opt/TLS/k8s/cfg/kubelet02.conf vm02:/opt/kubernetes/cfg/kubelet.conf

#替换kube-proxy-config.yml
scp /opt/TLS/k8s/cfg/kube-proxy-config02.yml vm02:/opt/kubernetes/cfg/kube-proxy-config.yml


#删除kubelet证书和kubeconfig文件
ssh vm02 "rm -f /opt/kubernetes/cfg/kubelet.kubeconfig"
ssh vm02 "rm -f /opt/kubernetes/ssl/kubelet*"

16.2.2核对文件

#此操作在vm02上进行

[root@vm02 ~]# ll /opt/kubernetes
total 12
drwxr-xr-x 2 root root  114 Apr  3 15:47 bin
drwxr-xr-x 2 root root 4096 Apr  3 15:48 cfg
drwxr-xr-x 2 root root 4096 Apr  3 15:47 logs
drwxr-xr-x 2 root root 4096 Apr  3 15:48 ssl

[root@vm02 ~]# ll /usr/lib/systemd/system/{kubelet,kube-proxy}.service
-rw-r--r-- 1 root root 246 Apr  3 15:47 /usr/lib/systemd/system/kubelet.service
-rw-r--r-- 1 root root 253 Apr  3 15:47 /usr/lib/systemd/system/kube-proxy.service

[root@vm02 ~]# ll /opt/kubernetes/ssl/ca.pem
-rw-r--r-- 1 root root 1310 Apr  3 15:47 /opt/kubernetes/ssl/ca.pem

[root@vm02 ~]# ll /opt/kubernetes/cfg/kubelet.conf
-rw-r--r-- 1 root root 382 Apr  3 15:48 /opt/kubernetes/cfg/kubelet.conf

[root@vm02 ~]# cat /opt/kubernetes/cfg/kubelet.conf
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--hostname-override=vm02 \
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=ibmcom/pause-amd64:3.1"

[root@vm02 ~]# ll /opt/kubernetes/cfg/kube-proxy-config.yml
-rw-r--r-- 1 root root 320 Apr  3 15:48 /opt/kubernetes/cfg/kube-proxy-config.yml

[root@vm02 ~]# cat /opt/kubernetes/cfg/kubelet.conf
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--hostname-override=vm02 \
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=ibmcom/pause-amd64:3.1"
[root@vm02 ~]# cat /opt/kubernetes/cfg/kube-proxy-config.yml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: vm02
clusterCIDR: 10.244.0.0/16
mode: ipvs
ipvs:
  scheduler: "rr"
iptables:
  masqueradeAll: true

[root@vm02 ~]# ll /opt/kubernetes/cfg/kubelet.kubeconfig
ls: cannot access /opt/kubernetes/cfg/kubelet.kubeconfig: No such file or directory

[root@vm02 ~]# ll /opt/kubernetes/ssl/kubelet*
ls: cannot access /opt/kubernetes/ssl/kubelet*: No such file or directory

16.2.3启动kubelet

#此操作在vm02上进行
[root@vm02 ~]# systemctl daemon-reload && systemctl start kubelet && systemctl enable kubelet && systemctl status kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-04-03 15:52:53 CST; 109ms ago
 Main PID: 11629 (kubelet)
   CGroup: /system.slice/kubelet.service
           └─11629 /opt/kubernetes/bin/kubelet --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --hostname-override=vm02 --network-plugin=cni --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig ...

Apr 03 15:52:53 vm02 systemd[1]: Started Kubernetes Kubelet.

16.2.3批准新Node证书申请

#此操作在Master(vm01)上进行

#查看新的证书请求，状态为Pending
[root@vm01 cfg]# kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           REQUESTEDDURATION   CONDITION
node-csr-6mDDHTg4HuOsVY_7oJRUqtS-6YQFe7JytpYdbRs9kek   31m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued
node-csr-ktjmR4VegWx92ELE3IskISfkdatpXBTKBrq8ZOCVObc   56s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Pending

#批准新的请求，并加入集群
[root@vm01 cfg]# kubectl certificate approve node-csr-ktjmR4VegWx92ELE3IskISfkdatpXBTKBrq8ZOCVObc
certificatesigningrequest.certificates.k8s.io/node-csr-ktjmR4VegWx92ELE3IskISfkdatpXBTKBrq8ZOCVObc approved

#查看证书批准状态
[root@vm01 cfg]# kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           REQUESTEDDURATION   CONDITION
node-csr-6mDDHTg4HuOsVY_7oJRUqtS-6YQFe7JytpYdbRs9kek   31m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued
node-csr-ktjmR4VegWx92ELE3IskISfkdatpXBTKBrq8ZOCVObc   75s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued

#查看集群节点
[root@vm01 cfg]# kubectl get nodes
NAME   STATUS     ROLES    AGE   VERSION
vm01   NotReady   <none>   30m   v1.23.4
vm02   NotReady   <none>   14s   v1.23.4

# 由于网络插件还没有部署，节点会没有准备就绪 NotReady

16.2.4启动kube-proxy

#此操作在vm02上进行

[root@vm02 ~]# systemctl daemon-reload && systemctl start kube-proxy && systemctl enable kube-proxy && systemctl status kube-proxy
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
● kube-proxy.service - Kubernetes Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-04-03 15:57:40 CST; 143ms ago
 Main PID: 12241 (kube-proxy)
   CGroup: /system.slice/kube-proxy.service
           ├─12241 /opt/kubernetes/bin/kube-proxy --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --config=/opt/kubernetes/cfg/kube-proxy-config.yml
           └─12269 modprobe -- ip_vs_wrr

Apr 03 15:57:40 vm02 systemd[1]: Started Kubernetes Proxy.
Apr 03 15:57:40 vm02 kube-proxy[12241]: Flag --logtostderr has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-ins...k8s-components
Apr 03 15:57:40 vm02 kube-proxy[12241]: Flag --log-dir has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrum...k8s-components
Hint: Some lines were ellipsized, use -l to show in full.

16.3新增vm03

16.3.1分发文件

#此操作在Master（vm01）上进行

#分发kubernetes工作目录
scp -r /opt/kubernetes root@192.168.10.13:/opt/

#分发kubelet,kube-proxy的管理文件
scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.10.13:/usr/lib/systemd/system

#分发证书文件
scp /opt/kubernetes/ssl/ca.pem root@192.168.10.13:/opt/kubernetes/ssl

#替换kubelet.conf文件
scp /opt/TLS/k8s/cfg/kubelet03.conf vm03:/opt/kubernetes/cfg/kubelet.conf

#替换kube-proxy-config.yml
scp /opt/TLS/k8s/cfg/kube-proxy-config03.yml vm03:/opt/kubernetes/cfg/kube-proxy-config.yml


#删除kubelet证书和kubeconfig文件
ssh vm03 "rm -f /opt/kubernetes/cfg/kubelet.kubeconfig"
ssh vm03 "rm -f /opt/kubernetes/ssl/kubelet*"

16.3.2核对文件

#此操作在vm02上进行
[root@vm03 ~]# ll /opt/kubernetes
total 12
drwxr-xr-x 2 root root  114 Apr  4 12:21 bin
drwxr-xr-x 2 root root 4096 Apr  4 12:22 cfg
drwxr-xr-x 2 root root 4096 Apr  4 12:21 logs
drwxr-xr-x 2 root root 4096 Apr  4 12:22 ssl

[root@vm03 ~]# ll /usr/lib/systemd/system/{kubelet,kube-proxy}.service
-rw-r--r-- 1 root root 246 Apr  4 12:21 /usr/lib/systemd/system/kubelet.service
-rw-r--r-- 1 root root 253 Apr  4 12:21 /usr/lib/systemd/system/kube-proxy.service

[root@vm03 ~]# ll /opt/kubernetes/ssl/ca.pem
-rw-r--r-- 1 root root 1310 Apr  4 12:22 /opt/kubernetes/ssl/ca.pem

[root@vm03 ~]# ll /opt/kubernetes/cfg/kubelet.conf
-rw-r--r-- 1 root root 382 Apr  4 12:22 /opt/kubernetes/cfg/kubelet.conf

[root@vm03 ~]# cat /opt/kubernetes/cfg/kubelet.conf
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--hostname-override=vm03 \
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=ibmcom/pause-amd64:3.1"

[root@vm03 ~]# ll /opt/kubernetes/cfg/kube-proxy-config.yml
-rw-r--r-- 1 root root 320 Apr  4 12:22 /opt/kubernetes/cfg/kube-proxy-config.yml

[root@vm03 ~]# cat /opt/kubernetes/cfg/kubelet.conf
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--hostname-override=vm03 \
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=ibmcom/pause-amd64:3.1"
[root@vm03 ~]# cat /opt/kubernetes/cfg/kube-proxy-config.yml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: vm03
clusterCIDR: 10.244.0.0/16
mode: ipvs
ipvs:
  scheduler: "rr"
iptables:
  masqueradeAll: true

[root@vm03 ~]# ll /opt/kubernetes/cfg/kubelet.kubeconfig
ls: cannot access /opt/kubernetes/cfg/kubelet.kubeconfig: No such file or directory

[root@vm03 ~]# ll /opt/kubernetes/ssl/kubelet*
ls: cannot access /opt/kubernetes/ssl/kubelet*: No such file or directory

16.3.3启动kubelet

#此操作在vm03上进行
[root@vm03 ~]# systemctl daemon-reload && systemctl start kubelet && systemctl enable kubelet && systemctl status kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-04-04 12:26:34 CST; 100ms ago
 Main PID: 11597 (kubelet)
   CGroup: /system.slice/kubelet.service
           └─11597 /opt/kubernetes/bin/kubelet --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --hostname-override=vm03 --network-plugin=cni --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig ...

Apr 04 12:26:34 vm03 systemd[1]: Started Kubernetes Kubelet.

16.3.4批准新Node证书请求

#此操作在Master(vm01)上进行

#查看新的证书请求，状态为Pending
[root@vm01 cfg]# kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           REQUESTEDDURATION   CONDITION
node-csr-6mDDHTg4HuOsVY_7oJRUqtS-6YQFe7JytpYdbRs9kek   43m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued
node-csr-imMAz2WtkeUWNSfVh_qFxax0V3U6fNcIrXgGWS-VRpk   50s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Pending
node-csr-ktjmR4VegWx92ELE3IskISfkdatpXBTKBrq8ZOCVObc   12m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued

#授权请求
[root@vm01 cfg]# kubectl certificate approve node-csr-imMAz2WtkeUWNSfVh_qFxax0V3U6fNcIrXgGWS-VRpk
certificatesigningrequest.certificates.k8s.io/node-csr-imMAz2WtkeUWNSfVh_qFxax0V3U6fNcIrXgGWS-VRpk approved

#查看证书请求状态
[root@vm01 cfg]# kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           REQUESTEDDURATION   CONDITION
node-csr-6mDDHTg4HuOsVY_7oJRUqtS-6YQFe7JytpYdbRs9kek   43m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued
node-csr-imMAz2WtkeUWNSfVh_qFxax0V3U6fNcIrXgGWS-VRpk   73s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued
node-csr-ktjmR4VegWx92ELE3IskISfkdatpXBTKBrq8ZOCVObc   13m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued

查看集群节点
[root@vm01 cfg]# kubectl get nodes
NAME   STATUS     ROLES    AGE   VERSION
vm01   NotReady   <none>   42m   v1.23.4
vm02   NotReady   <none>   12m   v1.23.4
vm03   NotReady   <none>   9s    v1.23.4

# 由于网络插件还没有部署，节点会没有准备就绪 NotReady

16.3.51启动kube-proxy

#此操作在vm03上进行

[root@vm03 ~]# systemctl daemon-reload && systemctl start kube-proxy && systemctl enable kube-proxy && systemctl status kube-proxy
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
● kube-proxy.service - Kubernetes Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-04-04 12:30:37 CST; 122ms ago
 Main PID: 12152 (kube-proxy)
   CGroup: /system.slice/kube-proxy.service
           └─12152 /opt/kubernetes/bin/kube-proxy --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --config=/opt/kubernetes/cfg/kube-proxy-config.yml

Apr 04 12:30:37 vm03 systemd[1]: Started Kubernetes Proxy.
Apr 04 12:30:37 vm03 kube-proxy[12152]: Flag --logtostderr has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-ins...k8s-components
Apr 04 12:30:37 vm03 kube-proxy[12152]: Flag --log-dir has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrum...k8s-components
Hint: Some lines were ellipsized, use -l to show in full.

17.部署calico网络组件

Calico是一个纯三层的数据中心网络方案，是目前Kubernetes主流的网络方案。

17.1calico网络架构

17.2部署calico

#此操作在master（vm01）上进行
[root@vm01 cfg]# kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created

17.3查看网络组件状态

Every 2.0s: kubectl get pods -n kube-system                                                                                                                                          Sun Apr  3 13:01:58 2022

NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-858c9597c8-m4bvk   1/1     Running   0          14m
calico-node-j92d2                          1/1     Running   0          3m26s
calico-node-mwv5h                          1/1     Running   0          8m20s
calico-node-sb6hg                          1/1     Running   0          14m

当出现上面的信息之后，集群的网络插件已经部署完成。

18.部署coredns组件

18.1创建yaml组件

cd /opt/TLS/k8s/yml
vi  coredns.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
rules:
  - apiGroups:
    - ""
    resources:
    - endpoints
    - services
    - pods
    - namespaces
    verbs:
    - list
    - watch
  - apiGroups:
    - discovery.k8s.io
    resources:
    - endpointslices
    verbs:
    - list
    - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf {
          max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/name: "CoreDNS"
spec:
  # replicas: not specified here:
  # 1. Default is 1.
  # 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: coredns
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      nodeSelector:
        kubernetes.io/os: linux
      affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
           - weight: 100
             podAffinityTerm:
               labelSelector:
                 matchExpressions:
                   - key: k8s-app
                     operator: In
                     values: ["kube-dns"]
               topologyKey: kubernetes.io/hostname
      containers:
      - name: coredns
        image: registry.cn-beijing.aliyuncs.com/dotbalo/coredns:1.8.6 
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
          readOnly: true
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "CoreDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.0.0.10   #注意此处的内容要和PODS网络的地址在一个网段
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
  - name: metrics
    port: 9153
    protocol: TCP

18.2部署coredns组件

[root@vm01 yml]# kubectl apply -f coredns.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created

[root@vm01 yml]# kubectl get pods -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-858c9597c8-m4bvk   1/1     Running   0          22m
kube-system   calico-node-j92d2                          1/1     Running   0          11m
kube-system   calico-node-mwv5h                          1/1     Running   0          16m
kube-system   calico-node-sb6hg                          1/1     Running   0          22m
kube-system   coredns-75c59cb869-znpk8                   1/1     Running   0          16s

19.部署dashboard

19.1创建yaml文件

cd /opt/TLS/k8s/yml
vi dashboard.yaml

# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: v1
kind: Namespace
metadata:
  name: kubernetes-dashboard

---

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-certs
  namespace: kubernetes-dashboard
type: Opaque

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-csrf
  namespace: kubernetes-dashboard
type: Opaque
data:
  csrf: ""

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-key-holder
  namespace: kubernetes-dashboard
type: Opaque

---

kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-settings
  namespace: kubernetes-dashboard

---

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
rules:
  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
    verbs: ["get", "update", "delete"]
    # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["kubernetes-dashboard-settings"]
    verbs: ["get", "update"]
    # Allow Dashboard to get metrics.
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["heapster", "dashboard-metrics-scraper"]
    verbs: ["proxy"]
  - apiGroups: [""]
    resources: ["services/proxy"]
    resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
    verbs: ["get"]

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
rules:
  # Allow Metrics Scraper to get metrics from the Metrics server
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods", "nodes"]
    verbs: ["get", "list", "watch"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: kubernetes-dashboard
          image: kubernetesui/dashboard:v2.5.1
          imagePullPolicy: Always
          ports:
            - containerPort: 8443
              protocol: TCP
          args:
            - --auto-generate-certificates
            - --namespace=kubernetes-dashboard
            # Uncomment the following line to manually specify Kubernetes API server Host
            # If not specified, Dashboard will attempt to auto discover the API server and connect
            # to it. Uncomment only if the default does not work.
            # - --apiserver-host=http://my-address:port
          volumeMounts:
            - name: kubernetes-dashboard-certs
              mountPath: /certs
              # Create on-disk volume to store exec logs
            - mountPath: /tmp
              name: tmp-volume
          livenessProbe:
            httpGet:
              scheme: HTTPS
              path: /
              port: 8443
            initialDelaySeconds: 30
            timeoutSeconds: 30
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 1001
            runAsGroup: 2001
      volumes:
        - name: kubernetes-dashboard-certs
          secret:
            secretName: kubernetes-dashboard-certs
        - name: tmp-volume
          emptyDir: {}
      serviceAccountName: kubernetes-dashboard
      nodeSelector:
        "kubernetes.io/os": linux
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 8000
      targetPort: 8000
  selector:
    k8s-app: dashboard-metrics-scraper

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: dashboard-metrics-scraper
  template:
    metadata:
      labels:
        k8s-app: dashboard-metrics-scraper
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: dashboard-metrics-scraper
          image: kubernetesui/metrics-scraper:v1.0.7
          ports:
            - containerPort: 8000
              protocol: TCP
          livenessProbe:
            httpGet:
              scheme: HTTP
              path: /
              port: 8000
            initialDelaySeconds: 30
            timeoutSeconds: 30
          volumeMounts:
          - mountPath: /tmp
            name: tmp-volume
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 1001
            runAsGroup: 2001
      serviceAccountName: kubernetes-dashboard
      nodeSelector:
        "kubernetes.io/os": linux
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      volumes:
        - name: tmp-volume
          emptyDir: {}

19.2创建dashboard组件

[root@vm01 yml]# kubectl apply -f dashboard.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created

19.3查看组件状态

[root@vm01 yml]# kubectl get pods,svc -n kubernetes-dashboard
NAME                                             READY   STATUS    RESTARTS   AGE
pod/dashboard-metrics-scraper-5b8896d7fc-62t5g   1/1     Running   0          61s
pod/kubernetes-dashboard-7b5d774449-np99c        1/1     Running   0          61s

NAME                                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/dashboard-metrics-scraper   ClusterIP   10.0.0.206   <none>        8000/TCP   61s
service/kubernetes-dashboard        ClusterIP   10.0.0.128   <none>        443/TCP    62s

通过状态来看，组件已成功创建，但是还不能从外部进行访问，为了能一见dashboard的芳容，我们需要改造一下svc的类型。

19.4修改svc类型

[root@vm01 yml]# kubectl patch svc kubernetes-dashboard -p '{"spec":{"type":"NodePort"}}' -n kubernetes-dashboard
service/kubernetes-dashboard patched
[root@vm01 yml]# kubectl get pods,svc -n kubernetes-dashboard
NAME                                             READY   STATUS    RESTARTS   AGE
pod/dashboard-metrics-scraper-5b8896d7fc-62t5g   1/1     Running   0          4m23s
pod/kubernetes-dashboard-7b5d774449-np99c        1/1     Running   0          4m23s

NAME                                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
service/dashboard-metrics-scraper   ClusterIP   10.0.0.206   <none>        8000/TCP        4m23s
service/kubernetes-dashboard        NodePort    10.0.0.128   <none>        443:31054/TCP   4m24s

#此时svc中已经出现了对外可访问的端口31054

19.5访问web页面

在浏览器中访问https://192.168.10.11:31054

出现了以上界面，点击“继续访问 192.168.10.11(不安全)”即可。

这里需要我们输入token的值，怎么找呢？请按照下面的步骤进行操作即可。

19.6生成token

#创建service account
[root@vm01 yml]# kubectl create serviceaccount dashboard-admin -n kube-system
serviceaccount/dashboard-admin created

#绑定默认cluster-admin管理员集群角色
[root@vm01 yml]# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin created

#查看token值，最长的那一串字符就是token值了
[root@vm01 yml]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
Name:         dashboard-admin-token-jldjt
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: dashboard-admin
              kubernetes.io/service-account.uid: 65f21379-a38b-4e4a-b8a0-2bf8bc056faa

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1310 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6InJsRnVoZ1VtSWdMQ1U3VmxJMzRFVVI3T1VrdDU4REhiSVFQUl9naDUzdEEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tamxkanQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNjVmMjEzNzktYTM4Yi00ZTRhLWI4YTAtMmJmOGJjMDU2ZmFhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.j7nKBfiDUFuVTDhy9Nyjw3kp0w_CKvh9ec94j7VLZz6v5RupdlIZIQqqtyhcFPLj7ADIwqXcWG3kpuFT_u5-cg5j95D88-7bt0rAtqwtn0FeBpWhT8dX_WZm7efSnw2c3xciFMYfTo9Iffx9GF7O9UKyOoh-Sg4MDeQLD2f-1jN3hqz-zuebQcnOlpeS-ateaRQvcb9Lhac5quST8G10IOFXz0itFpuypbXdOxCbRmqxIiHR_7PGOq_0_NGOPRsn5n4d68-cK34dM-HNQUZxSGPpxo39wvnWmyZNnYx3jW_KVXl5Dt-w4xsBhxcXwlvPvGuwZlbiR0PZEXYgOGq2hw

19.7登录web

输入上面生成的token值后，就可以进入到dashboard界面了

此时，用户最亲切的dashboard界面也一览无余了。

20.部署MetricsServer

从 v1.8 开始，资源使用情况的监控可以通过 Metrics API的形式获取，具体的组件为Metrics Server，用来替换之前的heapster，heapster从1.11开始逐渐被废弃。

Metrics-Server是集群核心监控数据的聚合器，从 Kubernetes1.8 开始，它作为一个 Deployment对象默认部署在由kube-up.sh脚本创建的集群中，如果是其他部署方式需要单独安装，或者咨询对应的云厂商。

20.1创建yaml文件

cd /opt/TLS/k8s/yml

vi metrics-server.yml


apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: system:aggregated-metrics-reader
rules:
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        image: bitnami/metrics-server:0.4.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          periodSeconds: 10
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
      - emptyDir: {}
        name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100

20.2部署MetricsServer组件

[root@vm01 yml]# kubectl apply -f metrics-server.yml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

20.3查看组件状态

[root@vm01 yml]# kubectl get pods -n kube-system |grep metrics-server
metrics-server-68cf7d657c-9rfg4            1/1     Running   0          93s

20.4查看资源使用情况

#经过上述的操作之后，我们就可以按一定的排序规则来查看k8s集群的资源使用情况了
[root@vm01 yml]# kubectl top nodes
NAME   CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
vm01   113m         11%    1223Mi          66%
vm02   71m          7%     720Mi           38%
vm03   83m          8%     816Mi           44%

[root@vm01 yml]# kubectl top pods -n kube-system
NAME                                       CPU(cores)   MEMORY(bytes)
calico-kube-controllers-858c9597c8-m4bvk   2m           27Mi
calico-node-j92d2                          15m          160Mi
calico-node-mwv5h                          20m          162Mi
calico-node-sb6hg                          19m          177Mi
coredns-75c59cb869-znpk8                   1m           18Mi
metrics-server-68cf7d657c-9rfg4            2m           15Mi

再来看dashboard界面，多了一些资源使用情况的可视化展示，对于分析问题来讲，是个不错的手段。

21.安装kuboard

给搭建推荐一款非常不错的k8s管理工具，个人还是非常喜欢的。

21.1部署kuboard

[root@vm01 yml]# kubectl apply -f https://addons.kuboard.cn/kuboard/kuboard-v3.yaml
namespace/kuboard created
configmap/kuboard-v3-config created
serviceaccount/kuboard-boostrap created
clusterrolebinding.rbac.authorization.k8s.io/kuboard-boostrap-crb created
daemonset.apps/kuboard-etcd created
deployment.apps/kuboard-v3 created
service/kuboard-v3 created

21.2查看组件状态

执行指令 watch kubectl get pods -n kuboard，等待 kuboard 名称空间中所有的 Pod 就绪，如下所示，

Every 2.0s: kubectl get pods -n kuboard                                                                                                                                              Sun Apr  3 14:15:07 2022

NAME                               READY   STATUS    RESTARTS   AGE
kuboard-etcd-jltwh                 1/1     Running   0          3m29s
kuboard-etcd-rsmd9                 1/1     Running   0          3m3s
kuboard-etcd-wdtgl                 1/1     Running   0          3m2s
kuboard-questdb-8497b87d9f-82466   1/1     Running   0          2m12s
kuboard-v3-59ccddb94c-5g5v6        1/1     Running   1          4m49s

如果结果中没有出现 kuboard-etcd-xxxxx 的容器，请查看中关于缺少 Master Role 的描述。

解决办法如下：

kubectl label nodes vm01 k8s.kuboard.cn/role=etcd
kubectl label nodes vm02 k8s.kuboard.cn/role=etcd
kubectl label nodes vm03 k8s.kuboard.cn/role=etcd

# 参考：https://www.kuboard.cn/install/v3/install-in-k8s.html#%E5%AE%89%E8%A3%85

21.3访问 Kuboard

手里没有王

关注

8
点赞
踩
27

收藏

觉得还不错? 一键收藏
6
评论
二进制安装kubernetes(v1.20.16)

1.集群规划序号 IP 角色 Hostname 安装组件 1 192.168.10.11 Master,Node vm01 Apiserver,ControllerManager,Scheduler,Kubelet,Proxy,Etcd 2 192.168.10.12 Node vm02 Kubelet,Proxy,Etcd 3 192.168.10.13 Node vm03 Kubelet,Pr
复制链接

扫一扫

专栏目录