二进制部署k8s 1.21版本

IP地址	角色	centos版本	cpu个数	内存	sda	安装的软件
10.10.10.11	master1	Centos7.9	2	260G	200G	kube-apiserver、kube-controller-manager、kube-scheduler、etcd
10.10.10.12	master2 node1	Centos7.9	2	260G	5T	kube-apiserver、kube-controller-manager、kube-scheduler、etcd kubelet、kube-proxy、Containerd、runc
10.10.10.13	master3 node2	Centos7.9	2	260G	5T	kube-apiserver、kube-controller-manager、kube-scheduler、etcd kubelet、kube-proxy、Containerd、runc
10.10.10.14	ha					keepalived

VIP ：10.10.10.24/24

node网络：10.10.10.0/24

service网络（默认）：10.96.0.0/12

pod网络（默认）：10.244.0.0/16

三台主机均为master，作主从高可用架构。10.10.10.11为主master，由于该机器可用空间较少，不作为node节点使用。

防止以后添加机器，配置文件内预留了一些ip地址，分别为：10.10.10.204，10.10.10.104，10.10.10.230，10.10.10.231，10.10.10.232，10.10.10.233

二.准备工作+etcd集群部署

1. 主机准备

（1）对应主机名设置

sudo hostnamectl set-hostname  master
sudo hostnamectl set-hostname  node1
sudo hostnamectl set-hostname  node2

（2）IP地址解析

sudo vim /etc/hosts

10.10.10.11 master
10.10.10.12 node1
10.10.10.13 node2

（3）关闭防火墙 seliux 交换分区时间同步安装常用软件

#####关闭防火墙#####
#关闭
sudo systemctl stop firewalld
#开机自启关闭
sudo systmctl disable firewalld
#查看状态
sudo firewall-cmd --state

#####关闭selinux#####
#临时关闭
sudo setenforce 0
#始终关闭
sudo sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
#查看状态(如状态依旧显示运行，重启服务器后，即可显示关闭)
sudo sestatus

#####交换分区设置#####
sudo swapoff -a
sudo sed -ri 's/.*swap.*/#&/' /etc/fstab
sudo vim /etc/sysctl.conf
    vm.swappiness=0   添加到最后一行
sudo sysctl -p
#验证是否关闭，swap行应显示为0
free -m


#####主机系统时间同步#####
#安装软件
sudo yum -y install ntpdate
#制定时间同步计划任务
sudo crontab -e
0 */1 * * * ntpdate time1.aliyun.com
#查看定时任务
sudo crontab -l
#查看所有机器时间是否一致
date

#####安装常用软件#####
sudo yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git lrzsz -y

（4）主机系统优化——limit优化

提高limit限制，防止报错显示连接数过多导致无法访问

#####临时调整#####
ulimit -SHn 65535
#####永久调节#####
sudo su
cat <<EOF >> /etc/security/limits.conf
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF
#也可以直接vim添加进去。

（5）ipvs管理工具的安装及模块加载

k8s集群在网络底层调度时使用传统模式会比较慢，因此新的版本建议修改为ipvs的底层模式。ipvs属于内核空间模块。使用时要在用户空间部署一个ipvsadm的工具进行控制。

#####安装#####
sudo yum -y install ipvsadm ipset sysstat conntrack libseccomp

#####模块加载#####
sudo su
cat >/etc/modules-load.d/ipvs.conf <<EOF 
ip_vs 
ip_vs_lc 
ip_vs_wlc 
ip_vs_rr 
ip_vs_wrr 
ip_vs_lblc 
ip_vs_lblcr 
ip_vs_dh 
ip_vs_sh 
ip_vs_fo 
ip_vs_nq 
ip_vs_sed 
ip_vs_ftp 
ip_vs_sh 
nf_conntrack 
ip_tables 
ip_set 
xt_set 
ipt_set 
ipt_rpfilter 
ipt_REJECT 
ipip 
EOF

（6）加载containerd相关内核模块

#####临时加载#####
sudo modprobe overlay
sudo modprobe br_netfilter

#####永久加载######
sudo su
cat > /etc/modules-load.d/containerd.conf << EOF
overlay
br_netfilter
EOF

#####设置为开机自启######
sudo systemctl enable --now systemd-modules-load.service
注意：此处报错！！！！！！！怀疑是内核版本导致的，暂时放下，去下一步

#####查看状态#####
sudo systemctl status systemd-modules-load.service

后记：执行完 7升级内核 8内核优化，重启后执行此命令，可以成功，确实是内核版本问题导致的，下次可以先升级优化内核再执行 6加载containerd相关内核模块

（7）升级内核

有一篇专门写内核升级的，感兴趣的可以去查看，那篇更详细一些。

Centos7.2 系统升级内核_centos7.2升级内核_珂玥c的博客-CSDN博客

#####升级安装elrepo
sudo yum -y install perl
sudo rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
sudo yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
#####安装新版本内核#####
sudo yum  --enablerepo="elrepo-kernel"  -y install kernel-ml.x86_64
#####设置内核默认启动顺序#####
sudo grub2-set-default 0
#####生成grub配置文件#####
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

（8）内核优化

网桥转发和tcp的一些优化

sudo su
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 131072
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF

sysctl --system

#####重启服务器#####
reboot -h now

#####重启后查看ipvs模块加载情况#####
lsmod | grep --color=auto -e ip_vs -e nf_conntrack

#####重启后查看containerd相关模块加载情况#####
lsmod | grep br_netfilter
lsmod | grep overlay

2. keepalived准备

注：由于haproxy的端口与后面服务的端口重复，如果有单独一台机器放置当然是最好。两台node机器使用haproxy的意义不大，因此只做了记录，实际中并没有运行这个服务。大家可根据自身需求进行选择。翻车记录：最初为了节省资源将keepalived和haproxy启动在master1和master2上，apiserver和haproxy两个服务导致6443端口冲突，curl的时候总是出现curl: (7) Failed connect to 10.10.18.254:6443; Connection refused。且搭建的时候已经是设置的vip：6443端口。没办法，最后重新找了一台机器，启动了keepalived。实在是资源有限，无法再启动一台做haproxy+keepalived，后期看看有没有可能加上叭。因此，只能二选一，要么资源充足单独用服务器。要么资源不充足，haproxy修改成其他的端口，后续部署k8s的时候使用vip的其他端口，避免冲突！

（1）安装haproxy与keepalived

sudo yum -y install haproxy keepalived

（2）haproxy配置

sudo mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak

sudo vim /etc/haproxy/haproxy.cfg
#####写入以下内容#####
global
 maxconn 2000
 ulimit-n 16384
 log 127.0.0.1 local0 err
 stats timeout 30s

defaults
 log global
 mode http
 option httplog
 timeout connect 5000
 timeout client 50000
 timeout server 50000
 timeout http-request 15s
 timeout http-keep-alive 15s

frontend monitor-in
 bind *:33305
 mode http
 option httplog
 monitor-uri /monitor

frontend k8s-master
 bind 0.0.0.0:6443
 bind 127.0.0.1:6443
#####注意端口号，默认是6443，因为和k8s的apiserver所用到的端口会冲突，如果启动在k8s的master或者node上，一定要换成其他端口（后续部署也要记得修改，不能参照我下面文档中均使用的6443端口），如果是启动在新的机器上，无需修改#####
 mode tcp
 option tcplog
 tcp-request inspect-delay 5s
 default_backend k8s-master

backend k8s-master
 mode tcp
 option tcplog
 option tcp-check
 balance roundrobin
 default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
 server  master  10.10.10.11:6443 check
 server  node1   10.10.10.12:6443 check
 server  node2   10.10.10.13:6443 check

（3）keepalived主从配置文件

主从配置文件的内容是不一致的！！！

ha 10.10.10.14

sudo mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak

sudo vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
script_user root
   enable_script_security
}
vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
   interval 5
   weight -5
   fall 2 
rise 1
}
vrrp_instance VI_1 {
   state MASTER
#####ifconfig先查询一下网卡，写自己机器对应的名字，ens33 eth0 br1等等######
   interface br1
   mcast_src_ip 10.10.10.14
   virtual_router_id 51
#####主的权重数字大于从#####
   priority 100
   advert_int 2
   authentication {
       auth_type PASS
       auth_pass K8SHA_KA_AUTH
   }
   virtual_ipaddress {
       10.10.10.24/24
   }
   track_script {
      chk_apiserver
   }
}

ha2

sudo mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak

sudo vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
script_user root
   enable_script_security
}
vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
  interval 5
   weight -5
   fall 2 
rise 1
}
vrrp_instance VI_1 {
#####从这里必须是BACKUP的状态#####
   state BACKUP
#####ifconfig先查询一下网卡，必须是自己机器对应网卡的名字！！######
   interface 网卡名
   mcast_src_ip xx.xx.xx.xx
   virtual_router_id 51
   priority 99
   advert_int 2
   authentication {
       auth_type PASS
       auth_pass K8SHA_KA_AUTH
   }
   virtual_ipaddress {
       10.10.10.24/24
   }
   track_script {
      chk_apiserver
   }
}

（4）健康检查脚本

这个脚本是检查haproxy是否出现问题，并且对应的去调整keepalived服务的状态。由于我个人没有开启haproxy服务，所以此脚本也未运行，否则的话始终会给我关闭keepalived服务。

sudo vim /etc/keepalived/check_apiserver.sh 

#!/bin/bash
err=0
for k in $(seq 1 3)
do
   check_code=$(pgrep haproxy)
   if [[ $check_code == "" ]]; then
       err=$(expr $err + 1)
       sleep 1
       continue
   else
       err=0
       break
   fi
done

if [[ $err != "0" ]]; then
   echo "systemctl stop keepalived"
   /usr/bin/systemctl stop keepalived
   exit 1
else
   exit 0
fi

#####添加执行权限#####
sudo chmod +x /etc/keepalived/check_apiserver.sh

（5）启动服务&验证

sudo systemctl daemon-reload
sudo systemctl start keepalived
##### sudo systemctl enable --now haproxy
sudo systemctl enable --now keepalived

#####ha1上验证一下#####
ip addr   

#####ha1上停止keepalived后可以在ha2上看到vip出现了，实现了漂移#####
sudo systemctl stop keepalived
ip addr

3. 免密登录

只在master1 10.10.10.11上配置就可以了，后续用master分发内容。
也可以不执行此操作，就是后续传文件时麻烦一些，每次都输密码。
简单的方式生成秘钥对，直接将id_rsa.pub里面的东西复制到另外两台的authorized_keys中。

sudo su
ssh-keygen
cat ~/.ssh/id_rsa.pub

4. cfssl工具 & CA_ETCD证书准备

在master1 10.10.10.11 节点上操作

（1）创建工作目录

这里所有机器都准备

sudo mkdir -p /data/k8s-work
sudo su
cd /data/k8s-work
#####如果cd不过去，chmod -R 755 /data 改一下权限即可#####

（2）获取cfssl工具

做生成证书使用，需要三个工具，服务器无法下载的话，可以别处下载后,服务器使用scp拉上来，三台机器都操作。

#####实现证书读取，获取#####
sudo wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

#####cfssljson用来从cfssl程序获取JSON输出，并将证书，密钥，CSR和bundle写入文件中#####
sudo wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

#####证书相关信息查看工具#####
sudo wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64


chmod +x cfssl*

mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

#####验证是否安装成功#####
cfssl version

注：不显示可能为环境变量问题,可进行以下操作！！！

sudo vim /etc/profile
添加到末尾
PATH=$PATH:/usr/local/bin
export PATH

sudo source /etc/profile

（3）创建CA证书

这里，可以一台机器做，然后同步一下

第一步：配置CA证书请求文件

cat > ca-csr.json <<"EOF"
{
  "CN": "kubernetes",
  "key": {
      "algo": "rsa",
      "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "kubemsb",
      "OU": "CN"
    }
  ],
  "ca": {
          "expiry": "87600h"
  }
}
EOF

第二步：创建CA证书
（会生成证书请求文件，证书对应的key以及证书本身）

sudo su
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

第三步：配置CA策略

sudo su
cat > ca-config.json <<"EOF"
{
  "signing": {
      "default": {
          "expiry": "87600h"
        },
      "profiles": {
          "kubernetes": {
              "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
              ],
              "expiry": "87600h"
          }
      }
  }
}
EOF
#####设置了使用期限#####
server auth 表示client可以对使用该ca对server提供的证书进行验证
client auth 表示server可以使用该ca对client提供的证书进行验证

（4）创建etcd证书

第一步：配置etcd请求文件

此处并没有预留IP，etcd集群相对独立，日后也不准备再添加etcd的机器了。

cat > etcd-csr.json <<"EOF"
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "10.10.10.11",
    "10.10.10.12",
    "10.10.10.13"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "CN",
    "ST": "Beijing",
    "L": "Beijing",
    "O": "kubemsb",
    "OU": "CN"
  }]
}
EOF

第二步：生成etcd证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson  -bare etcd

#####基于etcd-csr.json来生成证书，本条命令参数比较多#####

5. 部署etcd集群

（1）下载etcd软件包

github.com上面搜etcd，下载需要的版本

点进去后下滑到最低部，找到需要的包，右键复制链接

wget https://github.com/etcd-io/etcd/releases/download/v3.5.2/etcd-v3.5.2-linux-amd64.tar.gz

（2）安装etcd

#####解压包#####
tar -xvf etcd-v3.5.2-linux-amd64.tar.gz
#####可执行文件放入对应位置#####
cp -p etcd-v3.5.2-linux-amd64/etcd* /usr/local/bin/
#####查看######
etcdctl version

（3）分发etcd软件

scp etcd-v3.5.2-linux-amd64/etcd* node1:/usr/local/bin/
scp etcd-v3.5.2-linux-amd64/etcd* node2:/usr/local/bin/

#####另外两台机器也验证一下######
etcdctl version

（4）创建配置文件

此步骤三台都进行，且配置文件不一致，ETCD_LISTEN_PEER_URLS，ETCD_LISTEN_CLIENT_URLS，ETCD_INITIAL_ADVERTISE_PEER_URLS，ETCD_ADVERTISE_CLIENT_URLS，ETCD_INITIAL_CLUSTER="etcd1,etcd2,etcd3"，更换成对应的IP即可

10.10.10.11

mkdir /etc/etcd

cat >  /etc/etcd/etcd.conf <<"EOF"
#[Member]
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.10.10.11:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.10.10.11:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.10.11:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.10.10.11:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://10.10.10.11:2380,etcd2=https://10.10.10.12:2380,etcd3=https://10.10.10.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

10.10.10.12

mkdir /etc/etcd

cat >  /etc/etcd/etcd.conf <<"EOF"
#[Member]
ETCD_NAME="etcd2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.10.10.12:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.10.10.12:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.10.12:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.10.10.12:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://10.10.10.11:2380,etcd2=https://10.10.10.12:2380,etcd3=https://10.10.10.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
                                      
EOF

10.10.10.13

mkdir /etc/etcd

cat >  /etc/etcd/etcd.conf <<"EOF"
#[Member]
ETCD_NAME="etcd3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.10.10.13:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.10.10.13:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.10.13:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.10.10.13:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://10.10.10.11:2380,etcd2=https://10.10.10.12:2380,etcd3=https://10.10.10.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
                                        
EOF

说明：

ETCD_NAME：节点名称，集群中唯一

ETCD_DATA_DIR：数据目录

ETCD_LISTEN_PEER_URLS：集群通信监听地址

ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址

ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址

ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址

ETCD_INITIAL_CLUSTER：集群节点地址

ETCD_INITIAL_CLUSTER_TOKEN：集群Token

ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

（5）创建服务配置文件

三台机器都创建目录

mkdir -p /etc/etcd/ssl
mkdir -p /var/lib/etcd/default.etcd

10.10.10.11

cd /data/k8s-work
cp ca*.pem /etc/etcd/ssl
cp etcd*.pem /etc/etcd/ssl

scp ca*.pem node1:/etc/etcd/ssl
scp etcd*.pem node1:/etc/etcd/ssl

scp ca*.pem node2:/etc/etcd/ssl
scp etcd*.pem node2:/etc/etcd/ssl

三台机器都进行操作

cat > /etc/systemd/system/etcd.service <<"EOF"
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/etc/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

说明：

EnvironmentFile 配置文件

WorkingDirectory 工作目录

ExecStart 启动文件（which etcd 显示的路径）

cert-file etcd相关证书

key-file etcd相关证书所对应的key

trusted-ca-file 信任的ca文件

peer-cert-file 伙伴间证书文件（也用的etcd的文件）

peer-key-file 伙伴间相关证书对应的key

peer-trusted-ca-file 伙伴间信任的ca文件

peer-client-cert-auth 伙伴间客户端的认证

client-cert-auth 客户断的认证

（6）启动etcd集群

三台都启动

systemctl daemon-reload
systemctl enable --now etcd.service
systemctl status etcd

（7）验证集群状态

以下的验证随便在哪台机器上都可以，它们是集群，都可以查到的

查看端点是否健康，能不能连上，能不能读取数据

ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://10.10.10.11:2379,https://10.10.10.12:2379,https://10.10.10.13:2379 endpoint health

检测etcd的性能

ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://10.10.10.11:2379,https://10.10.10.12:2379,https://10.10.10.13:2379 check perf

查看成员列表

ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://10.10.10.11:2379,https://10.10.10.12:2379,https://10.10.10.13:2379 member list

可以看到leader是10.10.10.11

ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://10.10.10.11:2379,https://10.10.10.12:2379,https://10.10.10.13:2379 endpoint status

三. master节点部署

1. 集群软件部署

#####软件包下载#####
cd /data/k8s-work/
wget https://dl.k8s.io/v1.21.10/kubernetes-server-linux-amd64.tar.gz
#####软件包安装#####
tar -xvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin/
cp kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/
(以上均为控制平面所需要的软件)
#####分发给其他机器#####
scp kube-apiserver kube-controller-manager kube-scheduler kubectl node1:/usr/local/bin/
scp kube-apiserver kube-controller-manager kube-scheduler kubectl node2:/usr/local/bin/
#####工作负载的软件（工作节点中使用就好，master可以有也可以没有，介于只有三台机器，那就都有吧）#####
cp kubelet kube-proxy /usr/local/bin
scp kubelet kube-proxy node1:/usr/local/bin
scp kubelet kube-proxy node2:/usr/local/bin
####所有节点创建一下工作目录#####
mkdir -p /etc/kubernetes/        
mkdir -p /etc/kubernetes/ssl     
mkdir -p /var/log/kubernetes

2. master节点部署——apiserver

（1）创建apiserver证书请求文件

cd /data/k8s-work
cat > kube-apiserver-csr.json << "EOF"
{
"CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "10.10.10.11",
    "10.10.10.12",
    "10.10.10.13",
    "10.10.10.204",
    "10.10.10.104",
    "10.10.10.230",
    "10.10.10.231",
    "10.10.10.232",
    "10.10.10.233",
    "10.10.10.254",
    "10.96.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "kubemsb",
      "OU": "CN"
    }
  ]
}
EOF

说明：

如果 hosts 字段不为空则需要指定授权使用该证书的 IP（含VIP）或域名列表。由于该证书被集群使用，需要将节点的IP都填上，为了方便后期扩容可以多写几个预留的IP。同时还需要填写 service 网络的首个IP(一般是 kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP，如 10.96.0.1)

（2）生成apiserver证书和token文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver

注意：此处报错显示没有cfssl的话，执行一下 source /etc/profile ，之前写在这里环境变量了。

ca ca对应的key ca对应的策略文件 profile前缀 .json | 会写到三个文件里去，.pem证书 key证书的私钥 csr请求文件

说明：
apiserver启动了TLS认证后，node节点的kubelet和kube-proxy与apiserver进行通信，必须使用CA签发的有效证书才行。当node节点很多时，颁发证书需要大量工作。为了简化流程引入了TLS的bootstraping机器自动颁发证书。kubelet会以一个低权限用户自动向apiserver申请证书，由apiserver自动签署。强烈建议在node使用这个种方式，目前主要用于kubelet，kube-proxy还是由我们统一颁发一个证书。

10.10.10.11上执行

cd /data/k8s-work

cat > token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

（3）创建服务配置文件

10.10.10.11上执行

cat > /etc/kubernetes/kube-apiserver.conf << "EOF"
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --bind-address=10.10.10.11 \
  --secure-port=6443 \
  --advertise-address=10.10.10.11 \
  --insecure-port=0 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.96.0.0/16 \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-node-port-range=30000-32767 \
  --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
  --service-account-issuer=api \
  --etcd-cafile=/etc/etcd/ssl/ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://10.10.10.11:2379,https://10.10.10.12:2379,https://10.10.10.13:2379 \
  --enable-swagger-ui=true \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/kube-apiserver-audit.log \
  --event-ttl=1h \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"
EOF

（4）apiserver服务的管理配置文件

如果想用systemd管理apiserver，那么就需要这个文件，里面主要是写了让服务的启动顺序放在etcd后面（After=etcd.service Wants=etcd.service）EnvironmentFile apiserver的配置文件
ExecStart apiserver的二进制文件
在10.10.10.11上执行

cat > /etc/systemd/system/kube-apiserver.service << "EOF"
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=etcd.service
Wants=etcd.service

[Service]
EnvironmentFile=-/etc/kubernetes/kube-apiserver.conf
ExecStart=/usr/local/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

（5）同步文件到其他master节点上

cd /data/k8s-work
cp ca*.pem /etc/kubernetes/ssl/
cp kube-apiserver*.pem /etc/kubernetes/ssl/
cp token.csv /etc/kubernetes/

scp /etc/kubernetes/token.csv node1:/etc/kubernetes
scp /etc/kubernetes/token.csv node2:/etc/kubernetes

scp /etc/kubernetes/ssl/kube-apiserver*.pem node1:/etc/kubernetes/ssl
scp /etc/kubernetes/ssl/kube-apiserver*.pem node2:/etc/kubernetes/ssl

scp /etc/kubernetes/ssl/ca*.pem node1:/etc/kubernetes/ssl
scp /etc/kubernetes/ssl/ca*.pem node2:/etc/kubernetes/ssl

配置文件分发

scp /etc/kubernetes/kube-apiserver.conf node1:/etc/kubernetes/kube-apiserver.conf
scp /etc/kubernetes/kube-apiserver.conf node2:/etc/kubernetes/kube-apiserver.conf

（6）启动apiserver

systemctl daemon-reload
systemctl enable --now kube-apiserver

systemctl status kube-apiserver

#####测试#####
curl --insecure https://10.10.10.11:6443/
curl --insecure https://10.10.10.12:6443/
curl --insecure https://10.10.10.13:6443/
curl --insecure https://10.10.10.24:6443/

注意：

这里遇到了报错：# curl --insecure https://10.10.18.254:6443/

curl: (7) Failed connect to 10.10.18.254:6443; Connection refused

解决方法：

#####所有机器上执行#####
ipvsadm -At 10.10.10.24:6443 -s rr
ipvsadm -at 10.10.10.24:6443 -r 10.10.10.24 -m
ipvsadm -at 10.10.10.24 -r 10.10.10.12 -m
ipvsadm -ln

3. master节点部署——kubectl

kubectl应该属于客户端工具，不属于组件

（1）创建证书请求文件

cd /data/k8s-work
cat > admin-csr.json << "EOF"
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:masters",             
      "OU": "system"
    }
  ]
}
EOF

说明：

后续 kube-apiserver 使用 RBAC 对客户端(如 kubelet、kube-proxy、Pod)请求进行授权；

kube-apiserver 预定义了一些 RBAC 使用的 RoleBindings，如 cluster-admin 将 Group system:masters 与 Role cluster-admin 绑定，该 Role 授予了调用kube-apiserver 的所有 API的权限；

O指定该证书的 Group 为 system:masters，kubelet 使用该证书访问 kube-apiserver 时，由于证书被 CA 签名，所以认证通过，同时由于证书用户组为经过预授权的 system:masters，所以被授予访问所有 API 的权限；

注：

这个admin 证书，是将来生成管理员用的kubeconfig 配置文件用的，现在我们一般建议使用RBAC 来对kubernetes 进行角色权限控制， kubernetes 将证书中的CN 字段作为User， O 字段作为 Group；

"O": "system:masters", 必须是system:masters，否则后面kubectl create clusterrolebinding报错。

（2）生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

（3）复制文件到指定目录

cp admin*.pem /etc/kubernetes/ssl/

（4）生成kubeconfig文件

kube.config 为 kubectl 的配置文件，包含访问 apiserver 的所有信息，如 apiserver 地址、CA 证书和自身使用的证书。最后会生成一个kube.config文件。

#####确定管理的集群，所对应的证书和证书的访问链接#####
kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://10.10.18.254:6443 --kubeconfig=kube.config
#找不到命令的时候，source一下/etc/profile文件即可

#####配置证书角色为admin.pem#####
kubectl config set-credentials admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=kube.config

#####设置安全上下文#####
kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=kube.config

#####使用安全上下文对kubernetes进行管理#####
kubectl config use-context kubernetes --kubeconfig=kube.config


cat kube.config 
apiVersion: v1
clusters:
- cluster: #####证书#####
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVQlE3bmZsZkVtdkVFQkEwZF等等
    server: https://10.10.18.254:6443   #####访问方法#####
  name: kubernetes #####集群名字#####
contexts: #####安全上下文#####
- context:
    cluster: kubernetes
    user: admin
  name: kubernetes
current-context: kubernetes  #####当前的上下文是谁#####
kind: Config
preferences: {}
users:
- name: admin #####用户是谁#####
  user:  #####用户使用的所属客户端是什么#####
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQzVENDQXNXZ0F3SUJBZ0等等
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBczNVOFE2M3d等等

（5）准备kubectl配置文件和角色绑定

为当前用户准备kubectl的配置文件，比如root用户

sudo su
mkdir ~/.kube
cp kube.config ~/.kube/config
kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes --kubeconfig=/root/.kube/config

（6）查看集群状态

自定义一个名为KUBECONFIG的环境变量

export KUBECONFIG=$HOME/.kube/config

#####查看集群信息#####
kubectl cluster-info

#####查看集群组件状态（controller-manager，scheduler还没有部署，所以是不健康状态，etcd是正常的）#####
kubectl get componentstatuses

#####查看命名空间中资源对象#####
kubectl get all --all-namespaces

（7）同步kubectl到其他master节点

10.10.10.12:
mkdir /root/.kube

10.10.10.13:
mkdir /root/.kube

10.10.10.11:
scp /root/.kube/config node1:/root/.kube/config
scp /root/.kube/config node2:/root/.kube/config

######去另外两台进行验证#####
kubectl cluster-info

（8）配置kubectl命令补全

yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
kubectl completion bash > ~/.kube/completion.bash.inc
source '/root/.kube/completion.bash.inc'  
source $HOME/.bash_profile

4. master节点部署——kube-controller-manager

kube-controller-manager在集群中的作用是对controller进行管理，一旦controller有问题时，controller-manager会介入。

（1）创建kube-controller-manager的证书请求文件

cd /data/k8s-work

cat > kube-controller-manager-csr.json << "EOF"
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "10.10.10.11",
      "10.10.10.12",
      "10.10.10.13",
      "10.10.10.204",
      "10.10.10.104",
      "10.10.10.230",
      "10.10.10.231",
      "10.10.10.232",
      "10.10.10.233",
      "10.10.10.24"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "Beijing",
        "L": "Beijing",
        "O": "system:kube-controller-manager",
        "OU": "system"
      }
    ]
}
EOF

说明：

hosts 列表包含所有 kube-controller-manager 节点 IP。CN 为 system:kube-controller-manager。O 为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限

（2）创建kube-controller-manager的证书文件

会出现三个文件：证书请求文件，证书，证书所对应的key

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

（3）创建kube-controller-manager的kubeconfig文件

#####设置集群#####
kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://10.10.10.24:6443 --kubeconfig=kube-controller-manager.kubeconfig

#####设置集群相关访问证书#####
kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig

#####设置集群访问的安全上下文#####
kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

#####使用已经设置的安全上下文#####
kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

#####生成了kube-controller-manager.kubeconfig文件#####

（4）创建kube-controller-manager的配置文件

此时是不成功的

kubectl get componentstatuses

去执行：

cd /data/k8s-work

cat > kube-controller-manager.conf << "EOF"
KUBE_CONTROLLER_MANAGER_OPTS="--port=10252 \
  --secure-port=10257 \
  --bind-address=127.0.0.1 \
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --service-cluster-ip-range=10.96.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --allocate-node-cidrs=true \
  --cluster-cidr=10.244.0.0/16 \
  --experimental-cluster-signing-duration=87600h \
  --root-ca-file=/etc/kubernetes/ssl/ca.pem \
  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --leader-elect=true \
  --feature-gates=RotateKubeletServerCertificate=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --horizontal-pod-autoscaler-use-rest-clients=true \
  --horizontal-pod-autoscaler-sync-period=10s \
  --tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \
  --use-service-account-credentials=true \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2"
EOF

（5）创建服务启动文件（进行systemd管理）

#####systemd对它进行管理#####
cat > kube-controller-manager.service << "EOF"
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/kube-controller-manager.conf
ExecStart=/usr/local/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

（6）多台主机实现kube-controller-manager高可用，同步文件到其他master节点

cp kube-controller-manager*.pem /etc/kubernetes/ssl/
cp kube-controller-manager.kubeconfig /etc/kubernetes/
cp kube-controller-manager.conf /etc/kubernetes/
cp kube-controller-manager.service /usr/lib/systemd/system/


scp  kube-controller-manager*.pem 10.10.10.12:/etc/kubernetes/ssl/
scp  kube-controller-manager*.pem 10.10.10.13:/etc/kubernetes/ssl/
scp  kube-controller-manager.kubeconfig kube-controller-manager.conf 10.10.10.12:/etc/kubernetes/
scp  kube-controller-manager.kubeconfig kube-controller-manager.conf 10.10.10.13:/etc/kubernetes/
scp  kube-controller-manager.service 10.10.10.12:/usr/lib/systemd/system/
scp  kube-controller-manager.service 10.10.10.13:/usr/lib/systemd/system/

#####查看证书#####
openssl x509 -in /etc/kubernetes/ssl/kube-controller-manager.pem -noout -text

（7）启动服务+验证

systemctl daemon-reload 
systemctl enable --now kube-controller-manager
systemctl status kube-controller-manager

#####查看状态，三台都是健康的#####
kubectl get componentstatuses

5. master节点部署——kube-scheduler

scheduler的作用：为集群中的pod进行调度。

（1）创建scheduler证书请求文件

cd /data/k8s-work

cat > kube-scheduler-csr.json << "EOF"
{
    "CN": "system:kube-scheduler",
    "hosts": [
      "127.0.0.1",
      "10.10.18.205",
      "10.10.18.102",
      "10.10.18.103",
      "10.10.18.204",
      "10.10.18.104",
      "10.10.18.230",
      "10.10.18.231",
      "10.10.18.232",
      "10.10.18.233",
      "10.10.18.254"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "ST": "Beijing",
        "L": "Beijing",
        "O": "system:kube-scheduler",
        "OU": "system"
      }
    ]
}
EOF

（2）生成scheduler证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

（3）创建scheduler的config文件

#####设置能够访问的集群，以及集群里面所提供的访问链接#####
kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://10.10.10.11:6443 --kubeconfig=kube-scheduler.kubeconfig

#####设置访问集群，管理集群所用到的证书#####
kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig

#####设置安全上下文#####
kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

#####使用安全上下文#####
kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

（4）创建scheduler服务配置文件

cd /data/k8s-work

cat > kube-scheduler.conf << "EOF"
KUBE_SCHEDULER_OPTS="--address=127.0.0.1 \
--kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
--leader-elect=true \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2"
EOF

（5）创建scheduler服务启动的配置文件（进行systemd管理）

cd /data/k8s-work

cat > kube-scheduler.service << "EOF"
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/kube-scheduler.conf
ExecStart=/usr/local/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

注意：
EnvironmentFile 配置文件的路径 ExecStart 执行的二进制文件的路径（使用which kube-scheduler进行查询，最开始分发包的时候已经做了）

（6）分发到集群其他master节点

cp kube-scheduler*.pem /etc/kubernetes/ssl/
cp kube-scheduler.kubeconfig /etc/kubernetes/
cp kube-scheduler.conf /etc/kubernetes/
cp kube-scheduler.service /usr/lib/systemd/system/

scp  kube-scheduler*.pem 10.10.10.12:/etc/kubernetes/ssl/
scp  kube-scheduler*.pem 10.10.10.13:/etc/kubernetes/ssl/
scp  kube-scheduler.kubeconfig kube-scheduler.conf 10.10.10.12:/etc/kubernetes/
scp  kube-scheduler.kubeconfig kube-scheduler.conf 10.10.10.13:/etc/kubernetes/
scp  kube-scheduler.service 10.10.10.12:/usr/lib/systemd/system/
scp  kube-scheduler.service 10.10.10.13:/usr/lib/systemd/system/

（7）服务启动+验证

systemctl daemon-reload
systemctl enable --now kube-scheduler
systemctl status kube-scheduler

#####查看状态，三台都是健康的#####
kubectl get componentstatuses

四. node节点部署

1. worked node节点部署——Containerd & runc

工作节点承担用户负载的运行。1.20版本之后不再唯一支持docker，也可以使用containerd轻量的容器管理工具，可以使机器占用资源更少。

（1）下载并安装Containerd

注意：

github上找自己需要的版本即可，我们用的是1.6.1版本的。一定要用cri-xxxxx-cni-xxxxx 这样形式的软件包。这种软件包里面既包含了对于容器管理方面的使用，由能够使用containerd，还能对容器网络插件进行使用。

#####所有node节点都执行#####
cd /data/k8s-work
wget https://github.com/containerd/containerd/releases/download/v1.6.1/cri-containerd-cni-1.6.1-linux-amd64.tar.gz


#####默认解压后会有如下目录：etc  opt  usr 会把对应的目解压到/下对应目录中，这样就省去复制文件步骤#####
tar -xf cri-containerd-cni-1.6.1-linux-amd64.tar.gz -C /

which containerd

#####这里的工具可以完成网络创建#####
ls /opt/cni/bin/

（2）生成配置文件并修改

mkdir /etc/containerd
#####创建配置文件模板#####
containerd config default >/etc/containerd/config.toml

# ls /etc/containerd/
config.toml

#####下面的配置文件中已修改，可不执行，仅修改默认时执行#####
sed -i 's@systemd_cgroup = false@systemd_cgroup = true@' /etc/containerd/config.toml
#####下面的配置文件中已修改，可不执行，仅修改默认时执行#####
sed -i 's@k8s.gcr.io/pause:3.6@registry.aliyuncs.com/google_containers/pause:3.6@' /etc/containerd/config.toml


cat >/etc/containerd/config.toml<<EOF
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = -999

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    enable_selinux = false
    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"
    stats_collect_period = 10
    systemd_cgroup = true
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    [plugins.cri.containerd]
      snapshotter = "overlayfs"
      no_pivot = false
      [plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = ""
        runtime_root = ""
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
    [plugins.cri.cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = "/etc/cni/net.d/10-default.conf"
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = [
            "https://docker.mirrors.ustc.edu.cn",
            "http://hub-mirror.c.163.com"
          ]
        [plugins.cri.registry.mirrors."gcr.io"]
          endpoint = [
            "https://gcr.mirrors.ustc.edu.cn"
          ]
        [plugins.cri.registry.mirrors."k8s.gcr.io"]
          endpoint = [
            "https://gcr.mirrors.ustc.edu.cn/google-containers/"
          ]
        [plugins.cri.registry.mirrors."quay.io"]
          endpoint = [
            "https://quay.mirrors.ustc.edu.cn"
          ]
        [plugins.cri.registry.mirrors."registry.sensetime.com"]
          endpoint = [
            "https://registry.sensetime.com"
          ]
    [plugins.cri.x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""
  [plugins.diff-service]
    default = ["walking"]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
  [plugins.opt]
    path = "/opt/containerd"
  [plugins.restart]
    interval = "10s"
  [plugins.scheduler]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"
EOF

（3）安装runc

runc才是真正运行容器的，containerd实际上也是一种封装

由于上述软件包中包含的runc对系统依赖过多，所以建议单独下载安装。

默认的runc执行时提示：runc: symbol lookup error: runc: undefined symbol: seccomp_notify_respond

#####本地下载#####
cd /data/k8s-work
wget https://github.com/opencontainers/runc/releases/download/v1.1.0/runc.amd64

#####10.10.18.102 10.10.18.103#####
cd /data/k8s-work
scp -r caominghui.vendor@10.4.48.14:/data/k8s-work/runc.amd64 .
chmod +x runc.amd64

#####替换掉原软件包中的runc，yes覆盖掉#####
mv runc.amd64 /usr/local/sbin/runc

runc -v


#####此时执行runc，会正常显示，不会在error####
runc

#####启动服务#####
systemctl enable containerd
systemctl start containerd
systemctl status containerd

2. worked node节点部署——kubelet

k8s集群不对底层容器进行管理，它最小的调度管理单元的pod。而pod中包含容器，kubelet是对容器进行管理的。

（1）创建kubelet-bootstrap.kubeconfig

#####这里的所有命令只在node1 node2执行即可#####
cd /data/k8s-work/

#####使用/etc/kubernetes/token.csv的一个token，先截取出来#####
BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /etc/kubernetes/token.csv)
echo $BOOTSTRAP_TOKEN

#####设置集群#####
kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://10.10.10.24:6443 --kubeconfig=kubelet-bootstrap.kubeconfig

#####设置访问证书#####
kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap.kubeconfig

#####设置安全上下文#####
kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig

#####使用安全上下文#####
kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig



iptables -P FORWARD ACCEPT

#####创建一个集群角色cluster-system-anonymous用户的绑定#####
kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=kubelet-bootstrap

iptables -P FORWARD ACCEPT

#####创建kubelet-bootstrap绑定关系#####
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig

#####进行查看#####
kubectl describe clusterrolebinding cluster-system-anonymous

iptables -P FORWARD ACCEPT

kubectl describe clusterrolebinding kubelet-bootstrap

（2）创建kubelet配置文件

#####没给node1和node2之间做免密登录，所以这里在node1和node2上同时进行 10.10.10.12 10.10.10.13#####
cd /data/k8s-work
#####"address": "10.10.10.12"这里写对应的本机的ip地址#####
cat > kubelet.json << "EOF"
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/ssl/ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "address": "10.10.10.12",
  "port": 10250,
  "readOnlyPort": 10255,
  "cgroupDriver": "systemd",                    
  "hairpinMode": "promiscuous-bridge",
  "serializeImagePulls": false,
  "clusterDomain": "cluster.local.",
  "clusterDNS": ["10.96.0.2"]
}
EOF

（3）创建kubelet服务启动管理文件（会对containerd进行调用）

#####没给node1和node2之间做免密登录，所以这里在node1和node2上同时进行 10.10.10.12 10.10.10.13#####
cat > kubelet.service << "EOF"
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \
  --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \
  --cert-dir=/etc/kubernetes/ssl \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --config=/etc/kubernetes/kubelet.json \
  --cni-bin-dir=/opt/cni/bin \
  --cni-conf-dir=/etc/cni/net.d \
  --container-runtime=remote \
  --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
  --network-plugin=cni \
  --rotate-certificates \
  --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2 \
  --root-dir=/etc/cni/net.d \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

（4）同步所有文件

#####没给node1和node2之间做免密登录，所以这里在node1和node2上同时进行 10.10.10.12 10.10.10.13#####
cp kubelet-bootstrap.kubeconfig /etc/kubernetes/
cp kubelet.json /etc/kubernetes/
cp kubelet.service /usr/lib/systemd/system/

（5）启动服务

#####在node1和node2上同时进行#####
mkdir -p /var/lib/kubelet
mkdir -p /var/log/kubernetes

systemctl daemon-reload
systemctl enable --now kubelet

systemctl status kubelet

kubectl get nodes


kubectl get csr

3. worked node节点部署——kube-proxy

达到为pod提供网络的目的。部署过程中不涉及containerd的使用。

（1）创建情求证书

#####没给node1和node2之间做免密登录，所以这里在node1和node2上同时进行#####
cat > kube-proxy-csr.json << "EOF"
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "kubemsb",
      "OU": "CN"
    }
  ]
}
EOF

（2）生成证书

#####没给node1和node2之间做免密登录，所以这里在node1和node2上同时进行#####
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

#####如果报错bash: cfssl: command not found,执行以下命令后再此生成#####
source /etc/profile
cfssl version

# ls kube-proxy*
kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem

（3）创建kubeconfig文件

#####没给node1和node2之间做免密登录，所以这里在node1和node2上同时进行#####

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://10.10.10.24:6443 --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

（4）创建服务配置文件

#####没给node1和node2之间做免密登录，所以这里在node1和node2上同时进行#####

cat > kube-proxy.yaml << "EOF"
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 10.10.10.12
clientConnection:
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 10.244.0.0/16
healthzBindAddress: 10.10.10.12:10256
kind: KubeProxyConfiguration
metricsBindAddress: 10.10.10.12:10249
mode: "ipvs"
EOF

注意：要改成对应的本地IP

（5）创建服务启动文件

cat >  kube-proxy.service << "EOF"
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \
  --config=/etc/kubernetes/kube-proxy.yaml \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

（6）同步文件

cp kube-proxy*.pem /etc/kubernetes/ssl/
cp kube-proxy.kubeconfig kube-proxy.yaml /etc/kubernetes/
cp kube-proxy.service /usr/lib/systemd/system/

scp kube-proxy*.pem  10.10.10.13:/etc/kubernetes/ssl/
scp kube-proxy.kubeconfig kube-proxy.yaml 10.10.10.13:/etc/kubernetes/
scp kube-proxy.service 10.10.10.13:/usr/lib/systemd/system/

（7）启动服务

mkdir -p /var/lib/kube-proxy

systemctl daemon-reload
systemctl enable --now kube-proxy

systemctl status kube-proxy

五. 网络组件部署Calico

在master节点上进行操作

1. 下载文件

cd /data/k8s-work/
wget https://docs.projectcalico.org/v3.19/manifests/calico.yaml

2. 修改文件

让calico可以分配pod的网络地址出来

修改calico.yaml文件的第3683 3684行，修改为预设的pod网段

3683             - name: CALICO_IPV4POOL_CIDR
3684               value: "10.244.0.0/16"

3. 应用文件

从文件或 stdin 对资源应用配置更改。

kubectl apply -f calico.yaml

4. 验证结果

kubectl get pods -n kube-system  
                      
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-7cc8dd57d9-6sngn   1/1     Running   0          92s
calico-node-k9n25                          1/1     Running   0          92s
calico-node-qqx7z                          1/1     Running   0          92s


kubectl get pods -n kube-system -o wide                                                                                                                                            
NAME                                       READY   STATUS    RESTARTS   AGE    IP             NODE    NOMINATED NODE   READINESS GATES
calico-kube-controllers-7cc8dd57d9-6sngn   1/1     Running   0          119s   10.88.0.2      node1   <none>           <none>
calico-node-k9n25                          1/1     Running   0          119s   10.10.18.102   node1   <none>           <none>
calico-node-qqx7z                          1/1     Running   0          119s   10.10.18.103   node2   <none>           <none>


kubectl get pods -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-7cc8dd57d9-6sngn   1/1     Running   0          3m4s
kube-system   calico-node-k9n25                          1/1     Running   0          3m4s
kube-system   calico-node-qqx7z                          1/1     Running   0          3m4s

kubectl get nodes
NAME    STATUS   ROLES    AGE   VERSION
node1   Ready    <none>   9d    v1.21.10
node2   Ready    <none>   9d    v1.21.10

六. 部署CoreDNS

为pod提供域名解析，使用了1.8.4版本，放在10.10.10.11

1. 生成yaml文件

cat >  coredns.yaml << "EOF"
apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
rules:
  - apiGroups:
    - ""
    resources:
    - endpoints
    - services
    - pods
    - namespaces
    verbs:
    - list
    - watch
  - apiGroups:
    - discovery.k8s.io
    resources:
    - endpointslices
    verbs:
    - list
    - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        kubernetes cluster.local  in-addr.arpa ip6.arpa {
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf {
          max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/name: "CoreDNS"
spec:
  # replicas: not specified here:
  # 1. Default is 1.
  # 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: coredns
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      nodeSelector:
        kubernetes.io/os: linux
      affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
           - weight: 100
             podAffinityTerm:
               labelSelector:
                 matchExpressions:
                   - key: k8s-app
                     operator: In
                     values: ["kube-dns"]
               topologyKey: kubernetes.io/hostname
      containers:
      - name: coredns
        image: coredns/coredns:1.8.4
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
          readOnly: true
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "CoreDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.96.0.2
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
  - name: metrics
    port: 9153
    protocol: TCP
 
EOF

2. 应用文件

kubectl apply -f coredns.yaml

3. 验证结果

kubectl get svc -n kube-system   
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.2    <none>        53/UDP,53/TCP,9153/TCP   51s

#####看一下coredns是否以容器的方式运行#####
kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-7cc8dd57d9-6sngn   1/1     Running   0          14m
calico-node-k9n25                          1/1     Running   0          14m
calico-node-qqx7z                          1/1     Running   0          14m
coredns-675db8b7cc-bg997                   1/1     Running   0          76s

#####可以看到它运行在哪台机器上#####
kubectl get pods -n kube-system -o wide

七. 验证集群是否可用

尝试构架一个nginx的pod，查看是否可用，映射到本地30001端口

cd /data/k8s-work

cat >  nginx.yaml  << "EOF"
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx-web
spec:
  replicas: 2
  selector:
    name: nginx
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.19.6
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service-nodeport
spec:
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30001
      protocol: TCP
  type: NodePort
  selector:
    name: nginx
EOF

#####生成pod#####

kubectl apply -f nginx.yaml

#####查看pod#####
kubectl get pod

NAME              READY   STATUS    RESTARTS   AGE
nginx-web-k6nvd   1/1     Running   0          19m
nginx-web-mld9f   1/1     Running   0          19m


#####查看具体信息#####
get pod -o wide
NAME              READY   STATUS    RESTARTS   AGE   IP               NODE    NOMINATED NODE   READINESS GATES
nginx-web-k6nvd   1/1     Running   0          21m   10.244.166.130   node1   <none>           <none>
nginx-web-mld9f   1/1     Running   0          21m   10.244.104.3     node2   <none>           <none>

#####查看所在的service#####
kubectl get svc 
       
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes               ClusterIP   10.96.0.1       <none>        443/TCP        18d
nginx-service-nodeport   NodePort    10.96.161.183   <none>        80:30001/TCP   22m

#####去node节点查看#####
[caominghui.vendor@node1 ~]$ ss -anput | grep ":30001"
tcp    LISTEN     0      16384     *:30001                 *:*   

[root@node2 caominghui.vendor]# ss -anput | grep ":30001"
tcp    LISTEN     0      16384     *:30001                 *:*               users:(("kube-proxy",pid=47586,fd=14))

八. 安装Dashboard 插件

#####下载对应版本的ymal#####

cd  /data/k8s-work
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.4.0/aio/deploy/recommended.yaml


#####为了方便，我们将Service改成NodePort类型，注意 YAML 中最下面的 Service 部分新增一个type=NodePort#####
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  ports:
    - port: 443
      targetPort: 8443
  type: NodePort                #####增加了这一行#####
  selector:
    k8s-app: kubernetes-dashboard

#####开始部署#####
kubectl apply -f recommended.yaml

#####查看状态 kubernetes-dashboard 是否为running#####
kubectl get pod -A

NAMESPACE              NAME                                        READY   STATUS    RESTARTS   AGE
default                nginx-web-k6nvd                             1/1     Running   0          4h14m
default                nginx-web-mld9f                             1/1     Running   0          4h14m
kube-system            calico-kube-controllers-7cc8dd57d9-6sngn    1/1     Running   0          2d
kube-system            calico-node-k9n25                           1/1     Running   0          2d
kube-system            calico-node-qqx7z                           1/1     Running   0          2d
kube-system            coredns-675db8b7cc-bg997                    1/1     Running   0          2d
kubernetes-dashboard   dashboard-metrics-scraper-c45b7869d-v8jdc   1/1     Running   0          20m
kubernetes-dashboard   kubernetes-dashboard-576cb95f94-bw8l7       1/1     Running   0          20m

#####查看端口#####
kubectl get svc kubernetes-dashboard -n kubernetes-dashboard

NAME                   TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.96.48.41   <none>        443:31667/TCP   22m

然后直接访问集群中的任何一个节点 IP 加上上面的31667端口即可打开 dashboard 页面了

访问了https://10.10.10.12:31667 https://10.10.10.13:31667

#####master 10.10.10.11执行#####
cd /data/k8s-work

vim dashboard-adminuser.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard

#####生成#####
kubectl apply -f dashboard-adminuser.yaml


#####执行下面的命令, 生成登录用的token#####
kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')
最后这一行的token, 保存下来，登录的时候粘贴上去
Name:         admin-user-token-c2zwd
Namespace:    kubernetes-dashboard
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: admin-user
              kubernetes.io/service-account.uid: c689d24c-924e-49b4-bb1d-0264631e6b18

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1359 bytes
namespace:  20 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IlNpOGdZNENGcENxSXJIOERVSXVXLVlzdVJUeEZYbFpEbEVGTWExXzVKX2MifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2V...................好长

清理和后续步骤（没试过，有机会试试吧）

删除管理员ServiceAccount和ClusterRoleBinding.

kubectl -n kubernetes-dashboard delete serviceaccount admin-user
kubectl -n kubernetes-dashboard delete clusterrolebinding admin-user

注：token默认时间只有12小时！！！！我们既然已经有了页面，可以直接在页面上尝试修改。（才不承认是自己没有试过配置文件修改的方法）

九. 添加镜像仓库+安装kubectx

kubectl create secret docker-registry regcred（regcred是给镜像仓库起的名字） \
  --docker-server=<你的镜像仓库服务器> \
  --docker-username=<你的用户名> \
  --docker-password=<你的密码> \
  --docker-email=<你的邮箱地址>

#####具体的操作命令举例#####
kubectl create secret docker-registry regcred   --docker-server=https://registry.xxxxx.com/   --docker-username=cmh   --docker-password=119.cmh   --docker-email=cmh@xxxxx.com

k8s每次查看指定命名空间的资源都需要加 -n 命名空间来指定命名空间，不加的话默认是default名称空间，这非常繁琐。可以通过开源项目kubens来切换当前命名空间，切换命名空间后，就无需每次都使用 -n 命令来指定命名空间了。

#####安装kubectx#####
cd /data/k8s-work/
git clone  https://github.com/ahmetb/kubectx 
cd kubectx/
cp  kubectx  kubens  /usr/bin/
 
#直接使用即可
kubens
kubens <name>  就可以完成切换了

总结

七七八八记录了一些流程，具体操作中也遇到了一些网络上的问题。实践出真知，一定要亲自尝试才能有更多的收获。