1. kubernetes高可用集群环境搭建（kubeadm-v1.20.6）
1.1 前置知识点
目前生产部署Kubernetes 集群主要有两种方式：
kubeadm
Kubeadm 是一个K8s 部署工具，提供kubeadm init 和kubeadm join，用于快速部署Kubernetes 集群。
官方地址：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/
二进制包
从github 下载发行版的二进制包，手动部署每个组件，组成Kubernetes 集群。
Kubeadm 降低部署门槛，但屏蔽了很多细节，遇到问题很难排查。如果想更容易可控，推荐使用二进制包部署Kubernetes 集群，虽然手动部署麻烦点，期间可以学习很多工作原理，也利于后期维护。
1.2 kubeadm 部署方式介绍
kubeadm 是官方社区推出的一个用于快速部署kubernetes 集群的工具，这个工具能通过两条指令完成一个kubernetes 集群的部署：
- 创建一个master01 节点kubeadm init
- 将Node 节点加入到当前集群中$ kubeadm join <master01 节点的IP 和端口>
1.3 安装要求
在开始之前，部署Kubernetes 集群机器需要满足以下几个条件： - 一台或多台机器，操作系统CentOS7.x-86_x64
- 硬件配置：2GB 或更多RAM，2 个CPU 或更多CPU，硬盘30GB 或更多
- 集群中所有机器之间网络互通
- 可以访问外网，需要拉取镜像
- 禁止swap 分区
1.4 最终目标 - 在所有节点上安装Docker 和kubeadm
- 部署Kubernetes 多master01
- 部署容器网络插件（calico）
- 部署高可用组件（nginx+keepalived）
- 部署Kubernetes Node，将节点加入Kubernetes 集群中
- 部署Dashboard Web 页面，可视化查看Kubernetes 资源
- 部署metrics-server
- 部署Prometheus+Grafana
- 部署EFK+logstash构建日志收集平台
- 部署Rancher管理k8s集群
- 部署helm v3包管理工具
- 部署ingress控制器
- 部署Harbor私有镜像仓库
1.5 准备环境
角色 IP地址 组件
master1 192.168.225.138 apiserver、controller-manager、scheduler、etcd、calico、docker，kubectl，kubeadm，kubelet、kube-proxy
master2 192.168.225.139 apiserver、controller-manager、scheduler、etcd、calico、docker，kubectl，kubeadm，kubelet、kube-proxy
master3 192.168.225.140 apiserver、controller-manager、scheduler、etcd、calico、docker，kubectl，kubeadm，kubelet、kube-proxy
node01 192.168.225.141 docker，kube-proxy，kubeadm，kubelet、calico、coredns
node02 192.168.225.142 docker，kube-proxy，kubeadm，kubelet、calico
node03 192.168.225.143 docker，kube-proxy，kubeadm，kubelet、calico
vip 192.168.4.150
1.6 环境初始化
1.6.1 检查操作系统的版本，安装规划修改所有节点hostname（所有节点操作）
#1 此方式下安装kubernetes集群要求Centos版本要在7.5或之上
[root@master01 ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
修改主机名称
[root@master01 ~]# hostnamectl set-hostname master01
1.6.2 配置集群节点免密登录（所有节点操作）
#1 生成ssh密钥对：
[root@master01 ~]# ssh-keygen -t rsa #一路回车，不输入密码
#2 把本地的ssh公钥文件安装到远程主机对应的账号中：
[root@master01 ~]# ssh-copy-id -i .ssh/id_rsa.pub master02
[root@master01 ~]# ssh-copy-id -i .ssh/id_rsa.pub master03
[root@master01 ~]# ssh-copy-id -i .ssh/id_rsa.pub node01
[root@master01 ~]# ssh-copy-id -i .ssh/id_rsa.pub node02
[root@master01 ~]# ssh-copy-id -i .ssh/id_rsa.pub node03
#其他几个节点的操作方式跟master01一致，在此只展示master01的操作方式；
1.6.3 主机名解析（所有节点操作）
为了方便集群节点间的直接调用，配置一下主机名解析，企业中推荐使用内部DNS服务器
#1 主机名成解析 编辑6台服务器的/etc/hosts文件，添加下面内容
[root@master0101 ~]# cat >> /etc/hosts <<EOF
192.168.225.138 master01
192.168.225.139 master02
192.168.225.140 master03
192.168.225.141 node01
192.168.225.142 node02
192.168.225.143 node03
EOF
1.6.4 时间同步（所有节点操作）
#1 安装ntpdate命令，
[root@master0101 ~]# yum install ntpdate -y
#跟网络源做同步
[root@master0101 ~]# ntpdate ntp.aliyun.com
#2 把时间同步做成计划任务
[root@master0101 ~]# crontab -e
* */1 * * * /usr/sbin/ntpdate ntp.aliyun.com
#3 重启crond服务，修改时区
[root@master0101 ~]# systemctl restart crond.service && timedatectl set-timezone Asia/Shanghai
1.6.5 禁用iptable和firewalld服务（所有节点操作）
kubernetes和docker 在运行的中会产生大量的iptables规则，为了不让系统规则跟它们混淆，直接关闭系统的规则
#1 关闭firewalld服务
[root@master0101 ~]# systemctl stop firewalld && systemctl disable firewalld
#2 安装iptables服务，同时关闭iptables服务
[root@master0101 ~]# yum install -y iptables-services.x86_64
[root@master0101 ~]# systemctl stop iptables && systemctl disable iptables
#3 清空iptables规则
[root@master0101 ~]# iptables -F
1.6.6 禁用selinux（所有节点操作）
selinux是linux系统下的一个安全服务，如果不关闭它，在安装集群中会产生各种各样的奇葩问题
#1 关闭selinux，注意修改完毕之后需要重启服务器才可以生效
[root@master0101 ~]# sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
1.6.7 禁用swap分区（所有节点操作）
swap分区指的是虚拟内存分区，它的作用是物理内存使用完，之后将磁盘空间虚拟成内存来使用，启用swap设备会对系统的性能产生非常负面的影响，因此kubernetes要求每个节点都要禁用swap设备，但是如果因为某些原因确实不能关闭swap分区，就需要在集群安装过程中通过明确的参数进行配置说明
#1 编辑分区配置文件/etc/fstab，注释掉swap分区一行，如果是克隆的虚拟机，需要删除UUID,注意修改完毕之后需要重启服务器
[root@master0101 ~]# vim /etc/fstsb
1.6.8 修改linux的内核参数（所有节点操作）
#1 修改linux的内核采纳数，添加网桥过滤和地址转发功能
#2 加载网桥过滤模块
[root@master01 ~]# modprobe br_netfilter
#3 编辑/etc/sysctl.d/k8s.conf文件，添加如下配置：
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
[root@master01 ~]#cat >> /etc/sysctl.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
#4 重新加载配置
[root@master01 ~]# sysctl -p /etc/sysctl.d/k8s.conf
#5 查看网桥过滤模块是否加载成功
[root@master01 ~]# lsmod | grep br_netfilter
#6 开机加载内核模块
[root@master01 ~]# echo "modprobe br_netfilter" >> /etc/profile
1.6.9 配置ipvs功能（所有节点操作）
在Kubernetes中Service有两种带来模型，一种是基于iptables的，一种是基于ipvs的两者比较的话，ipvs的性能明显要高一些，但是如果要使用它，需要手动载入ipvs模块
#1 安装ipset和ipvsadm
[root@master01 ~]# yum install ipvsadm.x86_64 ipset -y
#2 添加需要加载的模块写入脚本文件
[root@master01 ~]# cat <<EOF> /etc/sysconfig/modules/ipvs.modules
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
#3 为脚本添加执行权限
[root@master01 ~]# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs
1.6.10 配置部署集群所需要的yum源（所有节点操作）
阿里云官方镜像站：https://developer.aliyun.com/mirror/?spm=a2c6h.13651102.0.0.77ae1b112Ukw1G&serviceType=mirror
需要的yum.repo源
---CentOS-Base.repo
---docker-ce.repo
---epel.repo
---kubernetes.repo
#1 备份原有yum.repo源
[root@master01 ~]# mkdir /root/yum.bak
[root@master01 ~]# mv /etc/yum.repos.d/* /root/yum.repos.bak
#2 将准备好的yum源通过scp的方式上传到集群所有节点的yum.repo目录下，也可以通过一下方式下载，
安装一些依赖
[root@master01 yum.repos.d]# yum install -y yum-utils device-mapper-persistent-data lvm2
[root@master01 yum.repos.d]# wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
下载的CentOS-Base.repo后面安转启动docker会无法启动，
解决办法参考：https://blog.csdn.net/qq_47855463/article/details/119656678
[root@master01 yum.repos.d]# wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
[root@master01 yum.repos.d]# curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@master01 yum.repos.d]# sudo sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo
[root@master01 yum.repos.d]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
[root@master01 yum.repos.d]# ll
total 16
-rw-r--r--. 1 root root 2523 Mar 17 09:44 CentOS-Base.repo
-rw-r--r--. 1 root root 2081 Mar 22 21:43 docker-ce.repo
-rw-r--r--. 1 root root 1050 Mar 22 22:43 epel.repo
-rw-r--r--. 1 root root 133 Mar 22 22:23 kubernetes.repo
#3 更新yum源
[root@master01 ~]# yum makecache
#4 配置安装基础依赖包
[root@master01 ~]# yum install -y wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack
1.6.11 安装docker（所有节点操作）
#1 查看当前镜像源中支持的docker版本
[root@master01 ~]# yum list docker-ce --showduplicates
#2 安装特定版本的docker-ce，必须制定--setopt=obsoletes=0，否则yum会自动安装更高版本
[root@master01 ~]# yum install docker-ce docker-ce-cli containerd.io --skip-broken
#3 添加一个配置文件，Docker 在默认情况下使用Vgroup Driver为cgroupfs，而Kubernetes推荐使用systemd来替代cgroupfs，同时配置docker镜像源加速
[root@master01 ~]# mkdir /etc/docker
[root@master01 ~]# cat <<EOF> /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": ["https://kn0t2bca.mirror.aliyuncs.com"]
}
EOF
#4 启动dokcer
[root@master01 ~]# systemctl start docker.service && systemctl enable docker.service
[root@master01 ~]# systemctl status docker.service
1.6.12 安装Kubernetes组件（所有节点操作）
#1 安装kubeadm、kubelet和kubectl
[root@master01 ~]# yum install --setopt=obsoletes=0 kubelet-1.20.6 kubeadm-1.20.6 kubectl-1.20.6 -y
#2 启动kubelet，并设置为开机自启
[root@master01 ~]# systemctl enable --now kubelet
（systemctl enable --now kubelet = systemctl start kubelet && systemctl enable kubelet）
[root@master01 ~]# systemctl status kubelet
#可以看到kubelet状态不是running（auto-restart）状态，这个是正常的不用管，等k8s组件起来这个kubelet就正常了。
*tab补全键设置
[root@master01 ~]# kubectl completion bash >/etc/bash_completion.d/kubectl
[root@master01 ~]# kubeadm completion bash >/etc/bash_completion.d/kubeadm
1.6.13 准备集群镜像（所有节点操作）
下载路径：https://blog.51cto.com/transfer?https://dl.k8s.io/v1.20.7/kubernetes-server-linux-amd64.tar.gz
#1 在安装kubernetes集群之前，必须要提前准备好集群需要的镜像，将部署集群需要的镜像通过scp的方式上传到集群的每个节点上面
[root@master01 ~]# ll
-rw-r--r-- 1 root root 1083635200 Mar 17 10:00 k8simage-1-20-6.tar.gz
#2 在各个节点上面使用docker命令解压集群镜像文件，不能使用tar直接去解压
[root@master01 ~]# docker load -i kube-controller-manager-v1.20.6.tar.gz
把下载得tar包挨个导入
#3 查看解压完成后的docker镜像
[root@master01 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-proxy v1.20.6 9a1ebfd8124d 11 months ago 118MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.20.6 560dd11d4550 11 months ago 116MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.20.6 b93ab2ec4475 11 months ago 47.3MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.20.6 b05d611c1af9 11 months ago 122MB
calico/pod2daemon-flexvol v3.18.0 2a22066e9588 12 months ago 21.7MB
calico/node v3.18.0 5a7c4970fbc2 13 months ago 172MB
calico/cni v3.18.0 727de170e4ce 13 months ago 131MB
calico/kube-controllers v3.18.0 9a154323fbf7 13 months ago 53.4MB
registry.aliyuncs.com/google_containers/etcd 3.4.13-0 0369cf4303ff 19 months ago 253MB
registry.aliyuncs.com/google_containers/coredns 1.7.0 bfe3a36ebd25 21 months ago 45.2MB
registry.aliyuncs.com/google_containers/pause 3.2 80d28bedfe5d 2 years ago 683kB
1.6.14 配置nginx和keepalived（3个master节点操作）
#1 安装keepalived和nginx
[root@master01 ~]# yum install nginx keepalived -y
在三个master节点配置
检查nginx状态脚本：
cat > /etc/keepalived/check_nginx.sh << "EOF"
#!/bin/bash
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
exit 1
else
exit 0
fi
EOF
chmod +x /etc/keepalived/check_nginx.sh
Keepalived配置（master）
cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from fage@qq.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 100 # 优先级，备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
# 虚拟IP
virtual_ipaddress {
192.168.4.150
}
track_script {
check_nginx
}
}
EOF
Keepalived配置（backup）
cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from fage@qq.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_BACKUP
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.4.150
}
track_script {
check_nginx
}
}
EOF
检查nginx状态脚本：
cat > /etc/keepalived/check_nginx.sh << "EOF"
#!/bin/bash
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
exit 1
else
exit 0
fi
EOF
chmod +x /etc/keepalived/check_nginx.sh
``
#2 修改3个节点上面的keepalived配置文件，修改角色、优先级、网卡名称，其他保持不变即可，nginx配置文件不用修改
[root@master01 ~]# ll /etc/keepalived/
total 8
-rw-r--r-- 1 root root 134 Mar 22 23:16 check_nginx.sh
-rw-r--r-- 1 root root 986 Mar 23 22:51 keepalived.conf
[root@master01 ~]# ll /etc/nginx/nginx.conf
-rw-r--r-- 1 root root 1442 Mar 22 23:12 /etc/nginx/nginx.conf
#3 先启动3个节点的nginx服务，在启动keepalived服务
[root@master01 ~]# systemctl daemon-reload
[root@master01 ~]# systemctl restart nginx && systemctl enable nginx && systemctl status nginx
[root@master01 ~]# systemctl restart keepalived && systemctl enable keepalived && systemctl status keepalived
#4 查看VIP是否生成
[root@master01 ~]# ip addr
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:ba:33:03 brd ff:ff:ff:ff:ff:ff
inet 192.168.225.138/24 brd 192.168.225.255 scope global noprefixroute ens192
valid_lft forever preferred_lft forever
inet 192.168.225.150/24 scope global secondary ens192
valid_lft forever preferred_lft forever
inet6 fe80::88c2:50d5:146:e608/64 scope link noprefixroute
valid_lft forever preferred_lft forever
1.6.15 集群初始化（master01节点操作）
```#1 创建集群初始化文件
[root@master01 ~]# vim kubeadm-config.yaml
# 根据实际部署环境修改信息：
vim kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.20.6
controlPlaneEndpoint: "192.168.1.18:6443" #vip和端口
imageRepository: k8s.gcr.io
apiServer:
timeoutForControlPlane: 4m0s
certSANs:
- 192.168.4.138
- 192.168.4.139
- 192.168.4.140
- 192.168.4.141
- 192.168.4.142
- 192.168.4.143
- 192.168.4.150
extraArgs:
authorization-mode: "Node,RBAC"
service-node-port-range: 30000-36000 # service端口范围
networking:
dnsDomain: cluster.local
serviceSubnet: "10.96.0.0/16" # service网段信息
podSubnet: "10.244.0.0/16" # pod网段信息
---
#开启ipvs模式
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
也可以下面命令来集群初始化
[root@master01 ~]# kubeadm init \
--image-repository=registry.cn-hangzhou.aliyuncs.com/k8sos \
--kubernetes-version=v1.20.6 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16
#2 使用配置文件初始化集群
[root@master01 ~]# kubeadm init --config kubeadm-config.yaml --ignore-preflight-errors=SystemVerification
依次执行提示命令：
[root@master01 ~]# mkdir -p $HOME/.kube
[root@master01 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master01 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
1.6.16 master节点加入集群（master02-03节点操作）
#1 创建目录，拷贝证书
创建目录
[root@master02 ~]# cd /root && mkdir -p /etc/kubernetes/pki/etcd &&mkdir -p ~/.kube/
拷贝证书
[root@master01 ~]# scp /etc/kubernetes/pki/ca.crt master02:/etc/kubernetes/pki/
[root@master01 ~]# scp /etc/kubernetes/pki/ca.key master02:/etc/kubernetes/pki/
[root@master01 ~]# scp /etc/kubernetes/pki/sa.key master02:/etc/kubernetes/pki/
[root@master01 ~]# scp /etc/kubernetes/pki/sa.pub master02:/etc/kubernetes/pki/
[root@master01 ~]# scp /etc/kubernetes/pki/front-proxy-ca.crt master02:/etc/kubernetes/pki/
[root@master01 ~]# scp /etc/kubernetes/pki/front-proxy-ca.key master02:/etc/kubernetes/pki/
[root@master01 ~]# scp /etc/kubernetes/pki/etcd/ca.crt master02:/etc/kubernetes/pki/etcd/
[root@master01 ~]# scp /etc/kubernetes/pki/etcd/ca.key master02:/etc/kubernetes/pki/etcd/
[root@master03 ~]# cd /root && mkdir -p /etc/kubernetes/pki/etcd &&mkdir -p ~/.kube/
[root@master01 ~]# scp /etc/kubernetes/pki/ca.crt master03:/etc/kubernetes/pki/
[root@master01 ~]# scp /etc/kubernetes/pki/ca.key master03:/etc/kubernetes/pki/
[root@master01 ~]# scp /etc/kubernetes/pki/sa.key master03:/etc/kubernetes/pki/
[root@master01 ~]# scp /etc/kubernetes/pki/sa.pub master03:/etc/kubernetes/pki/
[root@master01 ~]# scp /etc/kubernetes/pki/front-proxy-ca.crt master03:/etc/kubernetes/pki/
[root@master01 ~]# scp /etc/kubernetes/pki/front-proxy-ca.key master03:/etc/kubernetes/pki/
[root@master01 ~]# scp /etc/kubernetes/pki/etcd/ca.crt master03:/etc/kubernetes/pki/etcd/
[root@master01 ~]# scp /etc/kubernetes/pki/etcd/ca.key master0:/etc/kubernetes/pki/etcd/
同上脚本进行拷贝：
vim cert-rsync-master.sh
#!/bin/bash
USER=root
CONTROL_PLANE_IPS="192.168.4.139 192.168.4.140"
for host in ${CONTROL_PLANE_IPS}; do
ssh "${USER}"@$host mkdir -p /etc/kubernetes/pki/etcd
scp /etc/kubernetes/pki/ca.crt "${USER}"@$host:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/ca.key "${USER}"@$host:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.key "${USER}"@$host:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.pub "${USER}"@$host:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host: /etc/kubernetes/pki/
scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:/etc/kubernetes/pki/etcd/ca.crt
scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:/etc/kubernetes/pki/etcd/ca.key
done
[root@master01 ~] bash cert-rsync-master.sh
#2 查看token文件，token文件的有效期为24h，过期后无法在使用该token加入集群
[root@master01 ~]# kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
6zqaxk.te6y81dkvy73j5hs 23h 2022-03-26T10:43:37+08:00 authentication,signing <none> system:bootstrappers:kubeadm:default-node-token
#3 重新生成token文件的方法
[root@master01 ~]# kubeadm token create --print-join-command
#4 查看token文件的sha256字段
[root@master01 ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
#5 使用token将master02和master03加入到集群中
[root@master02 ~]# kubeadm join 192.168.225.150:16443 --token 6zqaxk.te6y81dkvy73j5hs --discovery-token-ca-cert-hash sha256:f4694c021a9c1467b40254518358f2abf5e50f775e48907f3e08502a09dcc7db --control-plane --ignore-preflight-errors=SystemVerification
[root@master03 ~]# kubeadm join 192.168.225.150:16443 --token 6zqaxk.te6y81dkvy73j5hs --discovery-token-ca-cert-hash sha256:f4694c021a9c1467b40254518358f2abf5e50f775e48907f3e08502a09dcc7db --control-plane --ignore-preflight-errors=SystemVerification
1.6.18 安装calico网络插件（master01节点操作）
#1 将calico的yaml文件通过scp的方式上传到master01上面，然后使用kubectl安装calico
--- 在文末会提供calico.yaml的下载链接
[root@master01 ~]# kubectl apply -f calico.yaml
1.6.19 查看集群状态（master01节点操作）
[root@master01 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master01 Ready control-plane,master 35h v1.20.6
master02 Ready control-plane,master 35h v1.20.6
master03 Ready control-plane,master 22h v1.20.6
node01 Ready <none> 24h v1.20.6
node02 Ready <none> 24h v1.20.6
node03 Ready <none> 24h v1.20.6
操作过程中就会遇到每个人各种不中的问题，可根据报错百度解决，
1.6.20 为node节点打标签（扩展）（master01节点操作）
#1 我们可以看到，节点列表中ROLES这块，node节点为node，如果想标识node节点，使用下面命令实现
[root@master01 ~]# kubectl label node node01 node-role.kubernetes.io/worker=worker
[root@master01 ~]# kubectl label node node02 node-role.kubernetes.io/worker=worker
[root@master01 ~]# kubectl label node node03 node-role.kubernetes.io/worker=worker
[root@master01 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master01 Ready control-plane,master 35h v1.20.6
master02 Ready control-plane,master 35h v1.20.6
master03 Ready control-plane,master 22h v1.20.6
node01 Ready worker 24h v1.20.6
node02 Ready worker 24h v1.20.6
node03 Ready worker 24h v1.20.6
1.7 集群测试
[root@master01 ~]# kubectl create deployment nginx --image=nginx:1.14-alpine
deployment.apps/nginx created
1.7.2 暴露端口
[root@master01 ~]# kubectl expose deploy nginx --port=80 --target-port=80 --type=NodePort
service/nginx exposed
1.7.3 查看服务
[root@master01 ~]# kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/nginx-65c4bffcb6-2qxzh 1/1 Running 0 49s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nginx NodePort 10.10.153.8 <none> 80:31076/TCP 20s
1.7.4 查看pod
浏览器测试结果：
1.7.5 测试coredns服务是否正常
[root@master01 ~]# kubectl exec -it nginx-65c4bffcb6-2qxzh -- sh
/ # ping nginx
PING nginx (10.10.153.8): 56 data bytes
64 bytes from 10.10.153.8: seq=0 ttl=64 time=0.066 ms
64 bytes from 10.10.153.8: seq=1 ttl=64 time=0.151 ms
/ # nslookup nginx.default.svc.cluster.local
Name: nginx.default.svc.cluster.local
Address 1: 10.10.153.8 nginx.default.svc.cluster.local
#10.10.153.8 是创建的nginx的service ip
1.8 部署Dashboard Web
GitHub地址：https://github.com/kubernetes/dashboard
[root@master01 ~]# docker load -i dashboard-2.6.0.tar.gz
[root@master01 ~]# wget https://github.com/kubernetes/dashboard/blob/v2.6.0/aio/deploy/recommended.yaml
如出现一下问题
解决办法：
[root@master01 ~]# vim recommended.yaml
修改完后的配置如下：
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: v1
kind: Namespace
metadata:
name: kubernetes-dashboard
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 31443
selector:
k8s-app: kubernetes-dashboard
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
namespace: kubernetes-dashboard
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-csrf
namespace: kubernetes-dashboard
type: Opaque
data:
csrf: ""
---
###########因为自动生成的证书很多浏览器无法使用，所以自己创建证书
#apiVersion: v1
#kind: Secret
#metadata:
# labels:
# k8s-app: kubernetes-dashboard
# name: kubernetes-dashboard-key-holder
# namespace: kubernetes-dashboard
#type: Opaque
---
kind: ConfigMap
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-settings
namespace: kubernetes-dashboard
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
rules:
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["kubernetes-dashboard-settings"]
verbs: ["get", "update"]
# Allow Dashboard to get metrics.
- apiGroups: [""]
resources: ["services"]
resourceNames: ["heapster", "dashboard-metrics-scraper"]
verbs: ["proxy"]
- apiGroups: [""]
resources: ["services/proxy"]
resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
verbs: ["get"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
rules:
# Allow Metrics Scraper to get metrics from the Metrics server
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: kubernetes-dashboard
image: kubernetesui/dashboard:v2.6.0
imagePullPolicy: Always
ports:
- containerPort: 8443
protocol: TCP
args:
- --auto-generate-certificates
- --namespace=kubernetes-dashboard
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
nodeSelector:
"kubernetes.io/os": linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
ports:
- port: 8000
targetPort: 8000
selector:
k8s-app: dashboard-metrics-scraper
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: dashboard-metrics-scraper
template:
metadata:
labels:
k8s-app: dashboard-metrics-scraper
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: dashboard-metrics-scraper
image: kubernetesui/metrics-scraper:v1.0.8
ports:
- containerPort: 8000
protocol: TCP
livenessProbe:
httpGet:
scheme: HTTP
path: /
port: 8000
initialDelaySeconds: 30
timeoutSeconds: 30
volumeMounts:
- mountPath: /tmp
name: tmp-volume
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
serviceAccountName: kubernetes-dashboard
nodeSelector:
"kubernetes.io/os": linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
volumes:
- name: tmp-volume
emptyDir: {}
#2 修改配置文件，增加直接访问端口
#3 修改配置文件，注释原kubernetes-dashboard-certs对象声明
#4 自签证书
#1 新建证书存放目录
[root@master01 ~]# mkdir /etc/kubernetes/dashboard-certs
[root@master01 ~]# cd /etc/kubernetes/dashboard-certs/
#2 创建命名空间
[root@master01 ~]# kubectl create namespace kubernetes-dashboard
#3 创建key文件
[root@master01 ~]# openssl genrsa -out dashboard.key 2048
#4 证书请求
[root@master01 ~]# openssl req -days 36000 -new -out dashboard.csr -key dashboard.key -subj '/CN=dashboard-cert'
#5 自签证书
[root@master01 ~]# openssl x509 -req -in dashboard.csr -signkey dashboard.key -out dashboard.crt
#6 创建kubernetes-dashboard-certs对象
[root@master01 ~]# kubectl create secret generic kubernetes-dashboard-certs --from-file=dashboard.key --from-file=dashboard.crt -n kubernetes-dashboard
#4 运行dashboard
[root@master01 ~]# kubectl apply -f recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
#5 查看dashboard组件状态
[root@master01 dashboard-certs]# kubectl get all -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
pod/dashboard-metrics-scraper-7b59f7d4df-z24lb 1/1 Running 0 4m20s
pod/kubernetes-dashboard-74d688b6bc-wd54z 1/1 Running 0 78s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dashboard-metrics-scraper ClusterIP 10.10.42.68 <none> 8000/TCP 4m20s
service/kubernetes-dashboard NodePort 10.10.172.2 <none> 443:30001/TCP 4m21s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/dashboard-metrics-scraper 1/1 1 1 4m20s
deployment.apps/kubernetes-dashboard 1/1 1 1 4m20s
NAME DESIRED CURRENT READY AGE
replicaset.apps/dashboard-metrics-scraper-7b59f7d4df 1 1 1 4m20s
replicaset.apps/kubernetes-dashboard-74d688b6bc 1 1 1 4m20s
#6 创建Dashboard管理员账号dashboard-admin.yaml，并apply
[root@master01 ~]# vim dashboard-admin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: dashboard-admin
namespace: kubernetes-dashboard
[root@master01 ~]# kubectl apply -f dashboard-admin.yaml
#7 赋权dashboard-admin-bind-cluster-role.yaml，并apply
[root@master01 ~]# vim dashboard-admin-bind-cluster-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: dashboard-admin-bind-cluster-role
labels:
k8s-app: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: dashboard-admin
namespace: kubernetes-dashboard
[root@master01 ~]# kubectl apply -f dashboard-admin-bind-cluster-role.yaml
#8 获取token信息
[root@master01 ~]# kubectl -n kubernetes-dashboard get secret | grep dashboard-admin | awk '{print $1}'
dashboard-admin-token-w69c5
#9 登录访问https://192.168.225.138:30001/#/login
#1 设置集群参数
[root@master01 ~]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.crt --server="https://192.168.225.150:16443" --embed-certs=true --kubeconfig=/root/shibosen-admin.conf
#2 设置变量
[root@master01 ~]# DEF_NS_ADMIN_TOKEN=$(kubectl get secret dashboard-admin-token-hj6j4 -n kubernetes-dashboard -o jsonpath={.data.token}|base64 -d)
#3 设置客户端认证参数
[root@master01 ~]# kubectl config set-credentials dashboard-admin --token=$DEF_NS_ADMIN_TOKEN --kubeconfig=/root/shibosen-admin.conf
#4 设置安全上下文参数
[root@master01 ~]# kubectl config set-context dashboard-admin@kubernetes --cluster=kubernetes --user=dashboard-admin --kubeconfig=/root/shibosen-admin.conf
#5 设置当前上下文参数
[root@master01 ~]# kubectl config use-context dashboard-admin@kubernetes --kubeconfig=/root/shibosen-admin.conf
1.9 部署metrics-server
[root@master01 ~]# wget https://github.com/kubernetes-sigs/metrics-server/releases/tag/v0.6.0
[root@master01 ~]# https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.6.0/components.yaml
[root@master01 ~]# vim components.yaml
修改后的文件内容如下所示:
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-view: "true"
name: system:aggregated-metrics-reader
rules:
- apiGroups:
- metrics.k8s.io
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- pods
- nodes
- nodes/stats
- namespaces
- configmaps
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
ports:
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: metrics-server
strategy:
rollingUpdate:
maxUnavailable: 0
template:
metadata:
labels:
k8s-app: metrics-server
spec:
containers:
- args:
- --cert-dir=/tmp
- --secure-port=443
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
- --kubelet-insecure-tls
image: k8s.gcr.io/metrics-server/metrics-server:v0.6.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /livez
port: https
scheme: HTTPS
periodSeconds: 10
name: metrics-server
ports:
- containerPort: 443
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz
port: https
scheme: HTTPS
initialDelaySeconds: 20
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 200Mi
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- mountPath: /tmp
name: tmp-dir
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: metrics-server
volumes:
- emptyDir: {}
name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
labels:
k8s-app: metrics-server
name: v1beta1.metrics.k8s.io
spec:
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100
执行如下命令安装Metrics-Server。
[root@master01 ~]# kubectl apply -f components-v0.6.0.yaml
查看metrics-server的pod运行状态
[root@master01 ~]# kubectl get pods -n kube-system| egrep 'NAME|metrics-server'
[root@master01 ~]# kubectl get pod -n kube-system | grep metrics-server
metrics-server-6f9f86ddf9-zphlw 1/1 Running 0 11s
1.10 部署Prometheus+Grafana
下载地址：https://prometheus.io/download/
[root@master01 ~]# cd /usr/local/prometheus
[root@master01 prometheus ]# ./prometheus --config.file=/usr/local/prometheus/prometheus.yml &
[root@master01 ~]#
页面访问：192.168.255.183:9090
2122
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
