感谢：二进制安装Kubernetes（k8s） v1.24.0 IPv4/IPv6双栈 - 小陈运维
kubernetes 1.24 变化较大，详细见：Kubernetes 1.24 的删除和弃用 | Kubernetes
1.k8s基础系统环境配置
宿主机、部署软件：
为了节省资源，将Lb01中的资源部署到Node01、将Lb02中的资源部署到Node02，建议k8s集群与etcd集群分开安装
| 主机名称 | IP地址 | 说明 | 软件 |
|---|---|---|---|
| Master01 | 10.4.7.11 | master节点 | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、 kubelet、kube-proxy、nfs-client |
| Master02 | 10.4.7.12 | master节点 | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、 kubelet、kube-proxy、nfs-client |
| Master03 | 10.4.7.21 | master节点 | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、 kubelet、kube-proxy、nfs-client |
| Node01 | 10.4.7.22 | node节点 | kubelet、kube-proxy、nfs-client、haproxy、keepalived |
| Node02 | 10.4.7.200 | node节点 | kubelet、kube-proxy、nfs-client、haproxy、keepalived |
| 10.4.7.10 | VIP |
宿主机要求：
宿主机要求：必须是Centos8以上，因为 k8s v1.24.0 在CentOS 7有兼容性问题，会导致有些字段不识别kubelet Error getting node 问题求助_Jerry00713的博客-CSDN博客
| 软件 | 版本 |
|---|---|
| 内核 | 5.17.5-1.el8.elrepo 以上 |
| CentOS 8 | v8 或者 v7 |
| kube-apiserver、kube-controller-manager、kube-scheduler、kubelet、kube-proxy | v1.24.0 |
| etcd | v3.5.4 |
| containerd | v1.5.11 |
| cfssl | v1.6.1 |
| cni | v1.1.1 |
| crictl | v1.23.0 |
| haproxy | v1.8.27 |
| keepalived | v2.1.5 |
网段：
宿主机：10.4.7.0/24
k8s 集群IP(clusterIP)：192.168.0.0/16
k8s pod集群(cluster-cidr)：172.7.0.0/16
别名：
(all) 代表 k8s-master01、k8s-master02、k8s-master03、k8s-node01、k8s--node2
(k8s_all) 代表 k8s-master01、k8s-master02、k8s-master03、k8s-node01、k8s--node2
(k8s_master) 代表 k8s-master01、k8s-master02、k8s-master03
(k8s_node) 代表 k8s-node01、k8s--node2
(Lb) 正常找另外两台设备，作为负载均衡、VIP等操作，此文章为节约资源，部署到 k8s-node01、k8s--node2 中
1.1.配置IP (all)
略自己配置宿主机IP
1.2.设置主机名 (all)
10.4.7.11 hostnamectl set-hostname k8s-master01
10.4.7.12 hostnamectl set-hostname k8s-master02
10.4.7.21 hostnamectl set-hostname k8s-master03
10.4.7.22 hostnamectl set-hostname k8s-node01
10.4.7.200 hostnamectl set-hostname k8s-node02
1.3.配置yum源 (all)
1.修改 vi /etc/yum.repos.d/CentOS-Linux-BaseOS.repo 文件
将mirrorlist配置注释掉，并将baseurl修改为阿里云镜像地址
[BaseOS]
name=CentOS-$releasever - Base
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=BaseOS&infra=$infra
baseurl=https://mirrors.aliyun.com/centos/$releasever-stream/BaseOS/$basearch/os/
http://mirrors.aliyuncs.com/centos/$releasever-stream/BaseOS/$basearch/os/
http://mirrors.cloud.aliyuncs.com/centos/$releasever-stream/BaseOS/$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
2.修改 vi /etc/yum.repos.d/CentOS-Linux-AppStream.repo 文件
将mirrorlist配置注释掉，并将baseurl修改为阿里云镜像地址
[AppStream]
name=CentOS-$releasever - AppStream
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=AppStream&infra=$infra
baseurl=https://mirrors.aliyun.com/centos/$releasever-stream/AppStream/$basearch/os/
http://mirrors.aliyuncs.com/centos/$releasever-stream/AppStream/$basearch/os/
http://mirrors.cloud.aliyuncs.com/centos/$releasever-stream/AppStream/$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
3.修改 vi /etc/yum.repos.d/CentOS-Linux-Extras.repo 文件
将mirrorlist配置注释掉，并将baseurl修改为阿里云镜像地址[extras]
name=CentOS-$releasever - Extras
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
baseurl=https://mirrors.aliyun.com/centos/$releasever-stream/extras/$basearch/os/
http://mirrors.aliyuncs.com/centos/$releasever-stream/extras/$basearch/os/
http://mirrors.cloud.aliyuncs.com/centos/$releasever-stream/extras/$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
yum clean all && yum makecache
写成脚本：
sed -e 's|^mirrorlist=|#mirrorlist=|g' -e '/#baseurl=/d' -e '/#mirrorlist/a\baseurl=https://mirrors.aliyun.com/centos/$releasever-stream/BaseOS/$basearch/os/\n http://mirrors.aliyuncs.com/centos/$releasever-stream/BaseOS/$basearch/os/\n http://mirrors.cloud.aliyuncs.com/centos/$releasever-stream/BaseOS/$basearch/os/' -i /etc/yum.repos.d/CentOS-Linux-BaseOS.repo
sed -e 's|^mirrorlist=|#mirrorlist=|g' -e '/#baseurl=/d' -e '/#mirrorlist/a\baseurl=https://mirrors.aliyun.com/centos/$releasever-stream/AppStream/$basearch/os/\n http://mirrors.aliyuncs.com/centos/$releasever-stream/AppStream/$basearch/os/\n http://mirrors.cloud.aliyuncs.com/centos/$releasever-stream/AppStream/$basearch/os/' -i /etc/yum.repos.d/CentOS-Linux-AppStream.repo
sed -e 's|^mirrorlist=|#mirrorlist=|g' -e '/#baseurl=/d' -e '/#mirrorlist/a\baseurl=https://mirrors.aliyun.com/centos/$releasever-stream/extras/$basearch/os/\n http://mirrors.aliyuncs.com/centos/$releasever-stream/extras/$basearch/os/\n http://mirrors.cloud.aliyuncs.com/centos/$releasever-stream/extras/$basearch/os/' -i /etc/yum.repos.d/CentOS-Linux-Extras.repo
yum clean all && yum makecache1.4.安装一些必备工具 (all)
yum -y install vim wget jq psmisc vim net-tools nfs-utils telnet yum-utils device-mapper-persistent-data lvm2 git network-scripts tar curl -y1.5.下载需要工具
mkdir /opt/src/;cd /opt/src/
1.下载kubernetes1.24.+的二进制包 (在master01，通过master01发送给各个k8s节点)
github二进制包下载地址：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.mdwget https://dl.k8s.io/v1.24.0/kubernetes-server-linux-amd64.tar.gz
2.下载etcdctl二进制包 (etcd节点，此文章etcd部署在k8s-master01、k8s-master02、k8s-master03)
github二进制包下载地址：https://github.com/etcd-io/etcd/releaseswget https://github.com/etcd-io/etcd/releases/download/v3.5.4/etcd-v3.5.4-linux-amd64.tar.gz
3.docker-ce二进制包下载地址 (k8s_all)
二进制包下载地址：https://download.docker.com/linux/static/stable/x86_64/这里需要下载20.10.+版本
wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.14.tgz
4.containerd二进制包下载 (k8s_all)
github下载地址：https://github.com/containerd/containerd/releasescontainerd下载时下载带cni插件的二进制包。
wget https://github.com/containerd/containerd/releases/download/v1.6.4/cri-containerd-cni-1.6.4-linux-amd64.tar.gz
5.下载cfssl二进制包 (master01，通过master01制作证书发送给各个k8s节点)
github二进制包下载地址：https://github.com/cloudflare/cfssl/releaseswget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd646.cni插件下载 (k8s_all)
github下载地址：https://github.com/containernetworking/plugins/releaseswget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
7.crictl客户端二进制下载 (k8s_all)
github下载：https://github.com/kubernetes-sigs/cri-tools/releaseswget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.23.0/crictl-v1.23.0-linux-amd64.tar.gz
1.6.关闭防火墙、SELinux (all)
systemctl disable --now firewalld
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config关闭防火墙、selinux写成一句话：
systemctl stop firewalld.service;systemctl disable firewalld.service;sed -i -r 's#(^SELIN.*=)enforcing#\1disable#g' /etc/selinux/config;setenforce 0
1.7.关闭交换分区 (all)
sed -ri 's/.*swap.*/#&/' /etc/fstab
swapoff -a && sysctl -w vm.swappiness=0
cat /etc/fstab
# /dev/mapper/centos-swap swap swap defaults 0 01.8.关闭NetworkManager 并启用 network (k8s_all、Lb可不做)
systemctl disable --now NetworkManager
systemctl start network && systemctl enable network1.9.进行时间同步 (Lb可不做)
# 找一台机器做服务端、或者选择matser01也可，让作为服务端pool更新阿里云时间
yum install chrony -y
cat > /etc/chrony.conf << EOF
pool ntp.aliyun.com iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
allow 10.4.7.0/24
local stratum 10
keyfile /etc/chrony.keys
leapsectz right/UTC
logdir /var/log/chrony
EOF
systemctl restart chronyd
systemctl enable chronyd# 客户端，要保证所k8s-master、k8s-node的时间都一致
yum install chrony -y
vim /etc/chrony.conf
cat /etc/chrony.conf | grep -v "^#" | grep -v "^$"
pool 10.4.7.11 iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
keyfile /etc/chrony.keys
leapsectz right/UTC
logdir /var/log/chrony
systemctl restart chronyd ; systemctl enable chronyd
# 客户端安装一条命令
yum install chrony -y ; sed -i "s#2.centos.pool.ntp.org#10.4.7.11#g" /etc/chrony.conf ; systemctl restart chronyd ; systemctl enable chronyd
#使用客户端进行验证
chronyc sources -v1.10.配置ulimit (k8s_all、Lb可不做)
ulimit -SHn 655350
cat >> /etc/security/limits.conf <<EOF
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* seft memlock unlimited
* hard memlock unlimitedd
EOF1.11.配置免密登录 (master01节点)
在master01节点做ssh密钥对，拷贝ssh_pub到所有节点，其中SSHPASS为各个节点的统一密码，StrictHostKeyChecking=no取消交互式询问（也就是yes\no）的提示，sshpass进行自动登录分发密钥
yum install -y sshpass
ssh-keygen -f /root/.ssh/id_rsa -P ''
export IP="10.4.7.12 10.4.7.21 10.4.7.22 10.4.7.200"
export SSHPASS=123456
for HOST in $IP;do sshpass -e ssh-copy-id -o StrictHostKeyChecking=no $HOST;done1.12.添加启用源 (Lb可不做)
# 为 RHEL-8或 CentOS-8配置源
yum install -y https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
# 查看可用安装包
yum --disablerepo="*" --enablerepo="elrepo-kernel" list available1.13.升级内核至4.18版本以上 (Lb可不做)
# 查看当前使用的内核
[root@localhost ~]# grubby --default-kernel
/boot/vmlinuz-4.18.0-348.el8.x86_64
# 查看已安装那些内核
[root@localhost ~]# rpm -qa | grep kernel
kernel-modules-4.18.0-348.el8.x86_64
kernel-tools-4.18.0-348.el8.x86_64
kernel-4.18.0-348.el8.x86_64
kernel-core-4.18.0-348.el8.x86_64
kernel-tools-libs-4.18.0-348.el8.x86_64
[root@localhost yum.repos.d]#
# 安装最新的内核
# 我这里选择的是稳定版kernel-ml 如需更新长期维护版本kernel-lt
[root@localhost ~]# yum --enablerepo=elrepo-kernel install kernel-ml -y
# 查看已安装那些内核
[root@localhost ~]# rpm -qa | grep kernel
kernel-modules-4.18.0-348.el8.x86_64
kernel-tools-4.18.0-348.el8.x86_64
kernel-4.18.0-348.el8.x86_64
kernel-ml-core-5.18.3-1.el8.elrepo.x86_64
kernel-ml-5.18.3-1.el8.elrepo.x86_64 （此内核为刚刚yum安装的）
kernel-core-4.18.0-348.el8.x86_64
kernel-tools-libs-4.18.0-348.el8.x86_64
kernel-ml-modules-5.18.3-1.el8.elrepo.x86_64
# 查看默认内核
grubby --default-kernel
/boot/vmlinuz-5.16.7-1.el8.elrepo.x86_64
# 若不是最新的使用命令设置
grubby --set-default /boot/vmlinuz-「您的内核版本」.x86_64
grubby --set-default "/boot/vmlinuz-5.18.3-1.el8.elrepo.x86_64"
# 重启生效
reboot
1.14.安装ipvsadm (Lb可不做)
yum install ipvsadm ipset sysstat conntrack libseccomp -y
cat >> /etc/modules-load.d/ipvs.conf <<EOF
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF
systemctl restart systemd-modules-load.service
lsmod | grep -e ip_vs -e nf_conntrack
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs 180224 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack 176128 1 ip_vs
nf_defrag_ipv6 24576 2 nf_conntrack,ip_vs
nf_defrag_ipv4 16384 1 nf_conntrack
libcrc32c 16384 3 nf_conntrack,xfs,ip_vs1.15.修改内核参数 (Lb可不做)
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.all.forwarding = 1
EOF
sysctl --system1.16.所有节点搭建配置域名解析 (all)
域名解析的作用：
1、让各个pc机器能够互相通过域名访问彼此
2、外网访问一个网址，将流量通过域名解析抛给到apiserver，在通过ingress控制器转发到pod中，实现外网能够通过访问网址进行访问pod
[root@k8s-master01 ~]# yum install bind bind9.16-utils -y
[root@k8s-master01 ~]# vi /etc/named.conf # 修改options下的内容，注释去掉options { listen-on port 53 { 10.4.7.11; }; // 默认监听53端口，IP改成自己的本机的IP，要让内网的所有机器能够访问我的bind进行 dns解析，127.0.0.1只能自己用，注意bind语法严格，所以格式要写对 listen-on-v6 port 53 { ::1; }; //去掉这行，不需要IPV6 directory "/var/named"; dump-file "/var/named/data/cache_dump.db"; statistics-file "/var/named/data/named_stats.txt"; memstatistics-file "/var/named/data/named_mem_stats.txt"; recursing-file "/var/named/data/named.recursing"; secroots-file "/var/named/data/named.secroots"; allow-query { any; }; // 改成any，让任何人都可以访问你 forwarders { 10.4.7.254; }; // 增加一条，如果dns解析不了，王上一级dns，也就是网关 recursion yes; //递归模式 dnssec-enable no; //是否支持DNSSEC开关,默认为yes。 dnssec-validation no; //是否进行DNSSEC确认开关,默认为no。 dnssec-enable配置项用来设置是否启用DNSSEC支持，DNS安全扩展（DNSSEC）提供了验证DNS数据有效性的系统。[root@k8s-master01 ~]# named-checkconf # 检查是否有问题。无任何提示代表无问题
[root@k8s-master01 ~]# vi /etc/named.rfc1912.zones # 配置区域文件，在最末尾增加两条，注释去掉
# 主机域，在内部自己使用，比如物理机(10.4.7.11 ping 10.4.7.11） # 可以使用ping hdss7-12.host.com，所以host是可以变化的 zone "host.com" IN { type master; file "host.com.zone"; allow-update { 10.4.7.11; }; }; # 业务域，把流量引入到k8s中，比如访问jenkins，页面地址输入jenkins.od.com # 所以od是可以变化的，按照公司要求 zone "od.com" IN { type master; file "od.com.zone"; allow-update { 10.4.7.11; }; };
编写主机域的配置文件，需要域名解析的主机都写入才可通过域名访问：
[root@k8s-master01 ~]# vi /var/named/host.com.zone$ORIGIN host.com. $TTL 600 ; 10 minutes @ IN SOA dns.host.com. dnsadmin.host.com. ( 2020010501 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.host.com. $TTL 60 ; 1 minute dns A 10.4.7.11 k8s-master01 A 10.4.7.11 k8s-master02 A 10.4.7.12 k8s-master03 A 10.4.7.21 k8s-node01 A 10.4.7.22 k8s-node02 A 10.4.7.200 lb-vip A 10.4.7.10
业务域配置文件，流量经过bind，通过业务域配置文件中对应的关系，解析出对应的IP：
[root@k8s-master01 ~]# vi /var/named/od.com.zone$ORIGIN od.com. $TTL 600 ; 10 minutes @ IN SOA dns.od.com. dnsadmin.od.com. ( 2020010501 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.od.com. $TTL 60 ; 1 minute dns A 10.4.7.11[root@k8s-master01 ~]# named-checkconf 检查是否有问题。无任何提示代表无问题
启动named服务:
[root@k8s-master01 ~]# systemctl start named
[root@k8s-master01 ~]# systemctl enable named
测试:
[root@k8s-master01 ~]# dig -t A k8s-master02.host.com @10.4.7.11 +short # 代表主机域无问题
10.4.7.12
对所有的机器的网卡，配置DNS为10.4.7.11：
涉及机器：（hdss7-11、hdss7-12、hdss7-21、hdss7-22、hdss7-200）
vi /etc/sysconfig/network-scripts/ifcfg-ens33 |grep "DNS1"
DNS1=10.4.7.11
重启网卡：systemctl restart network
测试：ping k8s-master02.host.com 如果可以说明已经生效
[root@k8s-master01 ~]# ping k8s-master02.host.com
PING k8s-master02.host.com (10.4.7.12) 56(84) bytes of data.
64 bytes from 10.4.7.12 (10.4.7.12): icmp_seq=1 ttl=64 time=0.503 ms
64 bytes from 10.4.7.12 (10.4.7.12): icmp_seq=2 ttl=64 time=0.709 ms
实现短域名（不带host.com）
[root@k8s-master01 ~]# ping k8s-master02
ping: k8s-master02: 未知的名称或服务
涉及机器：（hdss7-11、hdss7-12、hdss7-21、hdss7-22、hdss7-200）
vi /etc/resolv.conf # 添加search host.com，比如hdss7-11的esolv.conf
# Generated by NetworkManager
search host.com
nameserver 10.4.7.11测试：ping hdss7-12 等等
[root@k8s-master01 ~]# ping k8s-master02
PING k8s-master02.host.com (10.4.7.12) 56(84) bytes of data.
64 bytes from 10.4.7.12 (10.4.7.12): icmp_seq=1 ttl=64 time=0.666 ms
64 bytes from 10.4.7.12 (10.4.7.12): icmp_seq=2 ttl=64 time=0.801 ms
问题：重启后，resolv.conf会恢复源文件，导致search host.com丢失
涉及机器：（hdss7-11、hdss7-12、hdss7-21、hdss7-22、hdss7-200）
[root@k8s-master01 ~]# vi /etc/rc.d/rc.local # 开机向resolv.conf 插入短域名，注意脚本提权
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# 请注意，您必须运行'chmod+x/etc/rc.d/rc.local“以确保
sed -i "2isearch host.com" /etc/resolv.conf [root@k8s-master01 ~]# chmod +x /etc/rc.d/rc.local
写成一句话：sed -i '$a\sed -i "2isearch host.com" /etc/resolv.conf' /etc/rc.d/rc.local;chmod +x /etc/rc.d/rc.local
2.k8s基本组件安装
2.1.所有k8s节点安装Containerd作为Runtime (k8s_all)
cd /opt/src
#创建cni插件所需目录
mkdir -p /etc/cni/net.d /opt/cni/bin
#解压cni二进制包
tar xf cni-plugins-linux-amd64-v1.1.1.tgz -C /opt/cni/bin/
#解压
tar -C / -xzf cri-containerd-cni-1.6.4-linux-amd64.tar.gz
#创建服务启动文件
cat > /etc/systemd/system/containerd.service <<EOF
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target
[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target
EOF2.1.1配置Containerd所需的模块
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF2.1.2加载模块
systemctl restart systemd-modules-load.service2.1.3配置Containerd所需的内核
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
# 加载内核
sysctl --system2.1.4创建Containerd的配置文件
mkdir -p /etc/containerd
containerd config default | tee /etc/containerd/config.toml
修改Containerd的配置文件
sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml
# 找到containerd.runtimes.runc.options，要存在SystemdCgroup = true
[root@k8s-master01 src]# grep -B 13 -A 2 "SystemdCgroup" /etc/containerd/config.toml
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
BinaryName = ""
CriuImagePath = ""
CriuPath = ""
CriuWorkPath = ""
IoGid = 0
IoUid = 0
NoNewKeyring = false
NoPivotRoot = false
Root = ""
ShimCgroup = ""
SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
[root@k8s-master01 src]#
# 将/etc/containerd/config.toml 的sandbox_image默认地址改为符合版本地址
[root@k8s-master01 src]# grep -C 2 "sandbox_image" /etc/containerd/config.toml
netns_mounts_under_state_dir = false
restrict_oom_score_adj = false
# sandbox_image = "k8s.gcr.io/pause:3.6"
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/chenby/pause:3.6"
selinux_category_range = 1024
stats_collect_period = 102.1.5启动并设置为开机启动
systemctl daemon-reload
systemctl enable --now containerd2.1.6配置crictl客户端连接的运行时位置
#解压
tar xf crictl-v1.23.0-linux-amd64.tar.gz -C /usr/bin/
#生成配置文件
cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF
#测试
systemctl restart containerd
crictl info
2.2.k8s与etcd下载及安装（仅在master01操作）
2.2.1解压k8s安装包
# 解压k8s安装文件
cd /opt/src
tar -xf kubernetes-server-linux-amd64.tar.gz --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}
# 解压etcd安装文件
tar -xf etcd-v3.5.4-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.5.4-linux-amd64/etcd{,ctl}
# 查看/usr/local/bin下内容
[root@k8s-master01 src]# ls /usr/local/bin/
containerd containerd-shim-runc-v1 containerd-stress critest ctr etcdctl kube-controller-manager kubelet kube-scheduler
containerd-shim containerd-shim-runc-v2 crictl ctd-decoder etcd kube-apiserver kubectl kube-proxy2.2.2查看版本
[root@k8s-master01 src]# kubelet --version
Kubernetes v1.24.0
[root@k8s-master01 src]# etcdctl version
etcdctl version: 3.5.4
API version: 3.5
[root@k8s-master01 src]# 2.2.3将组件发送至其他k8s节点
Master='k8s-master02 k8s-master03'
Work='k8s-node01 k8s-node02'
for NODE in $Master; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done
for NODE in $Work; do scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done
2.3创建证书相关文件 （仅在master01操作）
mkdir /root/pki;cd /root/pki
cat > admin-csr.json << EOF
{
"CN": "admin",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "system:masters",
"OU": "Kubernetes-manual"
}
]
}
EOF
cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "876000h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "876000h"
}
}
}
}
EOF
cat > etcd-ca-csr.json << EOF
{
"CN": "etcd",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "etcd",
"OU": "Etcd Security"
}
],
"ca": {
"expiry": "876000h"
}
}
EOF
cat > front-proxy-ca-csr.json << EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"ca": {
"expiry": "876000h"
}
}
EOF
cat > kubelet-csr.json << EOF
{
"CN": "system:node:$NODE",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "system:nodes",
"OU": "Kubernetes-manual"
}
]
}
EOF
cat > manager-csr.json << EOF
{
"CN": "system:kube-controller-manager",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "system:kube-controller-manager",
"OU": "Kubernetes-manual"
}
]
}
EOF
cat > apiserver-csr.json << EOF
{
"CN": "kube-apiserver",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "Kubernetes",
"OU": "Kubernetes-manual"
}
]
}
EOF
cat > ca-csr.json << EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "Kubernetes",
"OU": "Kubernetes-manual"
}
],
"ca": {
"expiry": "876000h"
}
}
EOF
cat > etcd-csr.json << EOF
{
"CN": "etcd",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "etcd",
"OU": "Etcd Security"
}
]
}
EOF
cat > front-proxy-client-csr.json << EOF
{
"CN": "front-proxy-client",
"key": {
"algo": "rsa",
"size": 2048
}
}
EOF
cat > kube-proxy-csr.json << EOF
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "system:kube-proxy",
"OU": "Kubernetes-manual"
}
]
}
EOF
cat > scheduler-csr.json << EOF
{
"CN": "system:kube-scheduler",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "system:kube-scheduler",
"OU": "Kubernetes-manual"
}
]
}
EOFmkdir /root/bootstrap;cd /root/bootstrap
cat > bootstrap.secret.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
name: bootstrap-token-c8ad9c
namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
description: "The default bootstrap token generated by 'kubelet '."
token-id: c8ad9c
token-secret: 2e4d610cf3e7426e
usage-bootstrap-authentication: "true"
usage-bootstrap-signing: "true"
auth-extra-groups: system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubelet-bootstrap
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: node-autoapprove-bootstrap
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: node-autoapprove-certificate-rotation
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kube-apiserver-to-kubelet
rules:
- apiGroups:
- ""
resources:
- nodes/proxy
- nodes/stats
- nodes/log
- nodes/spec
- nodes/metrics
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-apiserver
namespace: ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kube-apiserver
EOFmkdir /root/coredns;cd /root/coredns
cat > coredns.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: coredns
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:coredns
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:coredns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:coredns
subjects:
- kind: ServiceAccount
name: coredns
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/name: "CoreDNS"
spec:
# replicas: not specified here:
# 1. Default is 1.
# 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: kube-dns
template:
metadata:
labels:
k8s-app: kube-dns
spec:
priorityClassName: system-cluster-critical
serviceAccountName: coredns
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
nodeSelector:
kubernetes.io/os: linux
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: k8s-app
operator: In
values: ["kube-dns"]
topologyKey: kubernetes.io/hostname
containers:
- name: coredns
image: registry.cn-beijing.aliyuncs.com/dotbalo/coredns:1.8.6
imagePullPolicy: IfNotPresent
resources:
limits:
memory: 170Mi
requests:
cpu: 100m
memory: 70Mi
args: [ "-conf", "/etc/coredns/Corefile" ]
volumeMounts:
- name: config-volume
mountPath: /etc/coredns
readOnly: true
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- all
readOnlyRootFilesystem: true
livenessProbe:
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: /ready
port: 8181
scheme: HTTP
dnsPolicy: Default
volumes:
- name: config-volume
configMap:
name: coredns
items:
- key: Corefile
path: Corefile
---
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
annotations:
prometheus.io/port: "9153"
prometheus.io/scrape: "true"
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "CoreDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: 192.168.0.2
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
- name: metrics
port: 9153
protocol: TCP
EOFmkdir /root/metrics-server;cd /root/metrics-server
cat > metrics-server.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-view: "true"
name: system:aggregated-metrics-reader
rules:
- apiGroups:
- metrics.k8s.io
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- pods
- nodes
- nodes/stats
- namespaces
- configmaps
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
ports:
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: metrics-server
strategy:
rollingUpdate:
maxUnavailable: 0
template:
metadata:
labels:
k8s-app: metrics-server
spec:
containers:
- args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
- --kubelet-insecure-tls
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem # change to front-proxy-ca.crt for kubeadm
- --requestheader-username-headers=X-Remote-User
- --requestheader-group-headers=X-Remote-Group
- --requestheader-extra-headers-prefix=X-Remote-Extra-
image: registry.cn-beijing.aliyuncs.com/dotbalo/metrics-server:0.5.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /livez
port: https
scheme: HTTPS
periodSeconds: 10
name: metrics-server
ports:
- containerPort: 4443
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz
port: https
scheme: HTTPS
initialDelaySeconds: 20
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 200Mi
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- mountPath: /tmp
name: tmp-dir
- name: ca-ssl
mountPath: /etc/kubernetes/pki
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: metrics-server
volumes:
- emptyDir: {}
name: tmp-dir
- name: ca-ssl
hostPath:
path: /etc/kubernetes/pki
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
labels:
k8s-app: metrics-server
name: v1beta1.metrics.k8s.io
spec:
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100
EOF3.相关证书生成（仅在master01操作）
master01节点下载证书生成工具
[root@k8s-master01 bootstrap]# cd /opt/src/
[root@k8s-master01 src]# mv cfssl_1.6.1_linux_amd64 cfssl
[root@k8s-master01 src]# mv cfssl-certinfo_1.6.1_linux_amd64 cfssl-certinfo
[root@k8s-master01 src]# mv cfssljson_1.6.1_linux_amd64 cfssljson
[root@k8s-master01 src]# cp cfssl* /usr/local/bin/
[root@k8s-master01 src]# chmod +x /usr/local/bin/cfssl*
[root@k8s-master01 src]# ll /usr/local/bin/cfssl*
-rwxr-xr-x. 1 root root 16659824 6月 13 02:00 /usr/local/bin/cfssl
-rwxr-xr-x. 1 root root 13502544 6月 13 02:00 /usr/local/bin/cfssl-certinfo
-rwxr-xr-x. 1 root root 11029744 6月 13 02:00 /usr/local/bin/cfssljson
3.1.生成etcd证书
特别说明除外，以下操作在所有master节点操作
3.1.1所有master节点创建证书存放目录
mkdir /etc/etcd/ssl -p3.1.2master01节点生成etcd证书
cd /root/pki/
# 生成一个ca机构，给etcd认证
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca
# 生成etcd证书和etcd证书的key（如果你觉得以后可能会扩容，可以在ip那多写几个预留出来）
cfssl gencert \
-ca=/etc/etcd/ssl/etcd-ca.pem \
-ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
-config=ca-config.json \
-hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,10.4.7.11,10.4.7.12,10.4.7.21 \
-profile=kubernetes \
etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd3.1.3将证书复制到其他节点
Master='k8s-master02 k8s-master03'
for NODE in $Master; do ssh $NODE "mkdir -p /etc/etcd/ssl"; for FILE in etcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem; do scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}; done; done3.2.生成k8s相关证书
特别说明除外，以下操作在所有master节点操作
3.2.1所有k8s节点创建证书存放目录
mkdir -p /etc/kubernetes/pki3.2.2master01节点生成k8s证书
# 生成一个根证书
cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca
# 192.168.0.1是service网段的第一个地址，需要计算，10.4.7.10为高可用vip地址
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-hostname=192.168.0.1,10.4.7.10,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,10.4.7.11,10.4.7.12,10.4.7.21,10.4.7.22,10.4.7.200 \
-profile=kubernetes apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver3.2.3生成apiserver聚合证书
# 生成一个ca机构，给后续的证书认证
cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca
# 有一个警告，可以忽略
cfssl gencert \
-ca=/etc/kubernetes/pki/front-proxy-ca.pem \
-ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem \
-config=ca-config.json \
-profile=kubernetes front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client3.2.4生成controller-manage的证书
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager
# 自定义一个k8s用户，如下叫kubernetes，此用户绑定ca.pem根证书，传递给apiserver，这样此账户-证书-apiserver进行了关联
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:8443 \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
# 定义一个用户账号（system:kube-controller-manager），用这个用户账号，跟上述创建的自定义一个k8s用户kubernetes进行绑定，
用于使用提供的认证信息和命名空间将请求发送到指定的集群。给这个绑定起个名字system:kube-controller-manager@kubernetes
kubectl config set-context system:kube-controller-manager@kubernetes \
--cluster=kubernetes \
--user=system:kube-controller-manager \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
# 定义用于向 k8s 集群进行身份验证的客户端凭据。将客户端证书controller-manager.pem根controller-manager-key.pem
服务端证书上传进去，用于账户（system:kube-controller-manager）申请的验证
kubectl config set-credentials system:kube-controller-manager \
--client-certificate=/etc/kubernetes/pki/controller-manager.pem \
--client-key=/etc/kubernetes/pki/controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
# 把这写信息作为承载式文件
kubectl config use-context system:kube-controller-manager@kubernetes \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
3.2.5生成scheduler的证书
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:8443 \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig
kubectl config set-credentials system:kube-scheduler \
--client-certificate=/etc/kubernetes/pki/scheduler.pem \
--client-key=/etc/kubernetes/pki/scheduler-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig
kubectl config set-context system:kube-scheduler@kubernetes \
--cluster=kubernetes \
--user=system:kube-scheduler \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig
kubectl config use-context system:kube-scheduler@kubernetes \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig
3.2.6生成kube-proxy的证书
cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-profile=kubernetes \
admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:8443 \
--kubeconfig=/etc/kubernetes/admin.kubeconfig
kubectl config set-credentials kubernetes-admin \
--client-certificate=/etc/kubernetes/pki/admin.pem \
--client-key=/etc/kubernetes/pki/admin-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/admin.kubeconfig
kubectl config set-context kubernetes-admin@kubernetes \
--cluster=kubernetes \
--user=kubernetes-admin \
--kubeconfig=/etc/kubernetes/admin.kubeconfig
kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=/etc/kubernetes/admin.kubeconfig3.2.7创建ServiceAccount Key ——secret
openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub3.2.8将证书发送到其他master节点
for NODE in k8s-master02 k8s-master03; do ssh $NODE "mkdir -p /etc/kubernetes/pki"; for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done; for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}; done; done
3.2.9查看证书（k8s_all）
ls /etc/kubernetes/pki/
admin.csr apiserver-key.pem ca.pem front-proxy-ca.csr front-proxy-client-key.pem scheduler.csr
admin-key.pem apiserver.pem controller-manager.csr front-proxy-ca-key.pem front-proxy-client.pem scheduler-key.pem
admin.pem ca.csr controller-manager-key.pem front-proxy-ca.pem sa.key scheduler.pem
apiserver.csr ca-key.pem controller-manager.pem front-proxy-client.csr sa.pub
# 一共23个就对了
ls /etc/kubernetes/pki/ |wc -l
234.k8s系统组件配置
4.1 安装rsyslog
将 etcd 启动的命令、参数， 通过systemctl 进行启动，但 systemctl 启动后，日志输出怎么办？
systemctl 测试过用 StandardOutput=/tmp/services/logs/xxxx.log 根本无法输出，所以我们将systemctl 启动的程序输出的日志，通过rsyslogs转发写到固定的日志中
yum install -y rsyslogrsyslogs具体如何操作？
一、在自定义test.service文件中[Service]增加
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=your program identifier
二、编辑/etc/rsyslog.d/you_file.conf文件，其内容为
if $programname == 'your program identifier' then {
/data/logs/kubernetes/kube-kubelet/kubele.stdout.log
}
三、重启rsyslog
sudo systemctl restart rsyslog
您的程序stdout / stderr仍可通过journalctl(sudo journalctl -u your program identifier获得，但它们也将在您选择的文件中提供。
四、重新你的服务
systemctl restart test
五、查看/var/log/file.log 文件内容4.2.etcd配置
4.2.1master01配置
# 如果要用IPv6那么把IPv4地址修改为IPv6即可
cat > /etc/etcd/etcd.config.yml << EOF
name: 'k8s-master01'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://10.4.7.11:2380'
listen-client-urls: 'https://10.4.7.11:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://10.4.7.11:2380'
advertise-client-urls: 'https://10.4.7.11:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.4.7.11:2380,k8s-master02=https://10.4.7.12:2380,k8s-master03=https://10.4.7.21:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
auto-tls: true
peer-transport-security:
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
peer-client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF4.2.2master02配置
# 如果要用IPv6那么把IPv4地址修改为IPv6即可
cat > /etc/etcd/etcd.config.yml << EOF
name: 'k8s-master02'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://10.4.7.12:2380'
listen-client-urls: 'https://10.4.7.12:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://10.4.7.12:2380'
advertise-client-urls: 'https://10.4.7.12:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.4.7.11:2380,k8s-master02=https://10.4.7.12:2380,k8s-master03=https://10.4.7.21:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
auto-tls: true
peer-transport-security:
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
peer-client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF4.2.3master03配置
# 如果要用IPv6那么把IPv4地址修改为IPv6即可
cat > /etc/etcd/etcd.config.yml << EOF
name: 'k8s-master03'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://10.4.7.21:2380'
listen-client-urls: 'https://10.4.7.21:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://10.4.7.21:2380'
advertise-client-urls: 'https://10.4.7.21:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.4.7.11:2380,k8s-master02=https://10.4.7.12:2380,k8s-master03=https://10.4.7.21:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
auto-tls: true
peer-transport-security:
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
peer-client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF4.3.创建service（所有master节点操作）
4.3.1创建etcd.service并启动
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target
[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=etcd
[Install]
WantedBy=multi-user.target
Alias=etcd3.service
EOFvi /etc/rsyslog.d/etcd.conf
if $programname == 'etcd' then {
/data/logs/etcd/etcd.stdout.log
}4.3.2创建etcd证书目录、日志目录
mkdir /etc/kubernetes/pki/etcd
ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/
mkdir -p /data/logs/etcd/
touch /data/logs/etcd/etcd.stdout.log
sudo systemctl restart rsyslog
systemctl daemon-reload
systemctl enable --now etcd4.3.3查看etcd状态
如果检查后有报错，三台etcd重启后在查看
# 如果要用IPv6那么把IPv4地址修改为IPv6即可
export ETCDCTL_API=3
etcdctl --endpoints="10.4.7.11:2379,10.4.7.12:2379,10.4.7.21:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem endpoint status --write-out=table
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.0.0.83:2379 | c0c8142615b9523f | 3.5.4 | 20 kB | false | false | 2 | 9 | 9 | |
| 10.0.0.82:2379 | de8396604d2c160d | 3.5.4 | 20 kB | false | false | 2 | 9 | 9 | |
| 10.0.0.81:2379 | 33c9d6df0037ab97 | 3.5.4 | 20 kB | true | false | 2 | 9 | 9 | |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@k8s-master01 pki]# 4.3.4 etcd数据定时备份
mkdir -p /data/backup/etcd
vi /etc/etcd/backup.sh
#!/bin/sh
cd /var/lib
name="etcd-bak"`date "+%Y%m%d"`
tar -cvf "/data/backup/etcd/"$name".tar.gz" etcd
chmod 755 /etc/etcd/backup.sh
crontab -e
00 00 * * * /etc/etcd/backup.sh
或者
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
00 00 * * * (/path/to/backup.sh)
crontab -l
00 00 * * * /etc/etcd/backup.shsystemctl restart crond
5.高可用配置
5.1在lb01和lb02两台服务器上操作 (Lb)
5.1.1安装keepalived和haproxy服务
yum -y install keepalived haproxy5.1.2修改haproxy配置文件（两台配置文件一样）
# cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak
cat >/etc/haproxy/haproxy.cfg<<"EOF"
global
maxconn 2000
ulimit-n 16384
log 127.0.0.1 local0 err
stats timeout 30s
defaults
log global
mode http
option httplog
timeout connect 5000
timeout client 50000
timeout server 50000
timeout http-request 15s
timeout http-keep-alive 15s
frontend monitor-in
bind *:33305
mode http
option httplog
monitor-uri /monitor
frontend k8s-master
bind 0.0.0.0:8443
bind 127.0.0.1:8443
mode tcp
option tcplog
tcp-request inspect-delay 5s
default_backend k8s-master
backend k8s-master
mode tcp
option tcplog
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server k8s-master01 10.4.7.11:6443 check
server k8s-master02 10.4.7.12:6443 check
server k8s-master03 10.4.7.21:6443 check
EOF5.1.3lb01配置keepalived master节点
#cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state MASTER
interface eth1
mcast_src_ip 10.4.7.22
virtual_router_id 51
priority 100
nopreempt
advert_int 2
authentication {
auth_type PASS
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
10.4.7.10
}
track_script {
chk_apiserver
} }
EOF5.1.4lb02配置keepalived backup节点
# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state BACKUP
interface eth1
mcast_src_ip 10.4.7.200
virtual_router_id 51
priority 50
nopreempt
advert_int 2
authentication {
auth_type PASS
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
10.4.7.10
}
track_script {
chk_apiserver
} }
EOF5.1.5健康检查脚本配置（两台lb主机）
cat > /etc/keepalived/check_apiserver.sh << EOF
#!/bin/bash
err=0
for k in \$(seq 1 3)
do
check_code=\$(pgrep haproxy)
if [[ \$check_code == "" ]]; then
err=\$(expr \$err + 1)
sleep 1
continue
else
err=0
break
fi
done
if [[ \$err != "0" ]]; then
echo "systemctl stop keepalived"
/usr/bin/systemctl stop keepalived
exit 1
else
exit 0
fi
EOF
# 给脚本授权
chmod +x /etc/keepalived/check_apiserver.sh5.1.6启动服务
systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived5.1.7测试高可用
# 能ping同
[root@k8s-node02 ~]# ping 10.4.7.10
# 能telnet访问
[root@k8s-node02 ~]# telnet 10.4.7.10 8443
# 关闭主节点，看vip是否漂移到备节点6.k8s组件配置（区别于第4点）
所有k8s节点创建以下目录
mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes6.1.部署 supervisord 工具
使用supervisord 工具代替systemctl
yum install -y https://mirrors.aliyun.com/epel/epel-release-latest-8.noarch.rpm
yum clean all && yum makecache
yum install -y supervisor
启动supervisor
sudo supervisord -c /etc/supervisord.conf
sudo supervisorctl -c /etc/supervisord.conf
写入启动脚本中
echo -e "supervisord -c /etc/supervisord.conf\nsupervisorctl -c /etc/supervisord.conf" >> /etc/rc.d/rc.local6.2.创建apiserver（所有master节点）
6.2.1master01节点配置
mkdir -p /opt/kubernetes/
mkdir -p /data/logs/kubernetes/kube-apiserver/
touch /data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
vim /opt/kubernetes/kube-apiserver-startup.sh # 注意vim后:set paste
#!/bin/bash
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit
/usr/local/bin/kube-apiserver \
--v=2 \
--logtostderr=true \
--allow-privileged=true \
--bind-address=0.0.0.0 \
--secure-port=6443 \
--advertise-address=10.4.7.11 \
--service-cluster-ip-range=192.168.0.0/16 \
--feature-gates=IPv6DualStack=true \
--service-node-port-range=30000-32767 \
--etcd-servers=https://10.4.7.11:2379,https://10.4.7.12:2379,https://10.4.7.21:2379 \
--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
--etcd-certfile=/etc/etcd/ssl/etcd.pem \
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
--client-ca-file=/etc/kubernetes/pki/ca.pem \
--tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
--service-account-key-file=/etc/kubernetes/pki/sa.pub \
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
--authorization-mode=Node,RBAC \
--enable-bootstrap-token-auth=true \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
--requestheader-allowed-names=aggregator \
--requestheader-group-headers=X-Remote-Group \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-username-headers=X-Remote-User \
--enable-aggregator-routing=true
# --token-auth-file=/etc/kubernetes/token.csvchmod +x /opt/kubernetes/kube-apiserver-startup.sh
vim /etc/supervisord.d/kube-apiserver.ini # apiserver 使用 supervisord 启动的配置文件
[program:kube-apiserver-7-11]
command=/opt/kubernetes/kube-apiserver-startup.sh
numprocs=1
directory=/opt/kubernetes/
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false6.2.2master02节点配置
mkdir -p /opt/kubernetes/
mkdir -p /data/logs/kubernetes/kube-apiserver/
touch /data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
vim /opt/kubernetes/kube-apiserver-startup.sh # 注意vim后:set paste
#!/bin/bash
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit
/usr/local/bin/kube-apiserver \
--v=2 \
--logtostderr=true \
--allow-privileged=true \
--bind-address=0.0.0.0 \
--secure-port=6443 \
--advertise-address=10.4.7.12 \
--service-cluster-ip-range=192.168.0.0/16 \
--feature-gates=IPv6DualStack=true \
--service-node-port-range=30000-32767 \
--etcd-servers=https://10.4.7.11:2379,https://10.4.7.12:2379,https://10.4.7.21:2379 \
--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
--etcd-certfile=/etc/etcd/ssl/etcd.pem \
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
--client-ca-file=/etc/kubernetes/pki/ca.pem \
--tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
--service-account-key-file=/etc/kubernetes/pki/sa.pub \
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
--authorization-mode=Node,RBAC \
--enable-bootstrap-token-auth=true \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
--requestheader-allowed-names=aggregator \
--requestheader-group-headers=X-Remote-Group \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-username-headers=X-Remote-User \
--enable-aggregator-routing=true
# --token-auth-file=/etc/kubernetes/token.csvchmod +x /opt/kubernetes/kube-apiserver-startup.sh
vim /etc/supervisord.d/kube-apiserver.ini # apiserver 使用 supervisord 启动的配置文件
[program:kube-apiserver-7-12]
command=/opt/kubernetes/kube-apiserver-startup.sh
numprocs=1
directory=/opt/kubernetes/
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false6.2.3master03节点配置
mkdir -p /opt/kubernetes/
mkdir -p /data/logs/kubernetes/kube-apiserver/
touch /data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
vim /opt/kubernetes/kube-apiserver-startup.sh # 注意vim后:set paste
#!/bin/bash
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit
/usr/local/bin/kube-apiserver \
--v=2 \
--logtostderr=true \
--allow-privileged=true \
--bind-address=0.0.0.0 \
--secure-port=6443 \
--advertise-address=10.4.7.12 \
--service-cluster-ip-range=192.168.0.0/16 \
--feature-gates=IPv6DualStack=true \
--service-node-port-range=30000-32767 \
--etcd-servers=https://10.4.7.11:2379,https://10.4.7.12:2379,https://10.4.7.21:2379 \
--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
--etcd-certfile=/etc/etcd/ssl/etcd.pem \
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
--client-ca-file=/etc/kubernetes/pki/ca.pem \
--tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
--service-account-key-file=/etc/kubernetes/pki/sa.pub \
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
--authorization-mode=Node,RBAC \
--enable-bootstrap-token-auth=true \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
--requestheader-allowed-names=aggregator \
--requestheader-group-headers=X-Remote-Group \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-username-headers=X-Remote-User \
--enable-aggregator-routing=truechmod +x /opt/kubernetes/kube-apiserver-startup.sh
vim /etc/supervisord.d/kube-apiserver.ini # apiserver 使用 supervisord 启动的配置文件
[program:kube-apiserver-7-21]
command=/opt/kubernetes/kube-apiserver-startup.sh
numprocs=1
directory=/opt/kubernetes/
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false6.2.4启动apiserver（所有master节点）
[root@k8s-master01 ~]# supervisorctl update
[root@k8s-master02 ~]# supervisorctl update
[root@k8s-master03 ~]# supervisorctl update
[root@k8s-master01 ~]# supervisorctl status
kube-apiserver-7-21 RUNNING pid 1421, uptime 16:33:20
[root@k8s-master02 ~]# supervisorctl status
kube-apiserver-7-12 RUNNING pid 1989, uptime 0:36:19
[root@k8s-master03 ~]# supervisorctl status
kube-apiserver-7-21 RUNNING pid 1989, uptime 0:36:11使用supervisorctl update后报错
error: <class 'FileNotFoundError'>, [Errno 2] No such file or directory: file: /usr/lib/python3.6/site-packages/supervisor/xmlrpc.py line: 560没有使用如下命令启动
sudo supervisord -c /etc/supervisord.conf
sudo supervisorctl -c /etc/supervisord.conf
6.3.配置kube-controller-manager service
# 所有master节点配置，且配置相同
# 172.7.0.0/16 为pod网段，按需求设置你自己的网段
vim /opt/kubernetes/kube-controller-manager-startup.sh # 注意vim后:set paste
#!/bin/sh
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit
/usr/local/bin/kube-controller-manager \
--v=2 \
--logtostderr=true \
--bind-address=127.0.0.1 \
--root-ca-file=/etc/kubernetes/pki/ca.pem \
--cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
--service-account-private-key-file=/etc/kubernetes/pki/sa.key \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
--leader-elect=true \
--use-service-account-credentials=true \
--node-monitor-grace-period=40s \
--node-monitor-period=5s \
--pod-eviction-timeout=2m0s \
--controllers=*,bootstrapsigner,tokencleaner \
--allocate-node-cidrs=true \
--feature-gates=IPv6DualStack=true \
--service-cluster-ip-range=192.168.0.0/16 \
--cluster-cidr=172.7.0.0/16 \
--node-cidr-mask-size-ipv4=24 \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pemmkdir -p /data/logs/kubernetes/kube-controller-manager
chmod u+x /opt/kubernetes/kube-controller-manager-startup.sh
touch /data/logs/kubernetes/kube-controller-manager/controller.stdout.logvim /etc/supervisord.d/kube-controller-manager.ini # controller-manager 使用 supervisord 启动的配置文件
[program:kube-controller-manager-7-11]
command=/opt/kubernetes/kube-controller-manager-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/kubernetes/ ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-controller-manager/controller.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false6.3.1启动kube-controller-manager，并查看状态
[root@k8s-master01 ~]# supervisorctl update
[root@k8s-master02 ~]# supervisorctl update
[root@k8s-master03 ~]# supervisorctl update
[root@k8s-master01 ~]# supervisorctl status
kube-apiserver-7-11 RUNNING pid 2225, uptime 0:58:44
kube-controller-manager-7-11 RUNNING pid 2711, uptime 0:00:38
[root@k8s-master02 ~]# supervisorctl status
kube-apiserver-7-12 RUNNING pid 1989, uptime 1:00:42
kube-controller-manager-7-12 RUNNING pid 2092, uptime 0:02:06
[root@k8s-master03 ~]# supervisorctl status
kube-apiserver-7-21 RUNNING pid 1578, uptime 0:52:27
kube-controller-manager-7-21 RUNNING pid 1657, uptime 0:00:326.3.配置kube-scheduler service
6.3.1所有master节点配置，且配置相同
mkdir -p /data/logs/kubernetes/kube-scheduler
touch /data/logs/kubernetes/kube-scheduler/scheduler.stdout.logvim /opt/kubernetes/kube-scheduler-startup.sh # 注意vim后:set paste
#!/bin/sh
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit
/usr/local/bin/kube-scheduler \
--v=2 \
--logtostderr=true \
--bind-address=127.0.0.1 \
--leader-elect=true \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfigchmod u+x /opt/kubernetes/kube-scheduler-startup.sh
vim /etc/supervisord.d/kube-scheduler.ini # controller-scheduler 使用 supervisord 启动的配置文件
[program:kube-scheduler-7-11]
command=/opt/kubernetes/kube-scheduler-startup.sh
numprocs=1
directory=/opt/kubernetes/
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=4
stdout_capture_maxbytes=1MB
stdout_events_enabled=false6.3.2启动并查看服务状态
[root@k8s-master01 ~]# supervisorctl update
[root@k8s-master02 ~]# supervisorctl update
[root@k8s-master03 ~]# supervisorctl update
[root@k8s-master01 ~]# supervisorctl status
kube-apiserver-7-11 RUNNING pid 2225, uptime 2:38:21
kube-controller-manager-7-11 RUNNING pid 2711, uptime 1:40:15
kube-scheduler-7-11 RUNNING pid 2829, uptime 0:01:43
[root@k8s-master02 ~]# supervisorctl status
kube-apiserver-7-12 RUNNING pid 1989, uptime 2:35:56
kube-controller-manager-7-12 RUNNING pid 2092, uptime 1:37:20
kube-scheduler-7-12 RUNNING pid 2201, uptime 0:02:00
[root@k8s-master03 ~]# supervisorctl status
kube-apiserver-7-21 RUNNING pid 1578, uptime 2:27:49
kube-controller-manager-7-21 RUNNING pid 1657, uptime 1:35:54
kube-scheduler-7-21 RUNNING pid 1757, uptime 0:02:117.TLS Bootstrapping配置
7.1在master01上配置
cd /root/bootstrap
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true --server=https://10.4.7.10:8443 \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
kubectl config set-credentials tls-bootstrap-token-user \
--token=c8ad9c.2e4d610cf3e7426e \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
kubectl config set-context tls-bootstrap-token-user@kubernetes \
--cluster=kubernetes \
--user=tls-bootstrap-token-user \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
kubectl config use-context tls-bootstrap-token-user@kubernetes \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
# token的位置在bootstrap.secret.yaml，如果修改的话到这个文件修改
执行 kubectl get cs 发现会报错 The connection to the server localhost:8080 was refused - did you specify the right host or port?
原因是kubectl 默认使用/root/.kube/config文件中，配置的集群角色的账户，跟api通信，config文件其实就是我们上述配置的bootstrap-kubelet.kubeconfig
mkdir -p /root/.kube ; cp /etc/kubernetes/admin.kubeconfig /root/.kube/config[root@k8s-master01 ~]# cd /root/.kube/ [root@k8s-master01 .kube]# kubectl config view apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://10.4.7.10:8443 name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: REDACTED client-key-data: REDACTED [root@k8s-master01 .kube]#
7.2查看集群状态，没问题的话继续后续操作
[root@k8s-master01 .kube]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-1 Healthy {"health":"true","reason":""}
etcd-2 Healthy {"health":"true","reason":""}
etcd-0 Healthy {"health":"true","reason":""}
[root@k8s-master01 .kube]# cd /root/bootstrap/
[root@k8s-master01 bootstrap]# kubectl create -f bootstrap.secret.yaml
secret/bootstrap-token-c8ad9c created
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-certificate-rotation created
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created8.node节点配置
8.1.在master01上将证书复制到node节点
cd /etc/kubernetes/
for NODE in k8s-master02 k8s-master03 k8s-node01 k8s-node02 ; do ssh $NODE mkdir -p /etc/kubernetes/pki; for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}; done; done8.2.kubelet配置
8.2.1所有k8s节点创建相关目录
mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/vi /usr/lib/systemd/system/kubelet.service # 所有k8s节点配置kubelet service，set paste
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service
[Service]
ExecStart=/usr/local/bin/kubelet \
--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
--config=/etc/kubernetes/kubelet-conf.yml \
--container-runtime=remote \
--runtime-request-timeout=15m \
--container-runtime-endpoint=unix:///run/containerd/containerd.sock \
--cgroup-driver=systemd \
--node-labels=node.kubernetes.io/node=''
# --feature-gates=IPv6DualStack=true
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=kubelet
[Install]
WantedBy=multi-user.target
vi /etc/rsyslog.d/kubelet.conf
if $programname == 'kubelet' then {
/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
}mkdir -p /data/logs/kubernetes/kube-kubelet/
touch /data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
sudo systemctl restart rsyslog8.2.2所有k8s节点创建kubelet的配置文件
vi /etc/kubernetes/kubelet-conf.yml # set paste
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 192.168.0.2
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s8.2.3启动kubelet
systemctl daemon-reload
systemctl restart kubelet
systemctl enable --now kubelet8.2.4查看集群
[root@k8s-master01 rsyslog.d]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master01 NotRead <none> 108m v1.24.0
k8s-master02 NotRead <none> 10m v1.24.0
k8s-master03 NotRead <none> 7m5s v1.24.0
k8s-node01 NotRead <none> 4m8s v1.24.0
k8s-node02 NotRead <none> 11s v1.24.08.3.kube-proxy配置
8.3.1此配置只在master01操作
cp /etc/kubernetes/admin.kubeconfig /etc/kubernetes/kube-proxy.kubeconfig
8.3.2将kubeconfig发送至其他节点
for NODE in k8s-master02 k8s-master03; do scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig; done
for NODE in k8s-node01 k8s-node02 ; do scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig; done8.3.3所有k8s节点添加kube-proxy的配置和service文件
vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-proxy \
--config=/etc/kubernetes/kube-proxy.yaml \
--v=2
Restart=always
RestartSec=10s
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=kube-proxy
[Install]
WantedBy=multi-user.targetvi /etc/rsyslog.d/kube-proxy.conf
if $programname == 'kube-proxy' then {
/data/logs/kubernetes/kube-proxy/kube-proxy.stdout.log
}mkdir -p /data/logs/kubernetes/kube-proxy/
touch /data/logs/kubernetes/kube-proxy/kube-proxy.stdout.log
sudo systemctl restart rsyslogvi /etc/kubernetes/kube-proxy.yaml # set paste
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
acceptContentTypes: ""
burst: 10
contentType: application/vnd.kubernetes.protobuf
kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
qps: 5
clusterCIDR: 172.7.0.0/16
configSyncPeriod: 15m0s
conntrack:
max: null
maxPerCore: 32768
min: 131072
tcpCloseWaitTimeout: 1h0m0s
tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
masqueradeAll: false
masqueradeBit: 14
minSyncPeriod: 0s
syncPeriod: 30s
ipvs:
masqueradeAll: true
minSyncPeriod: 5s
scheduler: "rr"
syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms8.3.4启动kube-proxy
systemctl daemon-reload
systemctl restart kube-proxy
systemctl enable --now kube-proxy9. 部署件Calico
以下步骤只在master01操作
CNI网络插件最主要的功能就是实现POD资源能够跨宿主机进行通信
常见的CNI网络插件:
Flannel、Calico、Canal、Contiv、OpenContrail、NSx-T、Kube-router
9.1 Calico的讲解
https://blog.csdn.net/Jerry00713/article/details/125429199
9.2 部署Calico
为了贴合生产，我们直接模拟node节点是大于50个
下载 Kubernetes API 数据存储的 Calico 网络清单。
cd /root/ curl https://projectcalico.docs.tigera.io/manifests/calico-typha.yaml -o calico.yamlvi calico.yaml # 将CALICO_IPV4POOL_CIDR 跟对应的 Value 注释去掉，并修改cluster-cidr 值
vi calico.yaml # 在Calico的讲解中提到过，大于50node节点，名为calico-typha的Deployment资源的副本数最好为大于3，为了节省资源，使用1
最后使用以下命令应用清单，使其以 pod 的形式运行。
kubectl apply -f calico.yaml
9.3 查看容器状态
一开始会显示如下，在创建中
等待很长一段时间，等待都running，其中会出现
[root@k8s-master01 ~]# kubectl get pod -A
Unable to connect to the server: net/http: TLS handshake timeout
[root@k8s-master01 ~]# kubectl get pod -A
Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get pods)
[root@k8s-master01 ~]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6766647d54-46mpn 0/1 ContainerCreating 0 11s
kube-system calico-node-9dzc7 0/1 Init:0/2 0 12s
kube-system calico-node-b9jbw 0/1 Init:0/2 0 12s
kube-system calico-node-dhq2c 0/1 Init:0/2 0 12s
kube-system calico-node-l9hwj 0/1 Init:0/2 0 12s
kube-system calico-node-zm9n7 0/1 Init:0/2 0 12s
kube-system calico-typha-6c589979d4-rr8cn 0/1 ContainerCreating 0 12s
[root@k8s-master01 ~]# 会发现其实Calico 启动一个calico-kube-controllers 的pod。启动一个calico-typha的pod，这是由于我们设置的副本数为1。最后是每个节点启动一个pod，也说明了Calico把每个节点当作一个路由进行2层数据的转发
[root@k8s-master01 ~]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-56cdb7c587-8stfx 1/1 Running 6 (10m ago) 39m
kube-system calico-node-62ndd 1/1 Running 1 (9m18s ago) 39m
kube-system calico-node-ch624 1/1 Running 0 39m
kube-system calico-node-kgtk7 1/1 Running 0 39m
kube-system calico-node-p55wj 1/1 Running 4 (8m29s ago) 39m
kube-system calico-node-rhjdq 1/1 Running 1 (9m18s ago) 39m
kube-system calico-typha-6775694657-jd66w 1/1 Running 1 (22m ago) 39m
[root@k8s-master01 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master01 Ready <none> 8d v1.24.0
k8s-master02 Ready <none> 8d v1.24.0
k8s-master03 Ready <none> 8d v1.24.0
k8s-node01 Ready <none> 8d v1.24.0
k8s-node02 Ready <none> 8d v1.24.010.安装CoreDNS
10.1以下步骤只在master01操作
10.1.1修改文件
[root@k8s-master01 ~]# cd /root/coredns
[root@k8s-master01 coredns]# vi coredns.yaml # 将 clusterIP: 改成你自己的coredns的IP，我的IP是192.168.0.2 ，注意coredns的IP一定要跟集群的第一个地址区分，我的集群第一个地址是192.168.0.1（也就是配置apiserver证书的第一个地址）。
[root@k8s-master01 coredns]# cat coredns.yaml | grep clusterIP:
clusterIP: 192.168.0.210.1.2安装
[root@k8s-master01 coredns]# kubectl create -f coredns.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created11.安装Metrics Server
k8s各个组件将自己的健康状态输送给了Metrics 接口，以便比如 Promethe 监控软件进行监控
11.1以下步骤只在master01操作
11.1.1安装Metrics-server
在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率
# 安装metrics server
[root@k8s-master01 ~]# cd /root/metrics-server/
[root@k8s-master01 metrics-server]# kubectl apply -f metrics-server.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created11.1.2稍等片刻查看状态
[root@k8s-master01 metrics-server]# kubectl top node
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)
稍等片刻查看状态
[root@k8s-master01 metrics-server]# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master01 475m 47% 1233Mi 66%
k8s-master02 557m 55% 1120Mi 60%
k8s-master03 350m 35% 1100Mi 59%
k8s-node01 103m 10% 636Mi 34%
k8s-node02 992m 99% 713Mi 38% 12.集群验证
12.1部署pod资源
执行后会卡住，是正常的，因为此命令是写yaml文件并应用，其中本地没有busybox:1.28镜像还要从公网下载
cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: busybox:1.28
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always
EOF# 查看
kubectl get pod
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 17s
12.2用pod解析默认命名空间中的kubernetes
[root@k8s-master01 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 192.168.0.1 <none> 443/TCP 9d
[root@k8s-master01 metrics-server]# kubectl exec busybox -n default -- nslookup kubernetes
Server: 192.168.0.2
Address 1: 192.168.0.2 kube-dns.kube-system.svc.cluster.local
Name: kubernetes
Address 1: 192.168.0.1 kubernetes.default.svc.cluster.local
[root@k8s-master01 metrics-server]# 12.3测试跨命名空间是否可以解析
[root@k8s-master01 ~]# kubectl exec busybox -n default -- nslookup kube-dns.kube-system
Server: 192.168.0.2
Address 1: 192.168.0.2 kube-dns.kube-system.svc.cluster.local
Name: kube-dns.kube-system
Address 1: 192.168.0.2 kube-dns.kube-system.svc.cluster.local
12.4每个节点都必须要能访问Kubernetes的kubernetes svc 443和kube-dns的service 53
[root@k8s-master01 ~]# telnet 192.168.0.1 443
Trying 192.168.0.1...
Connected to 192.168.0.1.
Escape character is '^]'.
[root@k8s-master01 ~]# telnet 192.168.0.2 53
Trying 192.168.0.2...
Connected to 192.168.0.2.
Escape character is '^]'.
[root@k8s-master01 ~]# curl 192.168.0.2:53
curl: (52) Empty reply from server12.5Pod和Pod之间要能通
[root@k8s-master01 ~]# kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 0 53m 172.7.122.129 k8s-master02 <none> <none>
[root@k8s-master01 ~]# kubectl get pod -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-56cdb7c587-8stfx 1/1 Running 7 (7m32s ago) 3h8m 10.88.0.2 k8s-master03 <none> <none>
calico-node-62ndd 1/1 Running 1 (157m ago) 3h8m 10.4.7.200 k8s-node02 <none> <none>
calico-node-ch624 1/1 Running 0 3h8m 10.4.7.11 k8s-master01 <none> <none>
calico-node-kgtk7 1/1 Running 0 3h8m 10.4.7.12 k8s-master02 <none> <none>
calico-node-p55wj 1/1 Running 4 (157m ago) 3h8m 10.4.7.22 k8s-node01 <none> <none>
calico-node-rhjdq 1/1 Running 1 (157m ago) 3h8m 10.4.7.21 k8s-master03 <none> <none>
calico-typha-6775694657-jd66w 1/1 Running 1 (171m ago) 3h8m 10.4.7.22 k8s-node01 <none> <none>
coredns-6d86b45487-x5zwx 1/1 Running 0 3m7s 172.7.32.130 k8s-master01 <none> <none>
metrics-server-6d9df85947-5hssb 1/1 Running 0 59m 172.7.58.193 k8s-node02 <none> <none># 进入busybox ping其他节点上的pod
[root@k8s-master01 ~]# kubectl exec -ti busybox -- sh
/ # ping 172.7.32.130
PING 172.7.32.130 (172.7.32.130): 56 data bytes
64 bytes from 172.7.32.130: seq=0 ttl=62 time=4.384 ms
64 bytes from 172.7.32.130: seq=1 ttl=62 time=6.536 ms
64 bytes from 172.7.32.130: seq=2 ttl=62 time=9.680 ms
# 可以连通证明这个pod是可以跨命名空间和跨主机通信的12.6创建三个副本，可以看到3个副本分布在不同的节点上（用完可以删了）
cat > nginx.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
EOF[root@k8s-master01 ~]# kubectl apply -f nginx.yaml
deployment.apps/nginx-deployment created
[root@k8s-master01 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 70 (49m ago) 2d22h
nginx-deployment-6595874d85-9565p 1/1 Running 1 (10m ago) 14m
nginx-deployment-6595874d85-m5rmh 1/1 Running 0 14m
nginx-deployment-6595874d85-sd9fr 1/1 Running 0 14m
# 删除 nginx
[root@k8s-master01 ~]# kubectl delete -f nginx.yaml
deployment.apps "nginx-deployment" deleted13.安装dashboard
13.1 下载dashboard_v2.4.0安装包
官网：GitHub - kubernetes/dashboard at v2.4.0
找到 recommended.yaml 文件
13.2 部署dashboard_v2.4.0
[root@k8s-master01 ~]# mkdir /root/dashboard;cd /root/dashboard
[root@k8s-master01 dashboard]# rz recommended.yaml
[root@k8s-master01 dashboard]# vi recommended.yaml # 按照如下修改
原配置文件为：
修改后：
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort
ports:
- port: 443
nodePort: 31245
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
---
应用资源配置清单
[root@k8s-master01 dashboard]# kubectl apply -f recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
查看所属node以及端口
kubectl -n kubernetes-dashboard get pod,svc -o wide使用浏览器访问任意一个节点的31245端口，就可以访问dashboard页面，说明了，dashboard的页面是https://dashboard_IP:443，而我们通过NodePort类型，将31245端口映射到了dashboard容器中的443，而NodePort代表的是宿主机，node节点，所以访问node节点的IP+31245，就是访问的dashboard容器中的443端口（https://dashboard_IP:443）。注意，必须使用稍微低版本的火狐浏览器，如果使用谷歌等，会发现页面一直显示空白。是因为这个高内核的浏览器，无法容忍没有安全认证的https协议的网页，所以如果要显示，还需要在配置ssl，请看这篇文章
配置从k8s中配置出一个token，让我们有权限登录到dashboard，所以我需要创建一个admin-user用户，绑定一个k8s集群角色，然后通过admin-user用户生成一个token
[root@k8s-master01 dashboard]# vi create-admin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
应用资源配置清单
[root@k8s-master01 dashboard]# kubectl apply -f create-admin.yaml
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created[root@k8s-master01 dashboard]# kubectl get serviceaccount -n kubernetes-dashboard -o yaml
apiVersion: v1
items:
- apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"admin-user","namespace":"kubernetes-dashboard"}}
creationTimestamp: "2022-07-12T05:13:50Z"
name: admin-user
namespace: kubernetes-dashboard
resourceVersion: "115033"
uid: 6d3d9c79-b692-4a9f-a7a4-f569fad6c21f
- apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: "2022-07-11T23:54:08Z"
name: default
namespace: kubernetes-dashboard
resourceVersion: "72877"
uid: 5436446c-aa69-424a-9289-eca0a42feeda
- apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"labels":{"k8s-app":"kubernetes-dashboard"},"name":"kubernetes-dashboard","namespace":"kubernetes-dashboard"}}
creationTimestamp: "2022-07-11T23:54:08Z"
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
resourceVersion: "72879"
uid: 3e559071-9bcf-43cb-9b6d-10e61a270eb3
kind: List
metadata:
resourceVersion: ""创建token
[root@k8s-master01 dashboard]# kubectl -n kubernetes-dashboard create token admin-user
eyJhbGciOiJSUzI1NiIsImtpZCI6IjBua1VscDM0V1JJT3ZxRnNBdUFZdm5fd19XeDhvOHY4M25wOGxOcEdmUWsifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjU2NDg4ODc1LCJpYXQiOjE2NTY0ODUyNzUsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJhZG1pbi11c2VyIiwidWlkIjoiYjhlYzQ2N2EtYzRhZC00OTZjLWIyYTctN2YyYTNiYzdjOWFiIn19LCJuYmYiOjE2NTY0ODUyNzUsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDphZG1pbi11c2VyIn0.Ywh2TzMV3wGknts0WAVIUBhPXX-pOllI1EI_QHDSTiTNB-ii0a4iC6b8VtZaw1l3tznu3a7nO53sAUILhyhocnEUSLV7bXhWQK_kafIOTWxTOZneVi-4IA7wxrLxv06m7Xhh8somR9lfiMU4Tscnmu2_imT0aoItk3t6mAwY9umyc5PfyeCO4vjFre_ddG6B2IxVEnnzvHS8ViIOrWMQuEUYrB58RL1h7EA1AbGe0Wt2rJRGj0aZ23xf7JGTWS1slQnrjxCU_8YWXTYm0nJQWeqGqzEQbZXdKrf_aIq5ogZejPfylMOeZwXYVno_kx3NAIkjZGKtFt_--o8xyWySpg复制token到dashboard页面登录
14.ingress安装
14.1写入配置文件，并执行
原理：ingress-controller 就是把k8s内部的ningx调度器，ingress-controller跟宿主机共享网络空间，通过nodeport 80 把宿主机的80端口映射到ingress-controller的80端口，这时候配置ingress资源，比如 tomcat.od.com，其实也就是访问的是tomcat.od.com:80，这时候我们通过hosts或者named将，tomcat.od.com解析成ingress-controller的pod所在的宿主机IP，也就是访问的是宿主机的IP:80，由于访问宿主机的IP:80，也就是ingress-controller的pod的80，所以访问tomcat.od.com流量引入到了ingress-controller，ingress-controller跟api通讯获得有一个ingress资源是tomcat.od.com，这个ingress资源把流量给到了他指定的service资源，service在通过kube-proxy负载给到pod。
注：
1、在上述说的，我需要一个把访问tomcat.od.com给到一个宿主机，这个宿主机是谁，依赖于ingress-controller的pod在那个机器，所以我i们通过node选择器，固定在那些node节点运行
2、一定提前把镜像下载下来，否则会发现，ingress功能不好使，在被选定的node节点运行，此文章我们选择了使用daemonset类型，通过node选择器固定到master01、master02crictl pull willdockerhub/ingress-nginx-controller:v1.1.0 crictl registry.cn-hangzhou.aliyuncs.com/chenby/kube-webhook-certgen:v1.1.1
[root@k8s-master01 ~]# mkdir ingress;cd ingress/
[root@k8s-master01 ingress]# vim deploy.yaml
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
---
# Source: ingress-nginx/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
automountServiceAccountToken: true
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
data:
allow-snippet-annotations: 'true'
---
# Source: ingress-nginx/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
name: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
- namespaces
verbs:
- list
- watch
- apiGroups:
- ''
resources:
- nodes
verbs:
- get
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
---
# Source: ingress-nginx/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
name: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- namespaces
verbs:
- get
- apiGroups:
- ''
resources:
- configmaps
- pods
- secrets
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- configmaps
resourceNames:
- ingress-controller-leader
verbs:
- get
- update
- apiGroups:
- ''
resources:
- configmaps
verbs:
- create
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
---
# Source: ingress-nginx/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller-admission
namespace: ingress-nginx
spec:
type: ClusterIP
ports:
- name: https-webhook
port: 443
targetPort: webhook
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
type: NodePort
externalTrafficPolicy: Local
ipFamilyPolicy: SingleStack
ipFamilies:
- IPv4
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
appProtocol: http
- name: https
port: 443
protocol: TCP
targetPort: https
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
revisionHistoryLimit: 10
minReadySeconds: 0
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
nodeSelector:
app: ingress
containers:
- name: controller
image: willdockerhub/ingress-nginx-controller:v1.1.0
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
args:
- /nginx-ingress-controller
- --election-id=ingress-controller-leader
- --controller-class=k8s.io/ingress-nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
runAsUser: 101
allowPrivilegeEscalation: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LD_PRELOAD
value: /usr/local/lib/libmimalloc.so
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
- name: webhook
containerPort: 8443
protocol: TCP
volumeMounts:
- name: webhook-cert
mountPath: /usr/local/certificates/
readOnly: true
resources:
requests:
cpu: 100m
memory: 90Mi
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
---
# Source: ingress-nginx/templates/controller-ingressclass.yaml
# We don't support namespaced ingressClass yet
# So a ClusterRole and a ClusterRoleBinding is required
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: nginx
namespace: ingress-nginx
spec:
controller: k8s.io/ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
# before changing this value, check the required kubernetes version
# https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
name: ingress-nginx-admission
webhooks:
- name: validate.nginx.ingress.kubernetes.io
matchPolicy: Equivalent
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- ingresses
failurePolicy: Fail
sideEffects: None
admissionReviewVersions:
- v1
clientConfig:
service:
namespace: ingress-nginx
name: ingress-nginx-controller-admission
path: /networking/v1/ingresses
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
rules:
- apiGroups:
- admissionregistration.k8s.io
resources:
- validatingwebhookconfigurations
verbs:
- get
- update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
rules:
- apiGroups:
- ''
resources:
- secrets
verbs:
- get
- create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-create
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
template:
metadata:
name: ingress-nginx-admission-create
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: create
image: registry.cn-hangzhou.aliyuncs.com/chenby/kube-webhook-certgen:v1.1.1
imagePullPolicy: IfNotPresent
args:
- create
- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
- --namespace=$(POD_NAMESPACE)
- --secret-name=ingress-nginx-admission
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
securityContext:
allowPrivilegeEscalation: false
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-patch
namespace: ingress-nginx
annotations:
helm.sh/hook: post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
template:
metadata:
name: ingress-nginx-admission-patch
labels:
helm.sh/chart: ingress-nginx-4.0.10
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.1.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: patch
image: registry.cn-hangzhou.aliyuncs.com/chenby/kube-webhook-certgen:v1.1.1
imagePullPolicy: IfNotPresent
args:
- patch
- --webhook-name=ingress-nginx-admission
- --namespace=$(POD_NAMESPACE)
- --patch-mutating=false
- --secret-name=ingress-nginx-admission
- --patch-failure-policy=Fail
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
securityContext:
allowPrivilegeEscalation: false
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 2000
给node01 node02 添加label，使其ingress-controller指定跑在这两个节点
[root@k8s-master01 ingress]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master01 Ready <none> 19h v1.24.0
k8s-master02 Ready <none> 15h v1.24.0
k8s-master03 Ready <none> 19h v1.24.0
k8s-node01 Ready <none> 19h v1.24.0
k8s-node02 Ready <none> 19h v1.24.0
[root@k8s-master01 ingress]# kubectl label node k8s-node01 k8s-node02 app=ingress
node/k8s-node01 labeled
node/k8s-node02 labeled
[root@k8s-master01 ingress]# kubectl get label --show-lablekubectl get label --show-lable^C
[root@k8s-master01 ingress]# kubectl get node -A --show-labels
NAME STATUS ROLES AGE VERSION LABELS
k8s-master01 Ready <none> 19h v1.24.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master01,kubernetes.io/os=linux,node.kubernetes.io/node=
k8s-master02 Ready <none> 15h v1.24.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master02,kubernetes.io/os=linux,node.kubernetes.io/node=
k8s-master03 Ready <none> 19h v1.24.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master03,kubernetes.io/os=linux,node.kubernetes.io/node=
k8s-node01 Ready <none> 19h v1.24.0 app=ingress,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node01,kubernetes.io/os=linux,node.kubernetes.io/node=
k8s-node02 Ready <none> 19h v1.24.0 app=ingress,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node02,kubernetes.io/os=linux,node.kubernetes.io/node=
[root@k8s-master01 ingress]# 应用资源清单
[root@k8s-master01 ingress]# kubectl apply -f deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
daemonset.apps/ingress-nginx-controller created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created查看pod运行状态
[root@k8s-master01 ingress]# kubectl get pod -n ingress-nginx -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-admission-create-hqrb7 0/1 Completed 0 14h 172.7.122.130 k8s-master02 <none> <none>
ingress-nginx-admission-patch-bbkj4 0/1 Completed 2 14h 172.7.58.193 k8s-node02 <none> <none>
ingress-nginx-controller-dzvz4 1/1 Running 0 116s 10.4.7.200 k8s-node02 <none> <none>
ingress-nginx-controller-qmg6n 1/1 Running 0 37m 10.4.7.22 k8s-node01 <none> <none>14.2 配置反代
思路：目前已经部署了两个 ingress-controller 的 pod，分别在 node01 node02 ，其中ingress-controller 的 pod 跟宿主机共享网络名称空间，而ingress-controller启动的是80端口，所以只要把想要访问网站流量送入node01 node02 宿主机的80端口，就能够被ingress-controller接收到，后续就是k8s内部匹配ingressg规则转发给service资源的事情了。目前我们有两个 ingress-controller，所以完全可以让访问的流量通过LB负载均衡机制，分别访问两个ingress-controller。我们使用nginx作为负载均衡机制。
制作过程：我们找其中一个节点，比如是node02，安装nginx，在nginx中配置只要这个节点接收到是访问 *.od.com 这个域名的网站，全部转发给10.4.7.22:80 和 10.4.7.200:80，然后在named中做域名解析，只要是访问*.od.com ，统统抛给node02。但如果node02挂了怎么办，为了保证高可用，node01 node02 分别安装nginx，都配置一样的nginx反向代理配置，并安装ksspalived，使其实现如果node02挂了，可以飘逸到node01 。在第五章的时候，已经在node01 node02配置高可用，VIP为10.4.7.10
node01 node02 操作：
yum install -y nginx
vi /etc/nginx/conf.d/od.com.conf
server {
server_name *.od.com;
location / {
proxy_pass http://default_backend_nginx;
proxy_set_header Host $http_host;
proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
}
}
upstream default_backend_nginx {
# 使用的是daemonset，所有的nodes都放到upstream中
server 10.4.7.22:80 max_fails=3 fail_timeout=10s;
server 10.4.7.200:80 max_fails=3 fail_timeout=10s;
}systemctl restart nginx ;systemctl enable nginx
14.3 安装测试应用
[root@k8s-master01 ~]# mkdir/root/nginx; cd /root/nginx
[root@k8s-master01 nginx]# vi deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-dp
namespace: default
labels:
app: nginx-lable
spec:
selector:
matchLabels:
app: nginx-1.7.9
replicas: 1
template:
metadata:
labels:
app: nginx-1.7.9
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80[root@k8s-master01 nginx]# vi svc.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx-svc
namespace: default
labels:
app: nginx-svc-label
spec:
selector:
app: nginx-1.7.9
ports:
- port: 881
protocol: TCP
targetPort: 80[root@k8s-master01 nginx]# vi ingress.yaml
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
name: nginx-ingress
namespace: default
labels:
app: nginx-ingress-label
spec:
ingressClassName: nginx
rules:
- host: nginx.od.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-svc
port:
number: 881应用资源清单
[root@k8s-master01 nginx]# kubectl apply -f deploy.yaml
deployment.apps/nginx-dp created
[root@k8s-master01 nginx]# kubectl apply -f svc.yaml
service/nginx-svc created
[root@k8s-master01 nginx]# kubectl apply -f ingress.yaml
ingress.networking.k8s.io/nginx-ingress created[root@k8s-master01 nginx]# kubectl get all -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-dp-5d9dbd4c6d-zfv42 1/1 Running 0 14m 172.7.195.3 k8s-master03 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 192.168.0.1 <none> 443/TCP 23h <none>
service/nginx-svc ClusterIP 192.168.139.174 <none> 881/TCP 13m app=nginx-1.7.9
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/nginx-dp 1/1 1 1 14m nginx nginx:1.7.9 app=nginx-1.7.9
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/nginx-dp-5d9dbd4c6d 1 1 1 14m nginx nginx:1.7.9 app=nginx-1.7.9,pod-template-hash=5d9dbd4c6d验证nginx的pod跟service无问题
# 直接访问pod的80端口
[root@k8s-master01 nginx]# curl 172.7.195.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
# 直接访问service的881端口，因为service 配置的是 881 代理pod的 80
[root@k8s-master01 nginx]# curl 192.168.139.174:881
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
配置named，让访问nginx.od.com 的流量抛给10.4.7.10这个VIP，通过nginx的配置把流量引入到node的80端口
[root@k8s-master01 nginx]# vi /var/named/od.com.zone
[root@k8s-master01 nginx]# vi /var/named/od.com.zone
$ORIGIN od.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.od.com. dnsadmin.od.com. (
2020010501 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.od.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
nginx A 10.4.7.10[root@k8s-master01 nginx]# systemctl restart named
[root@k8s-master01 nginx]# dig -t A nginx.od.com @10.4.7.11 +short
10.4.7.10
验证访问curl nginx.od.com
[root@k8s-master01 nginx]# curl nginx.od.com
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@k8s-master01 nginx]# 14.3 DefaultBackend 配置
介绍：没有 rules 的 Ingress 将所有流量发送到同一个默认后端。 defaultBackend 通常是ingress控制器的配置选项，而非在 Ingress 资源中指定。如果 hosts 或 paths 都没有与 Ingress 对象中的 HTTP 请求匹配，则流量将路由到默认后端
理解：说白了意思就是一个请求的url，被ingress-controller接收到后，如果没有ingress规则匹配上，默认走defaultBackend的规则，自己测试了一下，有和没有DefaultBackend这个配置，无任何影响。具体为什么呢？比如有一个ingress资源匹配的是nginx.od.com，但访问nginx.od.com/123，是不是就匹配了defaultBackend呢？并不是。defaultBackend 只管理那些完全没有任何规则匹配上资源，比如nihao.od.com或者nihao.od.com/123，在ingress中根本连nihao.od.com资源都没有的。所以功能鸡肋，目前没发现任何价值。再就是我们本身都通过named域名解析，才能转发给ingress-controller，如果named没配置，根本转发不到ingress-controller。而且在配置中还发现了
[root@k8s-master01 backend]# mkdir /root/backend; cd /root/backend
[root@k8s-master01 backend]# vi deploy_backend.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: default-http-backend
labels:
app: default-http-backend
spec:
replicas: 1
selector:
matchLabels:
app: default-http-backend
template:
metadata:
labels:
app: default-http-backend
spec:
terminationGracePeriodSeconds: 60
containers:
- name: default-http-backend
# Any image is permissible as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: registry.cn-hangzhou.aliyuncs.com/chenby/defaultbackend-amd64:1.5
imagePullPolicy: IfNotPresent
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mivi svc.yaml
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
labels:
app: default-http-backend
spec:
ports:
- port: 80
targetPort: 8080
selector:
app: default-http-backend注意：既然是没有规则的url，所以必须的有ingress资源，但是怎么配置的。rules规则些什么的，默认给出的defaultBackend参数，指向default-http-backend，但是无语的是，如果就按照如下create -f 应用到k8s后，根本不好用。此参数还必须搭配在其他的ingress资源中使用，真实鸡肋
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: tomcat-demo spec: defaultBackend: service: name: default-http-backend port: number: 80
[root@k8s-master01 ~]# cd /root/nginx
[root@k8s-master01 nginx]# vi ingress.yaml # 修改nginx的资源配置清单
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
name: nginx-ingress
namespace: default
labels:
app: nginx-ingress-label
spec:
defaultBackend:
service:
name: default-http-backend
port:
number: 80
ingressClassName: nginx
rules:
- host: nginx.od.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-svc
port:
number: 881
[root@k8s-master01 nginx]# 应用资源清单
[root@k8s-master01 backend]# kubectl apply -f deploy_backend.yaml
deployment.apps/default-http-backend created
[root@k8s-master01 backend]# kubectl apply -f svc_backend.yaml
service.apps/default-http-backend created
[root@k8s-master01 nginx]# kubectl apply -f ingress.yaml
ingress.networking.k8s.io/nginx-ingress changed[root@k8s-master01 nginx]# vi /var/named/od.com.zone
[root@k8s-master01 nginx]# vi /var/named/od.com.zone
$ORIGIN od.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.od.com. dnsadmin.od.com. (
2020010501 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.od.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
nginx A 10.4.7.10
nihao A 10.4.7.10[root@k8s-master01 nginx]# systemctl restart named
[root@k8s-master01 nginx]# dig -t A nihao.od.com @10.4.7.11 +short
10.4.7.10
验证
[root@k8s-master01 nginx]# curl nginx.od.com/123
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.7.9</center>
</body>
</html>
[root@k8s-master01 nginx]# curl nihao.od.com
default backend - 404[root@k8s-master01 nginx]# 这个才是default backend的404网页
[root@k8s-master01 nginx]# curl nihao.od.com/123
default backend - 404[root@k8s-master01 nginx]#
[root@k8s-master01 nginx]# curl helloword.od.com
curl: (6) Could not resolve host: helloword.od.com若遇到Ingress-nginx-controller创建yaml文件报x509: certificate signed by unknown authority等错误，查看https://blog.csdn.net/Jerry00713/article/details/125743837
1566
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
