Kubernetes环境搭建

Kubernetes部署说明

K8S中文官网：https://kubernetes.io/zh-cn

主机名	ip	角色
harbor.huazi.org	172.25.254.250	harbor仓库
k8s-master.org	172.25.254.100	master，k8s集群控制节点
k8s-node1.org	172.25.254.10	worker，k8s集群工作节点
k8s-node2.org	172.25.254.20	worker，k8s集群工作节点

要求：除了harbor服务器的所有k8s节点

• 所有节点禁用selinux和防火墙
• 所有节点同步时间和解析
• 所有节点安装docker-ce
• 所有节点禁用swap，注意注释掉/etc/fstab文件中的定义（因为k8s集群中容器对内存要求非常高，如果不禁用swap，一些容器就会存到swap中，这样可能会使你的容器暂停或者运行缓慢）

环境准备工作

主机准备

harbor搭建

• harbor的搭建，我们可以参考这篇博客：
• https://blog.csdn.net/huaz_md/article/details/142671140?spm=1001.2014.3001.5501

这里博主只写出几个关键的操作

• 制作https的证书和key

[root@harbor ~]# openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/huazi.org.key -addext "subjectAltName = DNS:harbor.huazi.org" -x509 -days 365 -out certs/huazi.org.crt
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:shanxi
Locality Name (eg, city) [Default City]:xi'an
Organization Name (eg, company) [Default Company Ltd]:docker
Organizational Unit Name (eg, section) []:registry
Common Name (eg, your name or your server's hostname) []:harbor.huazi.org
Email Address []:admin@huazi.org

• 修改harbor的配置文件

[root@harbor harbor]# cp harbor.yml.tmpl harbor.yml
[root@harbor harbor]# vim harbor.yml

• 启动harbor

[root@harbor harbor]# ./install.sh --with-chartmuseum

关闭防火墙和selinux

[root@harbor ~]# systemctl is-active firewalld.service
inactive
[root@harbor ~]# getenforce
Disabled

k8s集群中的主机名和ip设定

• k8s-master

[root@k8s-master ~]# hostname -I
172.25.254.100

[root@k8s-master ~]# hostnamectl hostname k8s-master.org
[root@k8s-master ~]# hostname
k8s-master.org

• k8s-node1

[root@k8s-node1 ~]# hostname -I
172.25.254.10

[root@k8s-node1 ~]# hostnamectl hostname k8s-node1.org
[root@k8s-node1 ~]# hostname
k8s-node1.org

• k8s-node2

[root@k8s-node2 ~]# hostname -I
172.25.254.20

[root@k8s-node2 ~]# hostnamectl hostname k8s-node2.org
[root@k8s-node2 ~]# hostname
k8s-node2.org

k8s集群中设置hosts解析

• k8s-master

[root@k8s-master ~]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.25.254.100   k8s-master.org
172.25.254.10    k8s-node1.org
172.25.254.20    k8s-node2.org
172.25.254.250   harbor.huazi.org

• k8s-node1（这里我们使用scp命令）

[root@k8s-master ~]# scp   /etc/hosts   root@172.25.254.10:/etc/hosts
The authenticity of host '172.25.254.10 (172.25.254.10)' can't be established.
ED25519 key fingerprint is SHA256:oRI0QHrpuaAH8E6hepK2f2FymklDq9LifjGxkU86pMg.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '172.25.254.10' (ED25519) to the list of known hosts.
root@172.25.254.10's password:
hosts                                                         100%  286   253.6KB/s   00:00


[root@k8s-node1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.25.254.100   k8s-master.org
172.25.254.10    k8s-node1.org
172.25.254.20    k8s-node2.org
172.25.254.250   harbor.huazi.org

• k8s-node2（这里我们使用scp命令）

[root@k8s-master ~]# scp   /etc/hosts   root@172.25.254.20:/etc/hosts
The authenticity of host '172.25.254.20 (172.25.254.20)' can't be established.
ED25519 key fingerprint is SHA256:oRI0QHrpuaAH8E6hepK2f2FymklDq9LifjGxkU86pMg.
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:1: 172.25.254.10
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '172.25.254.20' (ED25519) to the list of known hosts.
root@172.25.254.20's password:
hosts                                                         100%  286   431.8KB/s   00:00



[root@k8s-node2 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.25.254.100   k8s-master.org
172.25.254.10    k8s-node1.org
172.25.254.20    k8s-node2.org
172.25.254.250   harbor.huazi.org

• ping测试

[root@k8s-master ~]# ping harbor.huazi.org
PING harbor.huazi.org (172.25.254.250) 56(84) 比特的数据。
64 比特，来自 harbor.huazi.org (172.25.254.250): icmp_seq=1 ttl=64 时间=0.629 毫秒
64 比特，来自 harbor.huazi.org (172.25.254.250): icmp_seq=2 ttl=64 时间=0.288 毫秒
^C
--- harbor.huazi.org ping 统计 ---
已发送 2 个包， 已接收 2 个包, 0% packet loss, time 1061ms
rtt min/avg/max/mdev = 0.288/0.458/0.629/0.170 ms
[root@k8s-master ~]# ping k8s-node1.org
PING k8s-node1.org (172.25.254.10) 56(84) 比特的数据。
64 比特，来自 k8s-node1.org (172.25.254.10): icmp_seq=1 ttl=64 时间=0.422 毫秒
64 比特，来自 k8s-node1.org (172.25.254.10): icmp_seq=2 ttl=64 时间=0.339 毫秒
^C
--- k8s-node1.org ping 统计 ---
已发送 2 个包， 已接收 2 个包, 0% packet loss, time 1038ms
rtt min/avg/max/mdev = 0.339/0.380/0.422/0.041 ms
[root@k8s-master ~]# ping k8s-node2.org
PING k8s-node2.org (172.25.254.20) 56(84) 比特的数据。
64 比特，来自 k8s-node2.org (172.25.254.20): icmp_seq=1 ttl=64 时间=0.660 毫秒
64 比特，来自 k8s-node2.org (172.25.254.20): icmp_seq=2 ttl=64 时间=0.256 毫秒
64 比特，来自 k8s-node2.org (172.25.254.20): icmp_seq=3 ttl=64 时间=0.666 毫秒
^C
--- k8s-node2.org ping 统计 ---
已发送 3 个包， 已接收 3 个包, 0% packet loss, time 2078ms
rtt min/avg/max/mdev = 0.256/0.527/0.666/0.191 ms

k8s中的所有节点关闭防火墙和selinux

• k8s-master

[root@k8s-master ~]# systemctl is-active firewalld.service
inactive
[root@k8s-master ~]# getenforce
Disabled

• k8s-node1

[root@k8s-node1 ~]# systemctl is-active firewalld.service
inactive
[root@k8s-node1 ~]# getenforce
Disabled

• k8s-node2

[root@k8s-node2 ~]# systemctl is-active firewalld.service
inactive
[root@k8s-node2 ~]# getenforce
Disabled

k8s集群中禁用swap分区

• k8s-master

[root@k8s-master ~]# vim /etc/fstab

[root@k8s-master ~]# swapon -s
Filename                                Type            Size            Used            Priority
/dev/nvme0n1p2                          partition       2097148         0               -2
[root@k8s-master ~]# swapoff -a
[root@k8s-master ~]# swapon -s  #发现swap分区已经没了
[root@k8s-master ~]#

• k8s-node1

[root@k8s-node1 ~]# vim /etc/fstab

[root@k8s-node1 ~]# systemctl mask swap.target
Created symlink /etc/systemd/system/swap.target → /dev/null.
[root@k8s-node1 ~]# swapoff -a
[root@k8s-node1 ~]# swapon -s

• k8s-node2

[root@k8s-node2 ~]# vim /etc/fstab

[root@k8s-node2 ~]# systemctl mask swap.target
Created symlink /etc/systemd/system/swap.target → /dev/null.
[root@k8s-node2 ~]# swapoff -a
[root@k8s-node2 ~]# swapon -s

k8s集群中安装docker-ce

[root@k8s-master yum.repos.d]# vim redhat.repo
[docker]
name=docker-ce
baseurl=https://mirrors.aliyun.com/docker-ce/linux/rhel/9/x86_64/stable/
gpgcheck=0


[root@k8s-master yum.repos.d]# scp /etc/yum.repos.d/redhat.repo root@172.25.254.10:/etc/yum.repos.d/
root@172.25.254.10's password:
redhat.repo                                                   100%  108   130.7KB/s   00:00



[root@k8s-master yum.repos.d]# scp /etc/yum.repos.d/redhat.repo root@172.25.254.20:/etc/yum.repos.d/
root@172.25.254.20's password:
redhat.repo                                                   100%  108   180.7KB/s   00:00

• k8s-master

[root@k8s-master ~]# yum install docker-ce -y


[root@k8s-master ~]# systemctl enable --now docker

• k8s-node1

[root@k8s-node1 ~]# yum install docker-ce -y


[root@k8s-node1 ~]# systemctl enable --now docker

• k8s-node2

[root@k8s-node2 ~]# yum install docker-ce -y


[root@k8s-node2 ~]# systemctl enable --now docker

当我们在执行docker info后，出现了如下WARNING，如何处理呢

[root@k8s-master ~]# docker info
......
......
......
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

[root@k8s-master ~]# echo br_netfilter > /etc/modules-load.d/docker_mod.conf

[root@k8s-master ~]# modprobe br_netfilter

[root@k8s-master ~]# vim /etc/sysctl.d/docker.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1


[root@k8s-master ~]# sysctl --system

[root@k8s-master ~]# systemctl restart docker

[root@k8s-node1 ~]# echo br_netfilter > /etc/modules-load.d/docker_mod.conf

[root@k8s-node1 ~]# modprobe br_netfilter

[root@k8s-node1 ~]# vim /etc/sysctl.d/docker.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1


[root@k8s-node1 ~]# sysctl --system

[root@k8s-node1 ~]# systemctl restart docker

[root@k8s-node2 ~]# echo br_netfilter > /etc/modules-load.d/docker_mod.conf

[root@k8s-node2 ~]# modprobe br_netfilter

[root@k8s-node2 ~]# vim /etc/sysctl.d/docker.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1


[root@k8s-node2 ~]# sysctl --system

[root@k8s-node2 ~]# systemctl restart docker

再次docker info时，就没有WARNING了

k8s集群中下载harbor证书

• k8s-master

[root@k8s-master ~]# mkdir -p /etc/docker/certs.d/harbor.huazi.org/


[root@k8s-master ~]# scp    root@harbor.huazi.org:/root/certs/huazi.org.crt   /etc/docker/certs.d/harbor.huazi.org/ca.crt
The authenticity of host 'harbor.huazi.org (172.25.254.250)' can't be established.
ED25519 key fingerprint is SHA256:wkrDVNwOtwB4XhstKSlH+BEeO8JE3gp9NDIKRe6aMu0.
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:5: 172.25.254.250
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'harbor.huazi.org' (ED25519) to the list of known hosts.
root@harbor.huazi.org's password:
huazi.org.crt                                                          100% 2175     1.9MB/s   00:00


[root@k8s-master ~]# ll /etc/docker/certs.d/harbor.huazi.org/ca.crt
-rw-r--r-- 1 root root 2163 10月  2 05:19 /etc/docker/certs.d/harbor.huazi.org/ca.crt


[root@k8s-master ~]# systemctl restart docker

• k8s-node1

[root@k8s-node1 ~]# mkdir -p /etc/docker/certs.d/harbor.huazi.org/


[root@k8s-node1 ~]# scp    root@harbor.huazi.org:/root/certs/huazi.org.crt   /etc/docker/certs.d/harbor.huazi.org/ca.crt
root@harbor.huazi.org's password:
huazi.org.crt                                                          100% 2175     2.4MB/s   00:00


[root@k8s-node1 ~]# ll /etc/docker/certs.d/harbor.huazi.org/ca.crt
-rw-r--r-- 1 root root 2175 10月  2 05:55 /etc/docker/certs.d/harbor.huazi.org/ca.crt


[root@k8s-node1 ~]# systemctl restart docker

• k8s-node2

[root@k8s-node2 ~]# mkdir -p /etc/docker/certs.d/harbor.huazi.org/


[root@k8s-node2 ~]# scp    root@harbor.huazi.org:/root/certs/huazi.org.crt   /etc/docker/certs.d/harbor.huazi.org/ca.crt
root@harbor.huazi.org's password:
huazi.org.crt                                                          100% 2175     2.5MB/s   00:00


[root@k8s-node2 ~]# ll /etc/docker/certs.d/harbor.huazi.org/ca.crt
-rw-r--r-- 1 root root 2175 10月  2 05:57 /etc/docker/certs.d/harbor.huazi.org/ca.crt


[root@k8s-node2 ~]# systemctl restart docker

k8s集群中配置harbor镜像加速器

• k8s-master

[root@k8s-master ~]# cd /etc/docker/
[root@k8s-master docker]# ls
certs.d
[root@k8s-master docker]# vim daemon.json
{
        "registry-mirrors": ["https://harbor.huazi.org"]
}
[root@k8s-master docker]# systemctl restart docker


[root@k8s-master ~]# docker info
......
......
......
 Registry Mirrors:
  https://harbor.huazi.org/

• k8s-node1

[root@k8s-node1 ~]# cd /etc/docker/
[root@k8s-node1 docker]# ls
certs.d
[root@k8s-node1 docker]# vim daemon.json
{
        "registry-mirrors": ["https://harbor.huazi.org"]
}
[root@k8s-node1 docker]# cd
[root@k8s-node1 ~]# systemctl restart docker



[root@k8s-node1 ~]# docker info
......
......
......
 Registry Mirrors:
  https://harbor.huazi.org/

• k8s-node2

[root@k8s-node2 ~]# cd /etc/docker/
[root@k8s-node2 docker]# ls
certs.d
[root@k8s-node2 docker]# vim daemon.json
{
        "registry-mirrors": ["https://harbor.huazi.org"]
}
[root@k8s-node2 docker]# cd
[root@k8s-node2 ~]# systemctl restart docker


[root@k8s-node2 ~]# docker info
......
......
......
 Registry Mirrors:
  https://harbor.huazi.org/

k8s节点登录harbor测试

• k8s-master

[root@k8s-master ~]# docker login harbor.huazi.org
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credential-stores

Login Succeeded

• k8s-node1

[root@k8s-node1 ~]# docker login harbor.huazi.org
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credential-stores

Login Succeeded

• k8s-node2

[root@k8s-node2 ~]# docker login harbor.huazi.org
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credential-stores

Login Succeeded

至此kubernetes环境搭建成功