kubeadm部署K8S集群
前言：
注意：本文档的所有操作请先在测环境进行实践，请不要直接在真实的服务器中操作！
如有疑问或建议，均可联系本人。
版权声明：
本文档以开源的形式发布，所有条款如下：
- 无担保：作者不保证文档内容的准确无误，亦不承担由于使用此文档所导致的任何后果
- 自由使用：任何人可以出于任何目的而自由地 阅读/链接/打印/转载/引用/再创作 此文档，无需任何附加条件
若您 阅读/链接/打印/转载/引用/再创作 本文档，则说明接受以上2个条款。
版本：v0.1
技术储备
- https证书
- docker
- 缓存rpm包
- k8s
资源储备
链接：https://pan.baidu.com/s/1PGfnv_ZSXfT86laZulDMaQ
提取码：prha
-
docker-ce-23.0.1-1.el7.x86_64及其依赖包（harbor节点使用）
-
docker-ce-20.10.24-3.el7.x86_64.rpm及其依赖包（k8s节点使用）
-
harbor-offline-installer-v1.10.10.tgz
-
k8s资源包
- kubeadm-1.23.17-0
- kubelet-1.23.17-0
- kubectl-1.23.17-0
- kube-flannel.yml
机器规划
| 主机名 | IP | 备注 |
|---|---|---|
| harbor188.westudy.com | 10.0.0.188/24 | 镜像仓库 |
| k8s181.westudy.com | 10.0.0.181/24 | control-plane |
| k8s182.westudy.com | 10.0.0.182/24 | worker01 |
| k8s183.westudy.com | 10.0.0.183/24 | worker01 |
目录
机器环境准备
- 修改主机名与IP
# habor机器执行
hostnamectl set-hostname harbor188.westudy.com && nmcli connection modify ens32 ipv4.addresses 10.0.0.188/24 && nmcli connection up ens32
# k8s181.westudy.com机器执行
hostnamectl set-hostname k8s181.westudy.com && nmcli connection modify ens32 ipv4.addresses 10.0.0.181/24 && nmcli connection up ens32
# k8s182.westudy.com机器执行
hostnamectl set-hostname k8s182.westudy.com && nmcli connection modify ens32 ipv4.addresses 10.0.0.182/24 && nmcli connection up ens32
# k8s183.westudy.com机器执行
hostnamectl set-hostname k8s183.westudy.com && nmcli connection modify ens32 ipv4.addresses 10.0.0.183/24 && nmcli connection up ens32
- 配置hosts文件
# 所有机器执行
cat > /etc/hosts <<'EOF'
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.188 harbor188.westudy.com
10.0.0.181 k8s181.westudy.com
10.0.0.182 k8s182.westudy.com
10.0.0.183 k8s183.westudy.com
EOF
- 配置免密登录
# 仅在k8s181.westudy.com机器执行
[root@k8s181.westudy.com ~]$ cat password_free_login.sh
#!/bin/bash
# auther: Jason Luo
# 创建密钥对
ssh-keygen -t rsa -P "" -f /root/.ssh/id_rsa -q
# 声明你服务器密码，建议所有节点的密码均一致，否则该脚本需要再次进行优化
export mypasswd=119119
# 定义主机列表
k8s_host_list=(harbor188.westudy.com k8s181.westudy.com k8s182.westudy.com k8s183.westudy.com)
# 配置免密登录，利用expect工具免交互输入
for i in ${k8s_host_list[@]};do
expect -c "
spawn ssh-copy-id -i /root/.ssh/id_rsa.pub root@$i
expect {
\"*yes/no*\" {send \"yes\r\"; exp_continue}
\"*password*\" {send \"$mypasswd\r\"; exp_continue}
}"
done
[root@k8s181.westudy.com ~]$ sh password_free_login.sh
- 其他
# 所有节点执行
# 关闭swap分区
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
# 允许iptable检查桥接流量
cat <<EOF | tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
# 关闭防火墙
systemctl disable --now firewalld
# 关闭selinux
getenforce # 这条命令仅查看selinux状态
# 修改终端颜色
cat > ~/.bashrc <<'EOF'
PS1='[\[\e[34;1m\]\u@\[\e[0m\]\[\e[32;1m\]\H\[\e[0m\]\[\e[31;1m\] \W\[\e[0m\]]# '
EOF
source ~/.bashrc
搭建harbor仓库
安装docker
- 导入软件包
[root@harbor188.westudy.com ~]$ ls
anaconda-ks.cfg docker-ce-23_0_1.tar.gz harbor-offline-installer-v1.10.10.tgz
[root@harbor188.westudy.com ~]$
- 解压并安装软件包
[root@harbor188.westudy.com ~]$ mkdir docker
[root@harbor188.westudy.com ~]$ tar xf docker-ce-23_0_1.tar.gz -C docker/
[root@harbor188.westudy.com ~]$ yum localinstall -y docker/*.rpm
- 配置镜像加速并设置docker开机自启
[root@harbor188.westudy.com ~]$ mkdir -p /etc/docker
[root@harbor188.westudy.com ~]$ cat > /etc/docker/daemon.json <<'EOF'
{
"registry-mirrors": ["https://tuv7rqqq.mirror.aliyuncs.com"]
}
EOF
[root@harbor188.westudy.com ~]$ systemctl daemon-reload
[root@harbor188.westudy.com ~]$ systemctl restart docker
[root@harbor188.westudy.com ~]$
[root@harbor188.westudy.com ~]$ docker info | grep "Registry Mirrors" -A 1
Registry Mirrors:
https://tuv7rqqq.mirror.aliyuncs.com/
[root@harbor188.westudy.com ~]$
[root@harbor188.westudy.com ~]$ systemctl enable --now docker
- 安装docker-compose
# 添加epel源
[root@harbor188.westudy.com ~]$ curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
# 安装docker-compose
[root@harbor188.westudy.com ~]$ yum -y install docker-compose
[root@harbor188.westudy.com ~]$ docker-compose version
安装harbor
- 导入软件，并解压到指定目录
[root@harbor188.westudy.com ~]$ ls
anaconda-ks.cfg docker docker-ce-23_0_1.tar.gz harbor-offline-installer-v1.10.10.tgz
[root@harbor188.westudy.com ~]$ mkdir -pv /westudy/softwares
mkdir: created directory ‘/westudy’
mkdir: created directory ‘/westudy/softwares’
[root@harbor188.westudy.com ~]$ tar xf harbor-offline-installer-v1.10.10.tgz -C /westudy/softwares
- 证书
# 生成证书相关目录
[root@harbor188.westudy.com ~]$ mkdir -pv /westudy/softwares/harbor/certs/{ca,server,client}
# 生成私有CA的私钥
[root@harbor188.westudy.com /westudy/softwares/harbor/certs]$ openssl genrsa -out ca/ca.key 4096
# 生成私有CA的自签名证书
[root@harbor188.westudy.com /westudy/softwares/harbor/certs]$ openssl req -x509 -new -nodes -sha512 -days 3650 \
> -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=westudy.com" \
> -key ca/ca.key \
> -out ca/ca.crt
# 生成harbor188.westudy.com域名的私钥
[root@harbor188.westudy.com /westudy/softwares/harbor/certs]$ openssl genrsa -out server/harbor188.westudy.com.key 4096
# 通过harbor188.westudy.com域名的私钥向私有CA生成该域名的证书申请 (类似银行的填表)
[root@harbor188.westudy.com /westudy/softwares/harbor/certs]$ openssl req -sha512 -new \
> -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=harbor188.westudy.com" \
> -key server/harbor188.westudy.com.key \
> -out server/harbor188.westudy.com.csr
# 创建扩展配置文件
[root@harbor188.westudy.com /westudy/softwares/harbor/certs]$ cat > v3.ext <<'EOF'
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=westudy.com
DNS.2=westudy
DNS.3=harbor188.westudy.com
EOF
# 指定私有CA签发harbor188.westudy.com域名的证书 (类似银行正式发卡)
[root@harbor188.westudy.com /westudy/softwares/harbor/certs]$ openssl x509 -req -sha512 -days 3650 \
> -extfile v3.ext \
> -CA ca/ca.crt -CAkey ca/ca.key -CAcreateserial \
> -in server/harbor188.westudy.com.csr \
> -out server/harbor188.westudy.com.crt
# csr: Certificate Signing Request（证书签名请求）
# crt: Certificate（证书）
# 转换证书类型
[root@harbor188.westudy.com /westudy/softwares/harbor/certs]$ openssl x509 -inform PEM -in server/harbor188.westudy.com.crt -out server/harbor188.westudy.com.cert
# 拷贝私有CA的证书、harbor188.westudy.com的证书、私钥到client文件夹
[root@harbor188.westudy.com /westudy/softwares/harbor/certs]$ cp ca/ca.crt client/
[root@harbor188.westudy.com /westudy/softwares/harbor/certs]$ cp server/harbor188.westudy.com.{cert,key} client/
# 这里为什么要把harbor188.westudy.com的私钥也拷贝过去？
# 检查证书准备请客
[root@harbor188.westudy.com /westudy/softwares/harbor/certs]$ ls -l -R
.:
total 4
drwxr-xr-x 2 root root 48 May 16 15:17 ca
drwxr-xr-x 2 root root 87 May 16 15:21 client
drwxr-xr-x 2 root root 139 May 16 15:19 server
-rw-r--r-- 1 root root 272 May 16 15:14 v3.ext
./ca:
total 12
-rw-r--r-- 1 root root 2025 May 16 15:08 ca.crt
-rw-r--r-- 1 root root 3247 May 16 15:08 ca.key
-rw-r--r-- 1 root root 17 May 16 15:17 ca.srl
./client:
total 12
-rw-r--r-- 1 root root 2025 May 16 15:21 ca.crt
-rw-r--r-- 1 root root 2118 May 16 15:21 harbor188.westudy.com.cert
-rw-r--r-- 1 root root 3243 May 16 15:21 harbor188.westudy.com.key
./server:
total 16
-rw-r--r-- 1 root root 2118 May 16 15:19 harbor188.westudy.com.cert
-rw-r--r-- 1 root root 2118 May 16 15:17 harbor188.westudy.com.crt
-rw-r--r-- 1 root root 1716 May 16 15:12 harbor188.westudy.com.csr
-rw-r--r-- 1 root root 3243 May 16 15:10 harbor188.westudy.com.key
[root@harbor188.westudy.com /westudy/softwares/harbor/certs]$
- 初始化harbor服务
# 自定义过滤命令
[root@harbor188.westudy.com /westudy/softwares/harbor/certs]$ cd /westudy/softwares/harbor/
[root@harbor188.westudy.com /westudy/softwares/harbor]$ echo alias yy=\'egrep -v \"\^.*#\|\^\$\"\' >> /root/.bashrc
[root@harbor188.westudy.com /westudy/softwares/harbor]$ source /root/.bashrc
# 修改harbor.yaml
[root@harbor188.westudy.com /westudy/softwares/harbor]$ yy harbor.yml
hostname: harbor188.westudy.com # 修改主机名
...
certificate: /westudy/softwares/harbor/certs/server/harbor188.westudy.com.crt # 修改服务器的证书文件位置
private_key: /westudy/softwares/harbor/certs/server/harbor188.westudy.com.key # 修改服务器的私钥文件位置
harbor_admin_password: 1 # 修改登录密码
...
[root@harbor188.westudy.com /westudy/softwares/harbor]$ ./install.sh
- Windows访问测试
- 启停harbor服务
[root@harbor188.westudy.com /westudy/softwares/harbor]$ docker-compose down -v -t 1 # 停止harbor服务
[root@harbor188.westudy.com /westudy/softwares/harbor]$ docker-compose up -d # 启动harbor服务
- 配置docker客户端可以登陆harbor仓库
[root@harbor188.westudy.com /westudy/softwares/harbor]$ mkdir -p /etc/docker/certs.d/harbor188.westudy.com
[root@harbor188.westudy.com /westudy/softwares/harbor]$ cp /westudy/softwares/harbor/certs/client/* /etc/docker/certs.d/harbor188.westudy.com
# CA的证书、服务器的证书、服务器的私钥，都不能少，少了就无法登陆
[root@harbor188.westudy.com /westudy/softwares/harbor]$ docker login -u admin -p 1 harbor188.westudy.com # 登陆
[root@harbor188.westudy.com /westudy/softwares/harbor]$ docker logout harbor188.westudy.com # 登出
部署k8s
- 缓存rpm包
[root@k8s181.westudy.com ~]# yy /etc/yum.conf
[main]
cachedir=/var/cache/yum/$basearch/$releasever # 这是缓存目录
keepcache=1 # 把这个指定为1，也即是开启缓存功能
...
[root@k8s181.westudy.com ~]#
# yum localinstall 也会自动解决依赖包的问题，也就是说如果缺少依赖包，它依然会去下载
安装docker
# 添加docker相关源
[root@k8s181.westudy.com ~]# curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
[root@k8s181.westudy.com ~]# curl -o /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
[root@k8s181.westudy.com ~]# sed -i 's+download.docker.com+mirrors.tuna.tsinghua.edu.cn/docker-ce+' /etc/yum.repos.d/docker-ce.repo
[root@k8s181.westudy.com ~]# yum list docker-ce --showduplicates | sort -r
# 缓存docker相关的rpm包
[root@k8s181.westudy.com ~]# mkdir docker
[root@k8s181.westudy.com ~]# cd docker
[root@k8s181.westudy.com docker]# find /var/cache/yum/x86_64/7/ -name '*.rpm' -exec cp {} /root/docker \;
[root@k8s181.westudy.com docker]# tar zcf /root/docker-ce-20.10.24.tar.gz ./*
[root@k8s181.westudy.com ~]# tar tf docker-ce-20.10.24.tar.gz | wc -l
15
[root@k8s181.westudy.com ~]#
# 这里的rpm包不能通过时间来查找
# 将docker的rpm包传输给其他节点
[root@k8s181.westudy.com ~]# NODE='k8s182.westudy.com k8s183.westudy.com'
[root@k8s181.westudy.com ~]# for i in $NODE; do echo ===========; echo $i; scp docker-ce-20.10.24.tar.gz root@$i:/root; done
===========
k8s182.westudy.com
docker-ce-20.10.24.tar.gz 100% 99MB 70.4MB/s 00:01
===========
k8s183.westudy.com
docker-ce-20.10.24.tar.gz 100% 99MB 84.3MB/s 00:01
[root@k8s181.westudy.com ~]#
# 其他节点本地安装docker
[root@k8s182.westudy.com ~]$ mkdir docker
[root@k8s182.westudy.com ~]$ tar xf docker-ce-20.10.24.tar.gz -C docker/
[root@k8s182.westudy.com ~]$ yum localinstall -y docker/*.rpm
[root@k8s183.westudy.com ~]$ mkdir docker
[root@k8s183.westudy.com ~]$ tar xf docker-ce-20.10.24.tar.gz -C docker/
[root@k8s182.westudy.com ~]$ yum localinstall -y docker/*.rpm
# 所有节点优化docker
## 配置镜像加速
mkdir -pv /etc/docker &&
cat <<EOF | tee /etc/docker/daemon.json {
"registry-mirrors": ["https://tuv7rqqq.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
docker info | grep "Registry Mirrors" -A 1
## 传输证书
[root@harbor188.westudy.com ~]$ scp /westudy/softwares/harbor/certs/client/* 10.0.0.181:/etc/docker/certs.d/harbor188.westudy.com
# 先由harbor传给k8s181节点
[root@k8s181.westudy.com ~]# echo $NODE
k8s182.westudy.com k8s183.westudy.com
[root@k8s181.westudy.com ~]# for i in $NODE; do echo ===========; echo $i; scp /etc/docker/certs.d/harbor188.westudy.com/* root@$i:/etc/docker/certs.d/harbor188.westudy.com/; done
## 设置开机自启
systemctl enable --now docker
systemctl status docker
总共15个包
安装kubeadm，kubelet，kubectl
- 所有k8s集群节点(k8s181,k8s182,k8s183)
[root@k8s181.westudy.com ~]# yum -y install kubeadm-1.23.17-0 kubelet-1.23.17-0 kubectl-1.23.17-0
[root@k8s181.westudy.com ~]# systemctl enable --now kubelet
[root@k8s181.westudy.com ~]# systemctl status kubelet
[root@k8s182.westudy.com ~]$ yum -y install kubeadm-1.23.17-0 kubelet-1.23.17-0 kubectl-1.23.17-0
[root@k8s182.westudy.com ~]# systemctl enable --now kubelet
[root@k8s182.westudy.com ~]# systemctl status kubelet
[root@k8s183.westudy.com ~]$ yum -y install kubeadm-1.23.17-0 kubelet-1.23.17-0 kubectl-1.23.17-0
[root@k8s183.westudy.com ~]# systemctl enable --now kubelet
[root@k8s183.westudy.com ~]# systemctl status kubelet
初始化master节点
[root@k8s181.westudy.com ~]# kubeadm init --kubernetes-version=v1.23.17 --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.100.0.0/16 --service-cidr=10.200.0.0/16 --service-dns-domain=westudy.com # 这里要指定pod的网段和svc的网段以及域名
[init] Using Kubernetes version: v1.23.17
...
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.0.0.181:6443 --token ncu2tr.oe71idks5fg2env6 \
--discovery-token-ca-cert-hash sha256:8f85dba78215c5d43f277474f8b097bcea81ec66d4a308b2ca2018cc5ef58654
[root@k8s181.westudy.com ~]#
# 其他节点加入集群
[root@k8s182.westudy.com ~]$ kubeadm join 10.0.0.181:6443 --token ncu2tr.oe71idks5fg2env6 \
> --discovery-token-ca-cert-hash sha256:8f85dba78215c5d43f277474f8b097bcea81ec66d4a308b2ca2018cc5ef58654
[root@k8s183.westudy.com ~]$ kubeadm join 10.0.0.181:6443 --token ncu2tr.oe71idks5fg2env6 \
> --discovery-token-ca-cert-hash sha256:8f85dba78215c5d43f277474f8b097bcea81ec66d4a308b2ca2018cc5ef58654
# 查看集群情况
[root@k8s181.westudy.com ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s181.westudy.com Ready control-plane,master 35m v1.23.17
k8s182.westudy.com Ready <none> 9m34s v1.23.17
k8s183.westudy.com Ready <none> 9m31s v1.23.17
# 拷贝授权文件
[root@k8s181.westudy.com ~]# mkdir -p $HOME/.kube
[root@k8s181.westudy.com ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s181.westudy.com ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@k8s181.westudy.com ~]# kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true","reason":""}
[root@k8s181.westudy.com ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true","reason":""}
[root@k8s181.westudy.com ~]#
[root@k8s181.westudy.com ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s181.westudy.com NotReady control-plane,master 11m v1.23.17
[root@k8s181.westudy.com ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc && source ~/.bashrc
部署网络插件
# 部署网络插件
#[root@k8s181.westudy.com ~]# wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml 暂未验证最新版的flannel
[root@k8s181.westudy.com ~]# kubectl apply -f kube-flannel.yml
# 值得注意的是，Pod网络要和初始化的"10.100.0.0/16"一样，见下图
# 查看网络插件部署情况
[root@k8s181.westudy.com ~]# kubectl get pods -A -o wide| grep kube-flannel
kube-flannel kube-flannel-ds-dc4cb 1/1 Running 0 12m 10.0.0.183 k8s183.westudy.com <none> <none>
kube-flannel kube-flannel-ds-qqdrr 1/1 Running 0 22m 10.0.0.181 k8s181.westudy.com <none> <none>
kube-flannel kube-flannel-ds-w4q58 1/1 Running 0 12m 10.0.0.182 k8s182.westudy.com <none> <none>
部署资源测试
# 部署ds资源测试
[root@k8s181.westudy.com ~]# cat > westudy-k8s-ds.yaml <<'EOF'
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: westudy-ds
spec:
selector:
matchLabels:
school: westudy
class: k8s
template:
metadata:
labels:
school: westudy
class: k8s
spec:
containers:
- image: alpine
stdin: true
name: mylinux
EOF
[root@k8s181.westudy.com ~]#
[root@k8s181.westudy.com ~]# kubectl apply -f westudy-k8s-ds.yaml
[root@k8s181.westudy.com ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
westudy-ds-qdcwd 1/1 Running 0 12m 10.100.1.2 k8s182.westudy.com <none> <none>
westudy-ds-wfz64 1/1 Running 0 12m 10.100.2.2 k8s183.westudy.com <none> <none>
[root@k8s181.westudy.com ~]#
# 测试pod之间的通信
[root@k8s181.westudy.com ~]# kubectl exec westudy-ds-qdcwd -- ping -c4 10.100.2.2
PING 10.100.2.2 (10.100.2.2): 56 data bytes
64 bytes from 10.100.2.2: seq=0 ttl=62 time=0.706 ms
64 bytes from 10.100.2.2: seq=1 ttl=62 time=1.602 ms
64 bytes from 10.100.2.2: seq=2 ttl=62 time=3.838 ms
64 bytes from 10.100.2.2: seq=3 ttl=62 time=0.812 ms
--- 10.100.2.2 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.706/1.739/3.838 ms
[root@k8s181.westudy.com ~]#
469
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
