<img height='36pix' src='https://img.shields.io/badge/Kubernetes-v1.25.1-326CE5?style=for-the-badge&logo=kubernetes&logoColor=F5F5F5'> <img height='36pix' src='https://img.shields.io/badge/Ubuntu-22.04%20LTS-E95420?style=for-the-badge&logo=ubuntu&logoColor=F5F5F5'>
[toc]
准备练习环境
参考： [Kubernetes 文档](https://kubernetes.io/zh/docs/) / [入门](https://kubernetes.io/zh/docs/setup/) / [生产环境](https://kubernetes.io/zh/docs/setup/production-environment/) / [使用部署工具安装 Kubernetes](https://kubernetes.io/zh/docs/setup/production-environment/tools/) / [使用 kubeadm 引导集群](https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/) / [安装 kubeadm](https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)
<hr>
## B. 准备开始
- 一台兼容的 Linux 主机。Kubernetes 项目为基于 Debian 和 Red Hat 的 Linux 发行版以及一些不提供包管理器的发行版提供通用的指令
- 每台机器 `2 GB` 或更多的 RAM （如果少于这个数字将会影响你应用的运行内存)
- `2 CPU 核`或更多
- 集群中的所有机器的网络彼此均能相互连接(公网和内网都可以)
- 节点之中`不可以有重复的`主机名、MAC 地址或 product_uuid。请参见[这里](https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#verify-mac-address)了解更多详细信息。
- 开启机器上的某些端口。请参见[这里](https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#check-required-ports) 了解更多详细信息。
- 禁用交换分区。为了保证 kubelet 正常工作，你 **必须** `禁用交换分区`。
## U. 确保每个节点上 MAC 地址和 product_uuid 的唯一性
- 你可以使用命令 `ip link` 或 `ifconfig -a` 来获取网络接口的 MAC 地址
- 可以使用 `sudo cat /sys/class/dmi/id/product_uuid` 命令对 product_uuid 校验
一般来讲，硬件设备会拥有唯一的地址，但是有些虚拟机的地址可能会重复。 Kubernetes 使用这些值来唯一确定集群中的节点。 如果这些值在每个节点上不唯一，可能会导致安装 [失败](https://github.com/kubernetes/kubeadm/issues/31)。
<hr>
## V. <img height='36pix' src="https://www.vmware.com/content/dam/digitalmarketing/vmware/en/images/company/vmware-logo-grey.svg"> 虚拟机
> <kbd>新建...</kbd> / <kbd>创建自定虚拟机</kbd> /
> <kbd>Linux</kbd> / `Ubuntu 64位`
- 设置过程
| ID | 『虚拟机』设置 | 建议配置 | 默认值 | 说明 |
| :--: | :--------: | :--------------------: | :-----: | :----------: |
| 1 | 处理器 | - | 2 | 最低要求 |
| 2 | 内存 | - | 4096 MB | 节约内存 |
| 3 | 显示器 | 取消复选`加速 3D 图形` | 复选 | 节约内存 |
| 4 | 网络适配器 | - | nat | 需上网 |
| 5 | 硬盘 | `40`GB | 20 GB | 保证练习容量 |
| 6 | 选择固件类型 | UEFI | 传统 BIOS | VMware Fusion 支持嵌套虚拟化 |
<img src="https://img-blog.csdnimg.cn/f7aa3c4646974cfb96d669b4c26c9f87.png">
- 设置结果
| ID | Your computer's name | CPU 核 | RAM | DISK | NIC |
| :--: | :------------------: | :----------: | :------------: | :-------: | :-----: |
| 1 | `k8s-master` | 4 或更多 | 8 GB或更多 | 40 GB | nat |
| 2 | `k8s-worker1` | 同上 | 2 GB或更多 | 同上 | 同上 |
| 3 | `k8s-worker2` | 同上 | 同上 | 同上 | 同上 |
<hr>
## I. 安装 [Ubuntu 22.04 LTS](https://mirror.nju.edu.cn/ubuntu-releases/22.04/ubuntu-22.04.1-live-server-amd64.iso)
1. <div style="background: #D85E33; padding: 6px 12px; font-weight: bold; display: block; color: #fff; margin: -12px; margin-bottom: -12px; margin-bottom: 12px;" >Willkommen! Bienvenue! Welcome! Welkom!</div>
[ `English` ]
2. <div style="background: #D85E33; padding: 6px 12px; font-weight: bold; display: block; color: #fff; margin: -12px; margin-bottom: -12px; margin-bottom: 12px;" >Installer update available</div>
[ `Continue without updating` ]
3. <div style="background: #D85E33; padding: 6px 12px; font-weight: bold; display: block; color: #fff; margin: -12px; margin-bottom: -12px; margin-bottom: 12px;" >Keyboard configuration</div>
[ `Done` ]
4. <div style="background: #D85E33; padding: 6px 12px; font-weight: bold; display: block; color: #fff; margin: -12px; margin-bottom: -12px; margin-bottom: 12px;" >Choose type of install</div>
(`X`) Ubuntu Server (minimized)
/ [ `Done` ]
5. <div style="background: #D85E33; padding: 6px 12px; font-weight: bold; display: block; color: #fff; margin: -12px; margin-bottom: -12px; margin-bottom: 12px;" >Network connections</div>
[ `Done` ]
6. <div style="background: #D85E33; padding: 6px 12px; font-weight: bold; display: block; color: #fff; margin: -12px; margin-bottom: -12px; margin-bottom: 12px;" >Configure proxy</div>
[ `Done` ]
7. <div style="background: #D85E33; padding: 6px 12px; font-weight: bold; display: block; color: #fff; margin: -12px; margin-bottom: -12px; margin-bottom: 12px;" >Configure Ubuntu archive mirror</div>
Mirror address: http://mirror.nju.edu.cn/ubuntu
/ [ `Done` ]
8. <div style="background: #D85E33; padding: 6px 12px; font-weight: bold; display: block; color: #fff; margin: -12px; margin-bottom: -12px; margin-bottom: 12px;" >Guided storage configuration</div>
[ `Done` ]
9. <div style="background: #D85E33; padding: 6px 12px; font-weight: bold; display: block; color: #fff; margin: -12px; margin-bottom: -12px; margin-bottom: 12px;" >Storage configuration</div>
[ `Done` ]
10. <div style="background: #D85E33; padding: 6px 12px; font-weight: bold; display: block; color: #fff; margin: -12px; margin-bottom: -12px; margin-bottom: 12px;" >Profile setup</div>
Your name: `kiosk`
Your server 's name: `k8s-master`
Pick a username: `kiosk`
Choose a password: `ubuntu`
Confirm your password: `ubuntu`
/ [ `Done` ]
11. <div style="background: #D85E33; padding: 6px 12px; font-weight: bold; display: block; color: #fff; margin: -12px; margin-bottom: -12px; margin-bottom: 12px;" >SSH Setup</div>
[`X`] Install OpenSSH server
/ [ `Done` ]
12. <div style="background: #D85E33; padding: 6px 12px; font-weight: bold; display: block; color: #fff; margin: -12px; margin-bottom: -12px; margin-bottom: 12px;" >Featured Server Snaps</div>
[ `Done` ]
13. <div style="background: #D85E33; padding: 6px 12px; font-weight: bold; display: block; color: #fff; margin: -12px; margin-bottom: -12px; margin-bottom: 12px;" >Install complete!</div>
:a: [ `Cancel update and reboot` ]
:b: [ `Reboot Now` ]
14. 建议（可选）
关机后，做个快照
<hr>
## P. 准备工作
**[kiosk@k8s-master|k8s-worker1|k8s-worker2]$**
1. 设置当前用户sudo免密[选做]
> 不想每次都输入密码 - 加速
```bash
# 缓存 sudo 密码
echo ubuntu | sudo -v -S
#
sudo tee /etc/sudoers.d/$USER >/dev/null <<EOF
$USER ALL=(ALL) NOPASSWD: ALL
EOF
```
2. 使用国内镜像仓库[选做]
> 软件安装 - 加速
```bash
# 国内镜像仓库
MIRROR_URL=http://mirror.nju.edu.cn/ubuntu
# 生成软件仓库源
sudo tee /etc/apt/sources.list >/dev/null <<EOF
deb $MIRROR_URL jammy main restricted universe multiverse
deb $MIRROR_URL jammy-updates main restricted universe multiverse
deb $MIRROR_URL jammy-backports main restricted universe multiverse
deb $MIRROR_URL jammy-security main restricted universe multiverse
EOF
```
3. 安装相关软件<必做>
| ID | PKG | COMMENT |
| :--: | :-------------: | ---------------------------- |
| 1 | openssh-server | 远程 |
| 2 | vim | 编辑文件 |
| 3 | sshpass | ssh 免交互 |
| 4 | nfs-common | storageClass |
| 5 | bash-completion | <kbd>Tab</kbd> 自动补全 |
| 6 | netcat-openbsd | nc 命令 |
| 7 | open-vm-tools | GUI 下桌面分辨率；剪贴板共享 |
```bash
# 手动更新
sudo apt -y update
# 安装
sudo apt install -y openssh-server \
vim sshpass nfs-common \
bash-completion netcat-openbsd \
open-vm-tools
```
4. 设置静态IP<必做>
```bash
# 配置IP
NICN=$(ip a | awk '/^2:/ {print $2}' | sed 's/://')
NICI=$(ip a | awk '/inet / {print $2}' | grep -v ^127)
NICG=$(ip route | awk '/^default/ {print $3}')
sudo tee /etc/netplan/00-installer-config.yaml >/dev/null <<EOF
network:
ethernets:
$NICN:
dhcp4: false
addresses: [$NICI]
routes:
- to: default
via: $NICG
nameservers:
addresses: [8.8.8.8]
version: 2
EOF
# dns
sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
# active
sudo netplan apply
```
5. 编辑 hosts<必做>
```bash
sudo tee -a /etc/hosts >/dev/null <<EOF
192.168.147.131 k8s-master
192.168.147.132 k8s-worker1
192.168.147.133 k8s-worker2
EOF
```
6. 设置 root 密码[选做]
```bash
(echo ubuntu; echo ubuntu) | sudo passwd root
echo PermitRootLogin yes | sudo tee -a /etc/ssh/sshd_config
sudo systemctl restart sshd
```
**[kiosk@k8s-master]$**
7. ssh 免密[选做]
```bash
# 生成 keypair
ssh-keygen -f ~/.ssh/id_rsa -N ''
# 拷贝公钥
for i in k8s-master k8s-worker1 k8s-worker2; do
sshpass -pubuntu ssh-copy-id -o StrictHostKeyChecking=no kiosk@$i
sshpass -pubuntu ssh-copy-id -o StrictHostKeyChecking=no root@$i
done
```
**[kiosk@k8s-master|k8s-worker1|k8s-worker2]$**
8. 禁用 swap<必做>
```bash
# 交换文件
SWAPF=$(awk '/swap/ {print $1}' /etc/fstab)
# 立即禁用
sudo swapoff $SWAPF
# 永久禁用
sudo sed -i '/swap/d' /etc/fstab
# 删除交换文件
sudo rm $SWAPF
```
9. 扩容<必做>
```bash
# 逻辑卷名
export LVN=$(sudo lvdisplay | awk '/Path/ {print $3}')
# 扩容
sudo lvextend -l 100%PVS $LVN
# 立即生效
sudo resize2fs $LVN
# 验证
df -h /
```
10. 模块支持<必做>
```bash
# 安装
sudo apt -y install bridge-utils
# 立即生效
sudo modprobe br_netfilter
# 内核支持
sudo tee /etc/sysctl.d/k8s.conf >/dev/null <<EOF
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
EOF
# 立即生效
sudo sysctl -p /etc/sysctl.d/k8s.conf
```
11. 安装运行时<必做>
```bash
# 安装 containerd
sudo apt install -y containerd
# 创建目录
sudo mkdir /etc/containerd
# 生成默认配置文件
containerd config default | \
sudo tee /etc/containerd/config.toml >/dev/null
# 修改配置文件
sudo sed -i \
-e '/sandbox_image/s?k8s.gcr.io?registry.aliyuncs.com/google_containers?' \
-e '/SystemdCgroup/s?false?true?' \
-e '/registry.mirrors/a\ [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]' \
-e '/registry.mirrors/a\ endpoint = ["https://docker.nju.edu.cn/"]' /etc/containerd/config.toml
# 服务重启
sudo systemctl restart containerd
# 安装 crictl 命令
curl -# https://vmcc.xyz:8443/k8s/crictl-v1.24.2-linux-amd64.tar.gz \
-o crictl-v1.24.2-linux-amd64.tar.gz
tar -xf crictl-v1.24.2-linux-amd64.tar.gz
sudo cp crictl /usr/bin/
# crictl 配置文件
sudo tee /etc/crictl.yaml >/dev/null <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
pull-image-on-create: true
EOF
```
## K. 安装 K8s
**[kiosk@k8s-master|k8s-worker1|k8s-worker2]$**
12. 安装 kubeadm、kubelet 和 kubectl
```bash
# 更新 apt 包索引并安装使用 Kubernetes apt 仓库所需要的包
sudo apt -y install apt-transport-https ca-certificates curl
# 下载 Google Cloud 公开签名秘钥
curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -
# 添加 Kubernetes apt 仓库
MIRROR_URL=https://mirror.nju.edu.cn/kubernetes/apt/
sudo tee /etc/apt/sources.list.d/kubernetes.list >/dev/null <<EOF
deb $MIRROR_URL kubernetes-xenial main
EOF
# 更新 apt 包索引
sudo cp /etc/apt/trusted.gpg /etc/apt/trusted.gpg.d
sudo apt update -y
```
- CKA
```bash
sudo apt-cache madison kubelet | grep 1.24
# 安装 kubelet、kubeadm 和 kubectl 考试版本
sudo apt install -y kubelet=1.24.1-00 kubeadm=1.24.1-00 kubectl=1.24.1-00
```
- CKS, CKAD
```bash
sudo apt-cache madison kubelet | grep 1.25
# 安装 kubelet、kubeadm 和 kubectl 考试版本
sudo apt install -y kubelet=1.25.1-00 kubeadm=1.25.1-00 kubectl=1.25.1-00
```
```bash
# 锁定版本
sudo apt-mark hold kubelet kubeadm kubectl
```
13. k8s 支持
```bash
# 增加 k8s 支持
sudo sed -i '/ExecStart=\//s|$| --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --cgroup-driver=systemd|' \
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# 重启 kubelet 服务
sudo systemctl daemon-reload
sudo systemctl restart kubelet
```
**[kiosk@k8s-master]$**
14. 初始化
```bash
# 生成初始文件
sudo kubeadm config print init-defaults > kubeadm-config.yaml
# 修改文件
NICP=$(ip a | awk '/inet / {print $2}' | grep -v ^127 | sed 's+/24++')
sudo sed -i \
-e "/advertiseAddress/s?:.*?: $NICP?" \
-e "/name/s?:.*?: $(hostname -s)?" \
-e "/clusterName/s?:.*?: ck8s?" \
-e "/imageRepository/s?:.*?: registry.aliyuncs.com/google_containers?" kubeadm-config.yaml
# 使用初始文件，初始化集群
sudo kubeadm init --config kubeadm-config.yaml
```
> ...输出省略...<br>
> Your Kubernetes control-plane has initialized `successfully`!
>
> PS: 普通用户管理集群
>
> To start using your cluster, you need to run the following as a regular user:
>
> ```bash
> mkdir -p $HOME/.kube
> sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
> sudo chown $(id -u):$(id -g) $HOME/.kube/config
> ```
>
> PS：root 用户管理集群
>
> Alternatively, if you are the root user, you can run:
>
> ```bash
> export KUBECONFIG=/etc/kubernetes/admin.conf
> ```
>
> You should now deploy a pod network to the cluster.
> Run "`kubectl apply -f [podnetwork].yaml`" with one of the options listed at:
> https://kubernetes.io/docs/concepts/cluster-administration/addons/
>
> Then you can join any number of worker nodes by running the following on each as root:
>
> ```bash
> kubeadm join 192.168.147.128:6443 --token abcdef.0123456789abcdef \
> --discovery-token-ca-cert-hash sha256:c4781194de65ebb47984fc5e7e64d4897875410825ce4d18df81da1a298afa1f
> ```
15. 配置文件 - Client
```bash
# 创建目录
mkdir -p $HOME/.kube
# user 复制配置文件
sudo \cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# root 变量
sudo tee -a ~root/.bashrc >/dev/null <<EOF
export KUBECONFIG=/etc/kubernetes/admin.conf
EOF
```
16. 创建网络
```bash
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
kubectl apply -f https://rt.vmcc.xyz:8080/K8s/calico/v3.25.0/manifests/calico.yaml
```
17. 命令补全 - Client[建议]
```bash
$ kubectl completion --help
# 立即生效
source <(kubectl completion bash)
# 永久生效
mkdir ~/.kube 2>/dev/null
kubectl completion bash > ~/.kube/completion.bash.inc
printf "
# Kubectl shell completion
source '$HOME/.kube/completion.bash.inc'
" >> $HOME/.bashrc
source $HOME/.bashrc
```
18. 命令别名 - Client[建议]
```bash
# 网址 https://kubernetes.io/zh-cn/docs/reference/kubectl/cheatsheet/
# 永久生效
tee -a $HOME/.bashrc >/dev/null <<EOF
alias k='kubectl'
complete -F __start_kubectl k
EOF
# 立即生效
source $HOME/.bashrc
```
**[kiosk@k8s-worker1|k8s-worker2]$**
19. 加入集群
```bash
sudo \
kubeadm join 192.168.147.128:6443 \
--token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:c4781194de65ebb47984fc5e7e64d4897875410825ce4d18df81da1a298afa1f
```
## C. 确认环境正常
**[kiosk@k8s-master]**
```bash
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master `Ready` control-plane 9m17s `v1.25.1`
k8s-worker1 `Ready` <none> 90s `v1.25.1`
k8s-worker2 `Ready` <none> 51s `v1.25.1`
$ kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler `Healthy` ok
controller-manager `Healthy` ok
etcd-0 `Healthy` {"health":"true","reason":""}
$ kubectl -n kube-system get pod -w
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-798cc86c47-dkdjl 1/1 Running 0 4m5s
calico-node-ftwk8 1/1 Running 0 4m5s
calico-node-hstcg 1/1 Running 0 109s
calico-node-lcnw6 1/1 Running 0 2m28s
coredns-c676cc86f-mxpb8 1/1 Running 0 10m
coredns-c676cc86f-vhzzh 1/1 Running 0 10m
etcd-k8s-master 1/1 Running 0 10m
kube-apiserver-k8s-master 1/1 Running 0 10m
kube-controller-manager-k8s-master 1/1 Running 0 10m
kube-proxy-g2tz9 1/1 Running 0 109s
kube-proxy-j4fgc 1/1 Running 0 10m
kube-proxy-nz8vj 1/1 Running 0 2m28s
kube-scheduler-k8s-master 1/1 Running 0 10m
<Ctrl-C>
```
213
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
