（k8s）kubernetes集群基于Containerd部署

最新推荐文章于 2024-09-27 11:19:34 发布

道法自然～

最新推荐文章于 2024-09-27 11:19:34 发布

阅读量1.3k

点赞数 13

分类专栏： kubernets 文章标签： kubernetes 容器云原生

本文链接：https://blog.csdn.net/richardlygo/article/details/142377690

版权

kubernets 专栏收录该内容

54 篇文章 4 订阅

订阅专栏

资源列表
基础环境
一、基础环境准备
- 1.1、关闭Swap分区
- 1.2、添加hosts解析
- 1.3、桥接的IPv4流量传递给iptables的链
二、准备Containerd容器运行时
- 2.1、安装Containerd
- 2.2、配置Containerd
- 2.3、启动Containerd
三、部署Kubernetes集群
- 3.1、安装Kubeadm工具
- 3.2、配置crictl工具
- 3.3、配置Containerd镜像加速器
- 3.4、测试crictl工具是否可用
- 3.5、初始化Master节点
- 3.6、Node节点加入集群
四、部署网络插件（CNI）
- 4.1、拉取必要镜像
- 4.2、部署网络插件
- 4.2、查看节点状态
- 4.3、查看组件状态
- 4.4、查看所有Pod状态
五、Containerd基本操作
- 4.1、Crictl镜像管理
- 4.2、Crictl容器管理
- 4.3、Crictl Pod管理

资源列表

操作系统	配置	主机名	IP
CentOS 7.9	2C4G	master	192.168.93.101
CentOS 7.9	2C4G	node1	192.168.93.102
CentOS 7.9	2C4G	node2	192.168.93.103

基础环境

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

关闭内核安全机制

setenforce 0
sed -i "s/^SELINUX=.*/SELINUX=disabled/g" /etc/selinux/config

修改主机名

hostnamectl set-hostname master
hostnamectl set-hostname node1
hostnamectl set-hostname node2

一、基础环境准备

三台主机都要操作，以master节点为例进行演示

1.1、关闭Swap分区

# 临时关闭
[root@master ~]# swapoff -a
# 永久关闭
[root@master ~]# sed -i 's/.*swap.*/#&/g' /etc/fstab

1.2、添加hosts解析

[root@master ~]# cat >> /etc/hosts << EOF
192.168.93.101 master
192.168.93.102 node1
192.168.93.103 node2
EOF

1.3、桥接的IPv4流量传递给iptables的链

# 加载 overlay 内核模块
[root@master ~]# modprobe overlay
# 加载 br_netfilter 模块
[root@master ~]# modprobe br_netfilter
[root@master ~]# cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
[root@master ~]# sysctl --system

二、准备Containerd容器运行时

三台主机都要操作，以master节点为例进行演示

2.1、安装Containerd

# 添加 docker 源，containerd也在docker源内的
[root@master ~]# cat <<EOF | sudo tee /etc/yum.repos.d/docker-ce.repo
[docker]
name=docker-ce
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/x86_64/stable/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
EOF


# 快速建立 yum 缓存
[root@master ~]# yum makecache fast


# 安装containerd
# 列出所有containerd版本
[root@master ~]# yum list containerd.io --showduplicates
[root@master ~]# yum -y install containerd.io-1.6.6-3.1.el7.x86_64

2.2、配置Containerd

# 生成配置文件
[root@master ~]# mkdir -p /etc/containerd
[root@master ~]# containerd config default | sudo tee /etc/containerd/config.toml 


# 修改/etc/containerd/config.toml文件中sandbox_image的值
[root@master ~]# grep 'sandbox_image' /etc/containerd/config.toml 
# 在原有的基础上进行修改，
    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"

2.3、启动Containerd

[root@master ~]# systemctl enable containerd
[root@master ~]# systemctl start containerd

三、部署Kubernetes集群

3.1、安装Kubeadm工具

所有节点都要操作

[root@master ~]# cat << EOF >> /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF


# 快速建立yum缓存
[root@master ~]# yum makecache fast


# 安装 kubectl：命令行管理工具、kubeadm：安装K8S集群工具、kubelet管理容器工具
[root@master ~]# yum install -y kubelet-1.27.0 kubeadm-1.27.0 kubectl-1.27.0


# 切记这个时候不要启动，只需要设置为开机自启
[root@master ~]# systemctl enable kubelet.service

3.2、配置crictl工具

crictl是CRI兼容的容器运行时命令行接口。你可以使用它来检查和调试Kubernetes节点上的容器运行时和应用程序。crictl和它的源代码在 cri-tools 代码库
更好 Containerd后，以上我们常用的docker命令也不再使用了，取而代之的分别是 crictl 和 ctr 两个命令行客户端
crictl是遵循CRI接口规范的一个命令行工具，通常用它来检查和管理kubelet节点上的容器运行时和镜像
ctr是containerd的一个客户端工具

# 所有节点都要操作
[root@master ~]# cat << EOF >> /etc/crictl.yaml
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10 
debug: false
EOF

3.3、配置Containerd镜像加速器

所有节点都要操作

# 在/etc/containerd/config.toml找到[plugins."io.containerd.grpc.v1.cri".registry.mirrors] ，默认在153行左右，在此行下面添加两行配置，阿里云的镜像加速器
[root@master ~]# vim /etc/containerd/config.toml 
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
   endpoint = ["https://i9h06ghu.mirror.aliyuncs.com"]


# 保存退出，刷新服务
[root@master ~]# systemctl restart containerd

3.4、测试crictl工具是否可用

# 以拉取一个Nginx镜像进行演示是否可用
[root@master ~]# crictl pull nginx:latest
Image is up to date for sha256:605c77e624ddb75e6110f997c58876baa13f8754486b461117934b24a9dc3a85


# 查看镜像
[root@master ~]# crictl images
IMAGE                     TAG                 IMAGE ID            SIZE
docker.io/library/nginx   latest              605c77e624ddb       56.7MB

3.5、初始化Master节点

在master节点上操作

# 生成配置文件
[root@master ~]# kubeadm config print init-defaults > kubeadm-init.yaml


# 修改kubeadm-init.yaml文件的advertiseAddress、name、imageRepository，添加Pod网络podSubnet
[root@master ~]# vim kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.93.101
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: master
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.27.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.244.0.0/16
scheduler: {}

# 初始化集群
[root@master ~]# kubeadm init --config=kubeadm-init.yaml
[init] Using Kubernetes version: v1.27.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W0629 11:07:42.117214    8814 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.0, falling back to the nearest etcd version (3.5.7-0)
W0629 11:08:11.311847    8814 checks.go:835] detected that the sandbox image "registry.aliyuncs.com/google_containers/pause:3.9" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master] and IPs [10.96.0.1 192.168.93.101]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [192.168.93.101 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [192.168.93.101 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
W0629 11:08:41.485522    8814 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.0, falling back to the nearest etcd version (3.5.7-0)
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 6.001513 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:
#####################################################################
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
#####################################################################
Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:
#####################################################################
kubeadm join 192.168.93.101:6443 --token abcdef.0123456789abcdef \
 --discovery-token-ca-cert-hash sha256:aa55ff35b12c120ff5a12098622122c77eff343d033a54d069d3e55364b4184d 
#####################################################################

# 初始化成功以后要根据提示执行以下3个命令，才可以操作集群
[root@master ~]# mkdir -p $HOME/.kube
[root@master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

3.6、Node节点加入集群

所有Node节点都要操作

# node1节点
[root@node1 ~]# kubeadm join 192.168.93.101:6443 --token abcdef.0123456789abcdef \
> --discovery-token-ca-cert-hash sha256:aa55ff35b12c120ff5a12098622122c77eff343d033a54d069d3e55364b4184d
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.


# node2节点
[root@node2 ~]# kubeadm join 192.168.93.101:6443 --token abcdef.0123456789abcdef \
> --discovery-token-ca-cert-hash sha256:aa55ff35b12c120ff5a12098622122c77eff343d033a54d069d3e55364b4184d
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.


# 如果加入集群的命令找不到了可以在master节点生成一个
[root@master ~]# kubeadm token create --print-join-command
kubeadm join 192.168.93.101:6443 --token 2d6hku.7esvsdqjfrvgkewg --discovery-token-ca-cert-hash sha256:aa55ff35b12c120ff5a12098622122c77eff343d033a54d069d3e55364b4184d

四、部署网络插件（CNI）

4.1、拉取必要镜像

所有节点都需要拉取以下两个镜像
在部署网络插件的过程中，实测是可能会有两个镜像拉取不来下，所以手动拉取一下，不过要使用魔法，拉取不到了评论或私信（开源免费）

# 以下是拉取镜像的命令，所有节点都需要有这两个镜像
[root@master ~]# crictl pull docker.io/flannel/flannel-cni-plugin:v1.1.2
[root@master ~]# crictl pull docker.io/flannel/flannel:v0.21.5

如果有镜像的话，使用以下命令把镜像导入即可，我的镜像名字是自己打的标签

[root@master ~]# ctr -n k8s.io images import flannel.tar
unpacking docker.io/flannel/flannel:v0.21.5 (sha256:6f2d991efb758c5530e7de90761dfb29637b7604a807d431312e20189e09f9e6)...done
[root@master ~]# ctr -n k8s.io images import flannel-cni-plugin-v1.1.2.tar
unpacking docker.io/flannel/flannel-cni-plugin:v1.1.2 (sha256:539d3bf046c8581557f0747dbad9d3b78a4de112d3c0bf9d291651593060fc9f)...done


# 查看镜像
[root@master ~]# crictl images | grep flannel
docker.io/flannel/flannel-cni-plugin                                          v1.1.2              7a2dcab94698c       8.25MB
docker.io/flannel/flannel                                                     v0.21.5             a6c0cb5dbd211       69.9MB
4.2、部署网络插件

4.2、部署网络插件

master节点操作即可
没有网络插件的私信或评论（免费开源）

[root@master ~]# kubectl apply -f kube-flannel.yaml 
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

4.3、查看节点状态

[root@master ~]# kubectl get node
NAME     STATUS   ROLES           AGE   VERSION
master   Ready    control-plane   15m   v1.27.0
node1    Ready    <none>          12m   v1.27.0
node2    Ready    <none>          12m   v1.27.0

4.4、查看组件状态

[root@master ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
controller-manager   Healthy   ok                              
scheduler            Healthy   ok                              
etcd-0               Healthy   {"health":"true","reason":""}

4.5、查看所有Pod状态

[root@master ~]# kubectl get pod -A
NAMESPACE      NAME                             READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-bkp7z            1/1     Running   0          2m5s
kube-flannel   kube-flannel-ds-q2565            1/1     Running   0          2m5s
kube-flannel   kube-flannel-ds-vz72p            1/1     Running   0          2m5s
kube-system    coredns-65dcc469f7-6x74z         1/1     Running   0          16m
kube-system    coredns-65dcc469f7-j8266         1/1     Running   0          16m
kube-system    etcd-master                      1/1     Running   0          16m
kube-system    kube-apiserver-master            1/1     Running   0          16m
kube-system    kube-controller-manager-master   1/1     Running   0          16m
kube-system    kube-proxy-2npvg                 1/1     Running   0          16m
kube-system    kube-proxy-8kqps                 1/1     Running   0          13m
kube-system    kube-proxy-zbc4h                 1/1     Running   0          13m
kube-system    kube-scheduler-master            1/1     Running   0          16m

五、Containerd基本操作

5.1、Crictl镜像管理

# 下载镜像
[root@master ~]# crictl pull nginx:latest

# 查看所有镜像
[root@master ~]# crictl images

# 查看镜像详细信息
[root@master ~]# crictl inspecti nginx:latest

# 删除镜像
[root@master ~]# crictl rmi nginx:latest

5.2、Crictl容器管理

# 列出运行中的容器，查看所有容器加-a选项
[root@master ~]# crictl ps

# 查看容器的详细信息
[root@master ~]# crictl inspect 07376dc402441

# 启动容器
[root@master ~]# crictl start 07376dc402441

# 停止容器
[root@master ~]# crictl stop 07376dc402441

# 删除容器
[root@master ~]# crictl rm 07376dc402441

# 打印日志
crictl logs <container-id 或 container-name>

# 进入容器执行命令
crictl exec -it <container-id 或 container-name> sh

5.3、Crictl Pod管理

# 打印所有 Pod
[root@node2 ~]# crictl pods
# 根据pod名称打印pod信息
[root@master ~]# crictl pods --name kube-proxy-trnjn

# 根据标签打印 Pod，crictl pods --label key=value
[root@master ~]# crictl pods --label k8s-app=kube-proxy