K8S基于Containerd安装

田秋浩

已于 2023-04-01 09:59:59 修改

阅读量997

点赞数 1

分类专栏：运维管理文章标签： kubernetes linux vim

于 2023-03-28 16:50:07 首次发布

本文链接：https://blog.csdn.net/tianqiuhao/article/details/129817381

版权

运维管理专栏收录该内容

18 篇文章 0 订阅

订阅专栏

参考资料

参考资料连接：Kubernetes-Containerd简化搭建 - 晓风残月的博客

1.节点创建并完成初始化

1.1 节点要求

2核，2G内存，禁用交换分区。

1.2 安装ifconfig工具

yum search ifconfig

yum install -y net-tools.x86_64

1.3 安装工具包

yum install -y chrony yum-utils zlib zlib-devel openssl openssl-devel \
net-tools vim wget lsof unzip zip bind-utils lrzsz telnet

1.4 处理无法挂起问题

参考资料：https://blog.csdn.net/nathan8/article/details/113916533

setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

1.5 关闭防火墙

# 防火墙停止
systemctl stop firewalld.service
# 查询防火墙状态
systemctl status firewalld
# 关闭防火墙开启自启
systemctl disable firewalld

1.6 关闭selinux

1.2.3完成处理。

1.7 修改hostname

hostnamectl set-hostname master

备注：kubeadm默认会将hostname中的值作为节点的name注册到k8s中。

1.8 关闭swap

swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab

1.9 时间同步

systemctl enable chronyd --now
chronyc sources

1.10 安装生效ipvs

1.10.1 ipvs

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF

1.10.2 配置权限

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

1.10.3 安装ipset和ipvsadm

yum install -y ipset  ipvsadm

1.11 配置ipvs

modprobe br_netfilter

备注：

modprobe命令 用于智能地向内核中加载模块或者从内核中移除模块。

br_netfilter 透明防火墙(Transparent Firewall)又称桥接模式防火墙（Bridge Firewall）。简单来说，就是在网桥设备上加入防火墙功能。透明防火墙具有部署能力强、隐蔽性好、安全性高的优点。

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

1.11.1 验证

sysctl --system

1.11.2 参数说明

bridge-nf 使得 netfilter 可以对 Linux 网桥上的 IPv4/ARP/IPv6 包过滤，常用的选项包括：
net.bridge.bridge-nf-call-arptables：是否在 arptables 的 FORWARD 中过滤网桥的 ARP 包
net.bridge.bridge-nf-call-ip6tables：是否在 ip6tables 链中过滤 IPv6 包
net.bridge.bridge-nf-call-iptables：是否在 iptables 链中过滤 IPv4 包
net.bridge.bridge-nf-filter-vlan-tagged：是否在 iptables/arptables 中过滤打了 vlan 标签的包。

1.11.3 备注

每次重启服务器后需要执行 modprobe br_netfilter 命令。具体原理不太清楚。

1.12 生成快照

这里完成后生成一份master节点服务器的快照。

2.安装containerd

2.1 安装libseccomp

yum install libseccomp -y

2.2 创建containerd目录并进入

mkdir -p /usr/etc/containerd && cd /usr/etc/containerd

2.3 下载containerd压缩包

2.3.1 在线

wget https://download.fastgit.org/containerd/containerd/releases/download/v1.5.5/cri-containerd-cni-1.5.5-linux-amd64.tar.gz

2.3.2 离线

2.4 解压缩

tar -C / -xzf cri-containerd-cni-1.5.5-linux-amd64.tar.gz

2.5 修改配置

echo "export PATH=$PATH:/usr/local/bin:/usr/local/sbin" >> ~/.bashrc
source ~/.bashrc

2.5.1 说明

.bashrc，这个文件主要保存个人的一些个性化设置，如命令别名、路径等。

2.6 创建文件夹，这个是为了安装配置文件

mkdir -p /etc/containerd

2.7 设置开启自动启动

systemctl enable containerd --now

2.7.1 说明

2.7.1.1 资料来源

https://blog.csdn.net/weixin_39963341/article/details/111107237

2.7.1.2 systemctl 相关命令

systemctl 提供了一组子命令来管理单个的 unit，其命令格式为：
systemctl [command] [unit]
command 主要有：
start：立刻启动后面接的 unit。
stop：立刻关闭后面接的 unit。
restart：立刻关闭后启动后面接的 unit，亦即执行 stop 再 start 的意思。
reload：不关闭 unit 的情况下，重新载入配置文件，让设置生效。
enable：设置下次开机时，后面接的 unit 会被启动。
disable：设置下次开机时，后面接的 unit 不会被启动。
status：目前后面接的这个 unit 的状态，会列出有没有正在执行、开机时是否启动等信息。
is-active：目前有没有正在运行中。
is-enable：开机时有没有默认要启用这个 unit。
kill ：不要被 kill 这个名字吓着了，它其实是向运行 unit 的进程发送信号。
show：列出 unit 的配置。
mask：注销 unit，注销后你就无法启动这个 unit 了。
unmask：取消对 unit 的注销。

2.8 查看版本，确认安装成功

ctr version

响应如下图所示：

2.9 配置containerd

2.9.1 创建文件

vi /etc/containerd/config.toml

2.9.2 配置文件内容

disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "k8s.gcr.io/pause:3.5"
    selinux_category_range = 1024
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = true
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      no_pivot = false
      snapshotter = "overlayfs"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = ""
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = true

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
      # 在这个地方新增镜像仓库地址
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://kvuwuws2.mirror.aliyuncs.com"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
          endpoint = ["https://registry.aliyuncs.com/k8sxio"]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false

  [plugins."io.containerd.runtime.v1.linux"]
    no_shim = false
    runtime = "runc"
    runtime_root = ""
    shim = "containerd-shim"
    shim_debug = false

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/amd64"]

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]

  [plugins."io.containerd.snapshotter.v1.aufs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.btrfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.devmapper"]
    async_remove = false
    base_image_size = ""
    pool_name = ""
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.native"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.overlayfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.zfs"]
    root_path = ""

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0

2.10 启动containerd

systemctl daemon-reload
systemctl enable containerd --now

2.10.1 验证

crictl version

响应信息：

2.11 保存快照，containerd安装

3 安装kubeadm

3.1 配置阿里源镜像

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

3.2 安装工具组件，kubelat、kubectl、kubeadm

yum makecache fast

yum install -y kubelet-1.22.10 kubeadm-1.22.10 kubectl-1.22.10 --disableexcludes=kubernetes

        kubeadm是一个快捷搭建kubernetes(k8s)的安装工具，它提供了kubeadm init 以及 kubeadm join这两个命令来快速创建kubernetes集群。
        kubelet负责创建Pod，并管理这个Pod的生命周期，这里面包括：Pod的增删改查等操作。
        kubectl 是 Kubernetes 的命令行工具（CLI），是 Kubernetes 用户和管理员必备的管理工具。

3.2.1 验证

kubeadm version

响应：

3.3 设置开机启动

--加载新的unit 配置文件
systemctl daemon-reload 
systemctl enable --now kubelet

3.3.1 说明

在CentOS中，运行systemctl enable --now command可以在系统启动时启用服务，同时立即启动该服务。
systemctl enable --now xxx 和 systemctl enable xxx --now 都行

3.4 安装命令补全工具

yum install -y bash-completion
source <(crictl completion bash)
crictl completion bash >/etc/bash_completion.d/crictl
source <(kubectl completion bash)
kubectl completion bash >/etc/bash_completion.d/kubectl
source /usr/share/bash-completion/bash_completion

3.5 拉取镜像

3.5.1 拉取coredns

ctr -n k8s.io i pull docker.io/coredns/coredns:1.8.4
ctr -n k8s.io i tag docker.io/coredns/coredns:1.8.4 registry.aliyuncs.com/k8sxio/coredns:v1.8.4

3.5.2 拉取pause

ctr -n k8s.io i pull registry.aliyuncs.com/k8sxio/pause:3.5
ctr -n k8s.io i tag registry.aliyuncs.com/k8sxio/pause:3.5 k8s.gcr.io/pause:3.5

3.6 保存快照，kubeadm部署成功

4. 创建node节点

4.1 克隆三个node节点

4.2 启动后先执行ipv命令

注：所有节点

modprobe br_netfilter

4.3 修改hostname

4.3.1 node01输入

通过hostnamectl修改临时hostname （Transient hostname）

hostnamectl set-hostname node01

修改hostname文件修改静态hostname

> /etc/hostname && echo 'node01' > /etc/hostname

备注：执行

hostnamectl

可以看到如下信息：

其中Static hostname就是在/etc/hostname文件中保存的hostname，Transient hostname就是通过hostnamectl修改的hostname值。如果只通过hostnamectl修改，重启后hostname会变回Static hostname，所以需要两个地方都变。这样第一不需要重启服务，在文件中变更是需要重启服务才能生效的；第二如果需要重启也不需要再次变更。

4.3.2 node02输入

> hostnamectl set-hostname node02

/etc/hostname && echo 'node02' > /etc/hostname

4.3.3 node03输入

> hostnamectl set-hostname node03

/etc/hostname && echo 'node03' > /etc/hostname

4.3.4 确认

hostnamectl

显示内容4.3.1中有说明。

4.3.5 说明，可能出现如下提示，不影响安装。

4.4 配置host（所有节点都执行）

备注：注意修改ip

cat >> /etc/hosts << EOF 
masterIp   master
node01Ip   node01
node02Ip   node02
node03Ip   node03
EOF

5. 启动master节点

5.1 kubeadm常用命令

kubeadm init	用于搭建控制平面节点
kubeadm join	用于搭建工作节点并将其加入到集群中
kubeadm upgrade	用于升级 Kubernetes 集群到新版本
kubeadm config	如果你使用了 v1.7.x 或更低版本的 kubeadm 版本初始化你的集群，则使用kubeadm upgrade来配置你的集群
kubeadm reset	用于恢复通过 kubeadm init 或者 kubeadm join 命令对节点进行的任何变更
kubeadm certs	用于管理 Kubernetes 证书
kubeadm kubeconfig	用于管理 kubeconfig 文件
kubeadm version	用于打印 kubeadm 的版本信息
kubeadm alpha	用于预览一组可用于收集社区反馈的特性

5.2 配置配置文件镜像

5.2.1 切换目录

mkdir -p /usr/etc/kubeadm && cd /usr/etc/kubeadm

5.2.2 创建配置文件

vi kubeadm.yaml

5.2.3 配置文件内容

注意：在advertiseAddress位置需要改成你的master节点的ip

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.127.131 #1.指定master节点内网IP
  bindPort: 6443
nodeRegistration:
  criSocket: /run/containerd/containerd.sock    #2.修改为containerd的socket地址
  imagePullPolicy: IfNotPresent
  name: master    #3.设置为master的主机名称
  taints:
  - effect: "NoSchedule" #4.给master添加污点，master节点不能调度应用
    key: "node-role.kubernetes.io/master"
--- # 新增如下kubeproxy的配置
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs  # 修改kube-proxy 模式为ipvs，默认为iptables
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/k8sxio #修改成阿里云的image镜像地址
kind: ClusterConfiguration
kubernetesVersion: 1.22.10  #指定k8s版本号，默认这里忽略了小版本号
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.244.0.0/16  #添加 pod 子网地址
scheduler: {}
---
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd # 默认已经是`systemd`，最好再检查下,如果不是改过来
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging: {}
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

5.2.4 执行验证命令

 cat kubeadm.yaml | grep advertiseAddress

响应内容如下，注意这里的地址是master地址：

advertiseAddress: 192.168.127.131 #1.指定master节点内网IP

5.2.5 上传镜像

kubeadm config images list --config kubeadm.yaml

5.3 启动master

5.3.1 拉取镜像

备注：速度可能有点慢

kubeadm config images pull --config kubeadm.yaml

响应如下：

5.3.2 初始化

kubeadm init --config kubeadm.yaml

可能出现异常提示为：/proc/sys/net/bridge/bridge-nf-call-iptables does not exist，内容如下图：

执行下文命令：

modprobe br_netfilter

5.3.3 成功响应

5.3.4 记录node节点加入命令

kubeadm join 192.168.127.131:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:09704479d2f6546e01d787ddc05f8b1ecfdb59b69a488bfef96539fde5ec40af

5.3.5 执行成功提示中提到的命令

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
#如果root用户安装k8s需要执行以下命令，如果非root忽略
export KUBECONFIG=/etc/kubernetes/admin.conf

5.3.6 验证

kubectl get node

响应内容如下：

[root@master kubeadm]# kubectl get node
NAME     STATUS   ROLES                  AGE     VERSION
master   Ready    control-plane,master   2m41s   v1.22.10

6. 部署node节点

6.1 复制config文件

注意：下文中涉及masterIp，注意变更。所有node节点都需要执行

mkdir /root/.kube && scp root@masterIP:/root/.kube/config /root/.kube/

6.2 执行join命令

5.3.4中记录的命令

如果忘记在master执行下述命令重新获取

kubeadm token create --print-join-command

6.3 验证

kubectl get node

响应如下：

NAME     STATUS     ROLES                  AGE    VERSION
master   Ready      control-plane,master   10m    v1.22.10
node01   Ready      <none>                 2m2s   v1.22.10
node02   Ready      <none>                 67s    v1.22.10
node03   NotReady   <none>                 7s     v1.22.10

7.master节点安装网络插件

注：所有节点都需要执行

7.1 跳转目录

cd /usr/etc/kubeadm

7.2 创建文件

vi kube-flannel.yml

7.3 填入内容

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
  - configMap
  - secret
  - emptyDir
  - hostPath
  allowedHostPaths:
  - pathPrefix: "/etc/cni/net.d"
  - pathPrefix: "/etc/kube-flannel"
  - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni-plugin
        image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.2
        command:
        - cp
        args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        volumeMounts:
        - name: cni-plugin
          mountPath: /opt/cni/bin
      - name: install-cni
        image: quay.io/coreos/flannel:v0.15.0
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.15.0
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni-plugin
        hostPath:
          path: /opt/cni/bin
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg

7.4 运行

kubectl apply -f kube-flannel.yml # 安装 flannel 网络插件

有warning提示是正常的，响应如下：

7.5 验证，查看pod状态

kubectl get pod -nkube-system -owide

响应如下：

NAME                             READY   STATUS    RESTARTS   AGE   IP                NODE     NOMINATED NODE   READINESS GATES
coredns-7568f67dbd-8ph2v         1/1     Running   0          27m   10.88.0.2         master   <none>           <none>
coredns-7568f67dbd-sg9nm         1/1     Running   0          27m   10.88.0.3         master   <none>           <none>
etcd-master                      1/1     Running   0          27m   192.168.127.131   master   <none>           <none>
kube-apiserver-master            1/1     Running   0          27m   192.168.127.131   master   <none>           <none>
kube-controller-manager-master   1/1     Running   0          27m   192.168.127.131   master   <none>           <none>
kube-flannel-ds-2dnd6            1/1     Running   0          10m   192.168.127.131   master   <none>           <none>
kube-flannel-ds-mtjjw            1/1     Running   0          10m   192.168.127.136   node03   <none>           <none>
kube-flannel-ds-vhxfn            1/1     Running   0          10m   192.168.127.135   node02   <none>           <none>
kube-flannel-ds-x92q6            1/1     Running   0          10m   192.168.127.134   node01   <none>           <none>
kube-proxy-cn8gm                 1/1     Running   0          17m   192.168.127.136   node03   <none>           <none>
kube-proxy-f6px9                 1/1     Running   0          27m   192.168.127.131   master   <none>           <none>
kube-proxy-pr6lj                 1/1     Running   0          18m   192.168.127.135   node02   <none>           <none>
kube-proxy-zv427                 1/1     Running   0          19m   192.168.127.134   node01   <none>           <none>
kube-scheduler-master            1/1     Running   0          27m   192.168.127.131   master   <none>           <none>

备注：注意kube-flannel-ds类pod，这个启动需要时间。

8.node节点安装网络插件

8.1 复制kube-flannel.yml文件

mkdir -p /usr/etc/kubeadm && scp root@masterIP:/usr/etc/kubeadm/kube-flannel.yml /usr/etc/kubeadm

8.2 运行

kubectl apply -f kube-flannel.yml

有warning提示是正常的，响应如下：

8.3 验证，查看pod状态

kubectl get pod -nkube-system -owide

响应如下：

NAME                             READY   STATUS    RESTARTS   AGE   IP                NODE     NOMINATED NODE   READINESS GATES
coredns-7568f67dbd-8ph2v         1/1     Running   0          27m   10.88.0.2         master   <none>           <none>
coredns-7568f67dbd-sg9nm         1/1     Running   0          27m   10.88.0.3         master   <none>           <none>
etcd-master                      1/1     Running   0          27m   192.168.127.131   master   <none>           <none>
kube-apiserver-master            1/1     Running   0          27m   192.168.127.131   master   <none>           <none>
kube-controller-manager-master   1/1     Running   0          27m   192.168.127.131   master   <none>           <none>
kube-flannel-ds-2dnd6            1/1     Running   0          10m   192.168.127.131   master   <none>           <none>
kube-flannel-ds-mtjjw            1/1     Running   0          10m   192.168.127.136   node03   <none>           <none>
kube-flannel-ds-vhxfn            1/1     Running   0          10m   192.168.127.135   node02   <none>           <none>
kube-flannel-ds-x92q6            1/1     Running   0          10m   192.168.127.134   node01   <none>           <none>
kube-proxy-cn8gm                 1/1     Running   0          17m   192.168.127.136   node03   <none>           <none>
kube-proxy-f6px9                 1/1     Running   0          27m   192.168.127.131   master   <none>           <none>
kube-proxy-pr6lj                 1/1     Running   0          18m   192.168.127.135   node02   <none>           <none>
kube-proxy-zv427                 1/1     Running   0          19m   192.168.127.134   node01   <none>           <none>
kube-scheduler-master            1/1     Running   0          27m   192.168.127.131   master   <none>           <none>

9. 配置CNI

9.1 删除containerd配置的CNI

备注，只在node节点执行。

9.1.1 删除配置

mv /etc/cni/net.d/10-containerd-net.conflist{,.bak}

9.1.2 删除网络配置

ifconfig  cni0 down && ip link delete cni0

9.1.3 重启daemon

systemctl daemon-reload

9.1.4 重启containerd和kubelet

systemctl restart containerd kubelet

9.2 重建POD

备注：在master节点执行

9.2.1 重建 coredns

kubectl delete pod coredns-7568f67dbd-t56jc coredns-7568f67dbd-vb889 -nkube-system

9.2.2 查看Pod

kubectl get pod -A -o wide

10. 完成

田秋浩

关注

1
点赞
踩
9

收藏

觉得还不错? 一键收藏
打赏
0
评论
K8S基于Containerd安装

K8S搭建，基于Containerd，1个master节点，3个node节点。
复制链接

扫一扫

专栏目录