使用kind测试集群联邦

最新推荐文章于 2024-06-17 08:30:00 发布

云天飞镜

最新推荐文章于 2024-06-17 08:30:00 发布

阅读量590

点赞数 1

分类专栏： kubefed kubernets 文章标签： kubernetes

本文链接：https://blog.csdn.net/u013488094/article/details/119949527

版权

kubernets 同时被 2 个专栏收录

4 篇文章 0 订阅

订阅专栏

kubefed

1 篇文章 0 订阅

订阅专栏

Kind 是 Kubernetes In Docker 的缩写，顾名思义是使用 Docker 容器作为 Node 并将 Kubernetes 部署至其中的一个工具。官方文档中也把 Kind 作为一种本地集群搭建的工具进行推荐。

安装二进制文件

安装kind

curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.11.1/kind-linux-amd64
chmod +x ./kind
mv ./kind /some-dir-in-your-PATH/kind

安装helm

参考 https://helm.sh/docs/intro/install/

wget https://get.helm.sh/helm-v3.6.3-linux-amd64.tar.gz
tar -xvzf helm-v3.6.3-linux-amd64.tar.gz
mv linux-amd64/helm /some-dir-in-your-PATH/helm
rm -rf linux-amd64
rm helm-v3.6.3-linux-amd64.tar.gz

安装kubectl

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.21.2/bin/linux/amd64/kubectl
chmod +x ./kubectl
mv ./kubectl /some-dir-in-your-PATH/kubectl

安装kubefedctl

wget https://ghproxy.com/https://github.com/kubernetes-sigs/kubefed/releases/download/v0.8.1/kubefedctl-0.8.1-linux-amd64.tgz
tar -xvzf kubefedctl-0.8.1-linux-amd64.tgz
mv kubefedctl /some-dir-in-your-PATH/kubefedctl

kind配置与介绍

使用 kind 创建 Kubernetes 集群非常的方便，只需要一行命令即可

kind create cluster

删除集群

kind delete cluster

以上操作的集群的默认名称是kind，可以使用--name参数指定集群名，更多创建的参数如下：

zxl@zxl:~$ kind create cluster  --help
Creates a local Kubernetes cluster using Docker container 'nodes'

Usage:
  kind create cluster [flags]

Flags:
      --config string       path to a kind config file
  -h, --help                help for cluster
      --image string        node docker image to use for booting the cluster
      --kubeconfig string   sets kubeconfig path instead of $KUBECONFIG or $HOME/.kube/config
      --name string         cluster name, overrides KIND_CLUSTER_NAME, config (default kind)
      --retain              retain nodes for debugging when cluster creation fails
      --wait duration       wait for control plane node to be ready (default 0s)

Global Flags:
      --loglevel string   DEPRECATED: see -v instead
  -q, --quiet             silence all stderr output
  -v, --verbosity int32   info log verbosity

--kubeconfig指定生成kubeconfig文件的路径

--name 指定集群的名字，当我们有多个k8s集群时，这很有用

--config用来指定启动的k8s集群的额外配置文件，必须是一个符合apiVersion: kind.x-k8s.io/v1alpha4 且kind: Cluster描述的一个yaml文件,如下：

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: cluster-name  #集群名
nodes: #节点配置，如下，启动一个master节点，一个worker节点
- role: control-plane   #master节点
  image: kindest/node:v1.21.1 #指定镜像，同 kind create cluster参数--image
  extraPortMappings:
  - containerPort: 6443
    hostPort: 45876
    listenAddress: "0.0.0.0"
    protocol: tcp #默认值，可不设置
- role: worker #

kind创建联邦集群

创建主集群

先创建一个k8s集群，作为master用来管理其他集群

kind create cluster --name fed-master

使用命令docker及kubectl命令查看如下：

zxl@zxl:~$ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS                       NAMES
853e3b89eddb   kindest/node:v1.21.1   "/usr/local/bin/entr…"   39 minutes ago   Up 39 minutes   127.0.0.1:45875->6443/tcp   fed-master-control-plane

kind@kind:~$ kubectl get node
NAME                       STATUS   ROLES                  AGE   VERSION
fed-master-control-plane   Ready    control-plane,master   40m   v1.21.1

kind@kind:~$ kubectl get pod -A
NAMESPACE            NAME                                               READY   STATUS    RESTARTS   AGE
kube-system          coredns-558bd4d5db-mqs7s                           1/1     Running   0          39m
kube-system          coredns-558bd4d5db-szvnc                           1/1     Running   0          39m
kube-system          etcd-fed-master-control-plane                      1/1     Running   0          39m
kube-system          kindnet-s4dxh                                      1/1     Running   0          39m
kube-system          kube-apiserver-fed-master-control-plane            1/1     Running   0          39m
kube-system          kube-controller-manager-fed-master-control-plane   1/1     Running   0          39m
kube-system          kube-proxy-hcj6h                                   1/1     Running   0          39m
kube-system          kube-scheduler-fed-master-control-plane            1/1     Running   0          40m
local-path-storage   local-path-provisioner-547f784dff-8qjgv            1/1     Running   0          39m

创建子集群

接下来我们需要创建两个子集群，用于创建联邦时使用，且子集群的kube-apiserver端口需要暴露供fed-master访问

kind create cluster --name fed-worker01
kind create cluster --name fed-worker02

因为kind未放开apiserver的外部访问，所以按照如下步骤执行，替换kubeconfig中的, 新建脚本change_server.sh如下:

#!/usr/bin/env bash

function usage() {
  echo "This script change kind k8s server by container ip and port."
  echo "Example: change_server.sh fed-worker01"
}
FED_CLUSTER_NAME=$1
if [[ -z "${FED_CLUSTER_NAME}" ]]; then
  usage
  exit 1
fi

container_ip=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "${FED_CLUSTER_NAME}-control-plane")
kubectl config set-cluster "kind-${FED_CLUSTER_NAME}" --server="https://${container_ip}:6443"

执行脚本，替换kubeconfig中的server

chmod +x change_server.sh
./change_server.sh fed-worker01
./change_server.sh fed-worker02

为方便访问查看，备份子集群的配置文件

kubectl config use-context kind-fed-worker01  #将kubectl的context切换到fed-worker01集群
cp ~/.kube/config ~/.kube/fed-worker01-config
kubectl config use-context kind-fed-worker02  #将kubectl的context切换到fed-worker02集群
cp ~/.kube/config ~/.kube/fed-worker02-config
alias fed01kubectl="kubectl --kubeconfig  ~/.kube/fed-worker01-config"
alias fed02kubectl="kubectl --kubeconfig  ~/.kube/fed-worker02-config"
kubectl config use-context kind-fed-master  #将kubectl的context切换到主集群

查看集群：

zxl@zxl:~$ kind get clusters
fed-master
fed-worker01
fed-worker02

zxl@zxl:~$ kubectl config use-context kind-fed-master
zxl@zxl:~$ kubectl get node
NAME                       STATUS   ROLES                  AGE    VERSION
fed-master-control-plane   Ready    control-plane,master   161m   v1.21.1

zxl@zxl:~$ fed01kubectl get node
NAME                         STATUS   ROLES                  AGE   VERSION
fed-worker01-control-plane   Ready    control-plane,master   38m   v1.21.1

zxl@zxl:~$ fed02kubectl get node
NAME                         STATUS   ROLES                  AGE     VERSION
fed-worker02-control-plane   Ready    control-plane,master   5m59s   v1.21.1

至次，集群准备完成

kubefed安装测试

使用helm安装kubefed到主集群

helm repo add kubefed-charts https://raw.githubusercontent.com/kubernetes-sigs/kubefed/master/charts
helm --namespace kube-federation-system upgrade -i kubefed kubefed-charts/kubefed --version=v0.8.1 --create-namespace

查看安装的资源

zxl@zxl:~$ kubectl -n kube-federation-system get pod
NAME                                          READY   STATUS    RESTARTS   AGE
kubefed-admission-webhook-5c96c86688-hkgbf    1/1     Running   0          4m1s
kubefed-controller-manager-74b97866c4-6g895   1/1     Running   0          2m49s
kubefed-controller-manager-74b97866c4-v784c   1/1     Running   0          2m46s

zxl@zxl:~$ kubectl get crd
NAME                                                 CREATED AT
clusterpropagatedversions.core.kubefed.io            2021-08-25T10:13:13Z
federatedclusterroles.types.kubefed.io               2021-08-25T10:13:12Z
federatedconfigmaps.types.kubefed.io                 2021-08-25T10:13:12Z
federateddeployments.types.kubefed.io                2021-08-25T10:13:12Z
federatedingresses.types.kubefed.io                  2021-08-25T10:13:12Z
federatedjobs.types.kubefed.io                       2021-08-25T10:13:12Z
federatednamespaces.types.kubefed.io                 2021-08-25T10:13:12Z
federatedreplicasets.types.kubefed.io                2021-08-25T10:13:12Z
federatedsecrets.types.kubefed.io                    2021-08-25T10:13:12Z
federatedserviceaccounts.types.kubefed.io            2021-08-25T10:13:12Z
federatedservices.types.kubefed.io                   2021-08-25T10:13:12Z
federatedservicestatuses.core.kubefed.io             2021-08-25T10:13:13Z
federatedtypeconfigs.core.kubefed.io                 2021-08-25T10:13:13Z
kubefedclusters.core.kubefed.io                      2021-08-25T10:13:13Z
kubefedconfigs.core.kubefed.io                       2021-08-25T10:13:13Z
propagatedversions.core.kubefed.io                   2021-08-25T10:13:13Z
replicaschedulingpreferences.scheduling.kubefed.io   2021-08-25T10:13:13Z

添加子集群到主集群kubefed的管控下：

kubefedctl join fed-worker01 --cluster-context kind-fed-worker01 --host-cluster-context kind-fed-master -v=2
kubefedctl join fed-worker02 --cluster-context kind-fed-worker02 --host-cluster-context kind-fed-master -v=2

查看添加的集群

zxl@zxl:~$ kubectl get kubefedclusters -A  #查看添加的集群
NAMESPACE                NAME           AGE   READY
kube-federation-system   fed-worker01   97s   True
kube-federation-system   fed-worker02   46s   True

kubefed架构介绍

如上，我们看到，在主集群部署Kubefed Controller Manger与Kubefed AdmissionWebhook，并创建了一系列的CRD，这种方式就是我们常说的Operator。

其架构如下：

主集群kube-apiserver监听来自api、kubectl、kubefedctl的请求、创建管理CRD资源
Kubefed AdmissionWebhook校验资源的有效性
Kubefed Controller Manger监听资源、组合数据，分发资源到各个子集群，并监听子集群资源状态，维护其生命周期

下图是kubefed的工作流程图：

Type配置：声明Kubefed需要处理的API类型
Cluster配置：声明Kubefed需要管理的目标集群
Propagation: 表示将资源分发到对应集群的版本信息等。
Templates: 定义资源在所有集群中共同的部分
Placement: 定义资源应该出现在哪些集群中
Overrides: 定义集群对Template字段的覆盖
Status: 收集被分发到所有集群的资源的状态
Policy: 决定一个资源应当被分发到联邦集群的什么子集中
Scheduling: 决定工作负载如何跨越不同集群分布

每一种联邦资源，都包含三个部分，分别是Template、Placement，Overrides；Kubefed Controller Manger根据这些信息，组装出分发往各个子集群的资源，并监听其状态，维护其生命周期

kubefed 相关CRD

Core CRD
- kubefedconfigs ：控制器的配置项，包含作用域、控制器选主配置、子集群健康检查、特性门控开关、SyncController、StatusController等
- kubefedclusters 声明Kubefed需要管理的目标集群,存储集群信息(Host、CA、Secret)
- federatedtypeconfigs （关联federatedType及targetType，控制能否传播）
- clusterpropagatedversions (记录集群资源的版本信息)
- propagatedversions (记录namespace资源的版本信息)
Types CRD
federatedxxxx.types.kubefed.io (声明Kubefed需要处理的API类型)
Schedulin CRD
replicaschedulingpreferences (提供了一种机制，来自动化的机制来维护所有集群成员中Deployment/Replicaset类型工作负载的副本总数)

zxl@zxl:~$ kubectl -n kube-federation-system get federatedtypeconfigs.core.kubefed.io
NAME                                     AGE
clusterroles.rbac.authorization.k8s.io   23h
configmaps                               23h
deployments.apps                         23h
ingresses.extensions                     23h
jobs.batch                               23h
namespaces                               23h
replicasets.apps                         23h
secrets                                  23h
serviceaccounts                          23h
services                                 23h

测试集群联邦

kubefedctl enable statefulsets  #创建默认不启用的federatedtypeconfigs
kubefedctl enable customresourcedefinitions #启用默认不启用的federatedtypeconfigs
#创建ns并联邦
zxl@zxl:~$ kubectl create ns ddcloud
zxl@zxl:~$ kubefedctl federate namespace ddcloud #会联邦namespace下的资源 使用--contents --skip-api-resources "configmaps,apps" 忽略指定的资源

查看新添加的联邦资源

zxl@zxl:~$ kubectl -n kube-federation-system get federatedtypeconfigs #可以看到新启用了customresourcedefinitions与statefulsets
NAME                                             AGE
clusterroles.rbac.authorization.k8s.io           39h
configmaps                                       39h
customresourcedefinitions.apiextensions.k8s.io   19s
deployments.apps                                 39h
ingresses.extensions                             39h
jobs.batch                                       39h
namespaces                                       39h
replicasets.apps                                 39h
secrets                                          39h
serviceaccounts                                  39h
services                                         39h
statefulsets.apps                                28s
#相应的，我们在子集群看到对应的ns已经创建
zxl@zxl:~$ fed01kubectl get ns ddcloud
NAME      STATUS   AGE
ddcloud   Active   2m35s
zxl@zxl:~$ fed02kubectl get ns ddcloud
NAME      STATUS   AGE
ddcloud   Active   2m40s
#查看联邦的federatednamespace
zxl@zxl:~$ kubectl get federatednamespace -A
NAMESPACE   NAME      AGE
ddcloud     ddcloud   11s

测试应用的部署与差异化配置

创建一个联邦的Deployment，fed-nginx.yaml如下：

apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: fed-nginx
  namespace: ddcloud
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - image: nginx
            name: nginx
  placement:  #选择应用可以部署的集群
    clusters: # or clusterSelector
    - name: fed-worker01
    - name: fed-worker02
  overrides:  #配置各个集群部署的deployment的差异化配置
  - clusterName: fed-worker01
    clusterOverrides:
    - path: "/spec/replicas"
      value: 2
    - path: "/spec/template/spec/containers/0/image"
      value: "nginx:1.17.0-alpine"
    - path: "/metadata/annotations"
      op: "add"
      value:
        foo: this_is_worker01
  - clusterName: fed-worker02
    clusterOverrides:
    - path: "/metadata/annotations"
      op: "add"
      value:
        foo: this_is_worker02

部署查看：

zxl@zxl:~$ kubectl apply -f fed-nginx.yaml #部署FederatedDeployment

zxl@zxl:~$ fed01ubectl -n ddcloud get deploy #查看子集群fed-worker01中的deploy
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
fed-nginx   1/1     1            1           2m17s
zxl@zxl:~$ fed01kubectl -n ddcloud get pod #查看子集群fed-worker02的pod
NAME                         READY   STATUS    RESTARTS   AGE
fed-nginx-8656cd9f7f-4t2zt   1/1     Running   0          48s
fed-nginx-8656cd9f7f-dww8t   1/1     Running   0          48s

zxl@zxl:~$ fed02ubectl -n ddcloud get deploy #查看子集群fed-worker02的deploy
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
fed-nginx   2/2     2            2           2m59s
zxl@zxl:~$ fed02kubectl -n ddcloud get pod
NAME                         READY   STATUS    RESTARTS   AGE
fed-nginx-6799fc88d8-st9dq   1/1     Running   0          55s

注解及镜像差异请自行查看Pod

ReplicaSchedulingPreference

控制Deployment或ReplicaSet副本数在各个集群中的权重、分布等

# 平均分配权重
apiVersion: scheduling.kubefed.io/v1alpha1
kind: ReplicaSchedulingPreference
metadata:
  name: fed-nginx
  namespace: ddcloud
spec:
  targetKind: FederatedDeployment
  totalReplicas: 2
  clusters:
    "*":
      weight: 1
---
# 指定各个集群的权重
apiVersion: scheduling.kubefed.io/v1alpha1
kind: ReplicaSchedulingPreference
metadata:
  name: fed-nginx
  namespace: ddcloud
spec:
  targetKind: FederatedDeployment
  totalReplicas: 3
  clusters:
    fed-worker02:
      weight: 1
    fed-worker02:
      weight: 2
---
# 指定各个集群的权重并设置最大、最小副本限制
apiVersion: scheduling.kubefed.io/v1alpha1
kind: ReplicaSchedulingPreference
metadata:
  name: fed-nginx
  namespace: ddcloud
spec:
  targetKind: FederatedDeployment
  totalReplicas: 3
  clusters:
    fed-worker02:
      minReplicas: 1
      maxReplicas: 2
      weight: 1
    fed-worker02:
      minReplicas: 1
      maxReplicas: 2
      weight: 2

参考：
kubefed
karmada

云天飞镜

关注

1
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
使用kind测试集群联邦

Kind 是 Kubernetes In Docker 的缩写，顾名思义是使用 Docker 容器作为 Node 并将 Kubernetes 部署至其中的一个工具。官方文档中也把 Kind 作为一种本地集群搭建的工具进行推荐。安装二进制文件安装kindcurl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.11.1/kind-linux-amd64chmod +x ./kindmv ./kind /some-dir-in-your-PATH/kind安装h
复制链接

扫一扫

专栏目录