Kubernetes中的虚拟集群简介

With the increasing adoption of Kubernetes within organizations, the need for Kubernetes access for applications and engineers is also growing. Since it is neither feasible nor cost-efficient to always use whole physical Kubernetes clusters, virtualization for Kubernetes is the obvious solution. In this article, I will describe an implementation of such a Kubernetes virtualization: Virtual clusters. I will also explain how virtual Kubernetes clusters work, how they can be used, and why they are a real alternative to current approaches for Kubernetes access.

随着组织内部越来越多地使用Kubernetes，对应用程序和工程师的Kubernetes访问的需求也越来越大。 由于始终使用整个物理Kubernetes集群既不可行也不具有成本效益，因此Kubernetes的虚拟化是显而易见的解决方案。 在本文中，我将描述这种Kubernetes虚拟化的实现：虚拟集群。 我还将解释虚拟Kubernetes集群如何工作，如何使用它们，以及为什么它们是当前Kubernetes访问方法的真正替代方案。

Kubernetes访问的当前方法 (Current Approaches For Kubernetes Access)

命名空间 (Namespaces)

The idea of virtualization within Kubernetes is not new: In the official Kubernetes documentation, namespaces are labeled “virtual clusters” that span a single physical cluster and provide a joint scope for related Kubernetes objects. With namespaces, it is possible to create separate environments for multiple apps and users in the same Kubernetes cluster.

Kubernetes中虚拟化的想法并不是什么新鲜事： 在Kubernetes官方文档中 ，命名空间被标记为“虚拟集群”，它们跨越单个物理集群，并为相关的Kubernetes对象提供了联合范围。 使用命名空间，可以为同一Kubernetes集群中的多个应用程序和用户创建单独的环境。

However, namespaces have some limitations: They cannot contain cluster scoped resources. While this may seem obvious, a lot of applications actually need to create or at least access cluster scoped resources like nodes, cluster roles, persistent volumes and storage classes. As soon as this happens, the application breaks its virtual namespace boundary and cannot be properly isolated from other applications anymore.

但是，名称空间有一些局限性：它们不能包含群集范围的资源。 尽管这似乎很明显，但实际上许多应用程序都需要创建或至少访问群集范围内的资源，例如节点，群集角色，持久卷和存储类。 一旦发生这种情况，该应用程序就会打破其虚拟名称空间边界，并且无法再与其他应用程序正确隔离。

Problems go even further if applications need to create their own custom resource definition or extend the API server via an APIService.

如果应用程序需要创建自己的自定义资源定义或通过APIService扩展API服务器，问题甚至会更加严重。

许多小型集群 (Many Small Clusters)

To solve these issues and create securely isolated ephemeral environments for applications during testing and development, the pattern of spinning up small, throw-away Kubernetes clusters has emerged.

为了解决这些问题并在测试和开发过程中为应用程序创建安全隔离的临时环境，出现了拆分小型，废弃的Kubernetes集群的模式。

Spinning up small, throw-away Kubernetes clusters solves the problem of coping with cluster scoped resources and isolation, but it is very cost-inefficient and negates one of the key advantages of Kubernetes itself: Being an orchestration system. Imagine the cost of a single cluster running 1000 containers vs 1000 Kubernetes clusters running a single container:

旋转丢弃的小型Kubernetes集群解决了解决集群范围的资源和隔离的问题，但它的成本效率非常低，并且否定了Kubernetes本身的主要优势之一：成为编排系统。 想象一下，运行1000个容器的单个群集与运行单个容器的1000个Kubernetes群集的成本：

• Each cluster has at least an additional API server, controller manager and etcd.
  每个集群至少具有一个附加的API服务器，控制器管理器和etcd。
• Each cluster needs at least one kubelet with kube-proxy, networking and container runtime.
  每个集群至少需要一个带有kube-proxy，网络和容器运行时的kubelet。
• Spinning up a new cluster takes way more time than starting a new container.
  与启动新容器相比，启动新集群要花费更多时间。

That is a lot of overhead, which can also result in a significant rise in your infrastructure bill (not even mentioning cluster management fees, such as in AWS and Google Cloud).

这是很多开销，也可能导致基础架构账单大幅增加(甚至没有提到集群管理费用，例如AWS和Google Cloud )。

Another solution is to extend namespaces and to virtualize Kubernetes itself.

另一个解决方案是扩展名称空间并虚拟化Kubernetes本身。

虚拟Kubernetes集群如何工作？ (How Do Virtual Kubernetes Clusters Work?)

The idea of virtualizing a Kubernetes cluster is similar to virtualizing a physical machine: The host system is used for actual computing, while everything else is emulated.

虚拟化Kubernetes集群的想法类似于虚拟化物理机：主机系统用于实际计算，而其他所有东西都是模拟的。

虚拟集群的现有解决方案 (Existing Solutions for Virtual Clusters)

There are already different implementations of the virtual cluster pattern in Kubernetes:

Kubernetes中已经存在虚拟集群模式的不同实现：

• loft vClusters: A closed source (but free for up to 3 users) implementation of virtual clusters.

  loft vClusters ：虚拟集群的封闭源( 最多3个用户免费 )实现。

• Multi-Tenancy SIG Virtual Cluster: An unreleased implementation of a virtual cluster operator for Kubernetes v1.16.

  多租户SIG虚拟集群 ： Kubernetes v1.16的虚拟集群运营商的未发布实现。

• k3v: A Proof of Concept of how virtual clusters could work with k3s.

  k3v ：有关虚拟集群如何与k3一起工作的概念证明。

This article will mostly talk about the implementation of virtual Kubernetes clusters (vClusters) with loft.

本文将主要讨论带有loft的虚拟Kubernetes集群(vClusters)的实现。

虚拟集群的技术实施 (Technical Implementation of Virtual Clusters)

The basic idea of a virtual cluster is to spin up a new Kubernetes cluster within an existing cluster and sync certain core resources between those two clusters.

虚拟集群的基本思想是在现有集群中启动一个新的Kubernetes集群，并在这两个集群之间同步某些核心资源。

A host cluster runs the actual virtual clusters pods and needs to be a fully working Kubernetes cluster. The virtual cluster itself only consists of the core Kubernetes components: API server, controller manager and etcd.

主机集群运行实际的虚拟集群Pod，并且需要是一个完全可用的Kubernetes集群。 虚拟集群本身仅由Kubernetes核心组件组成：API服务器，控制器管理器和etcd。

To reduce virtual cluster overhead for vClusters in loft, loft builds on k3s, which is a fully working, lightweight Kubernetes distribution that compiles the Kubernetes components into a single binary and disables all unnecessary Kubernetes features, such as the pod scheduler.

为了减少loft中的vClusters的虚拟集群开销，loft在k3s上构建， k3s是一个完全正常的轻量级Kubernetes发行版，它将Kubernetes组件编译为单个二进制文件，并禁用了所有不必要的Kubernetes功能，例如pod调度程序。

Besides k3s, there is a Kubernetes hypervisor that emulates a fully working Kubernetes setup in the virtual cluster. This component syncs certain virtual cluster resources to the host cluster and back:

除了k3之外，还有一个Kubernetes虚拟机管理程序，它可以在虚拟集群中模拟完全正常工作的Kubernetes设置。 该组件将某些虚拟集群资源同步到主机集群，然后再同步回去：

• Pods: All pods that are started in the virtual cluster are mutated and started in the namespace of the virtual cluster in the host cluster. Service account tokens, environment variables, DNS and other configurations are exchanged to point to the virtual cluster instead of the host cluster. Within the pod, it so seems that the pod is started in the virtual cluster instead of the host cluster.

  荚果：属于该虚拟集群中的所有启动荚突变，并在主机群集虚拟集群的命名空间开始。 服务帐户令牌，环境变量，DNS和其他配置被交换以指向虚拟群集而不是主机群集。 在Pod中，似乎Pod是在虚拟群集而不是主机群集中启动的。

• Services: All services and endpoints are mutated and created in the namespace of the virtual cluster in the host cluster. The virtual and host cluster share the same service cluster IPs.

  服务：所有服务和端点都在主机群集中虚拟群集的名称空间中进行了变异和创建。 虚拟群集和主机群集共享相同的服务群集IP。

• PersistentVolumeClaims: If persistent volume claims are created in the virtual cluster, they will be mutated and created in the namespace of the virtual cluster in the host cluster. If they are bound in the host cluster, the corresponding persistent volume will be synced back to the virtual cluster.

  PersistentVolumeClaims：如果在虚拟集群中创建了持久卷声明，则将在主机集群中的虚拟集群的名称空间中对其进行更改和创建。 如果它们绑定在主机群集中，则相应的持久卷将同步回虚拟群集。

• Others: Other resources such as configmaps, secrets, nodes, persistent volumes and storage classes are also synced between the clusters to assure correct functionality of pods.

  其他：其他资源(例如configmap，机密，节点，持久卷和存储类)也在群集之间同步，以确保Pod的正确功能。

Besides the synchronization of virtual and host cluster resources, the hypervisor also redirects certain Kubernetes API requests to the host cluster, such as port forwarding or pod/service proxying. It essentially acts as a reverse proxy for the virtual cluster.

除了虚拟和主机群集资源的同步之外，虚拟机管理程序还将某些Kubernetes API请求重定向到主机群集，例如端口转发或Pod /服务代理。 它实质上是虚拟集群的反向代理。

In the host cluster, all created resources by a virtual cluster are encapsulated in a single namespace (it is also possible to have multiple virtual clusters within a single namespace), which allows system admins to restrict resources of a virtual cluster via resource quotas. With this architecture, virtual clusters improve isolation:

在主机群集中，虚拟群集创建的所有资源都封装在单个名称空间中(也可以在单个名称空间中包含多个虚拟群集)，这允许系统管理员通过资源配额限制虚拟群集的资源。 通过这种架构，虚拟集群可以改善隔离度：

• Only certain namespaced resources are synced and available in the host cluster (such as Pods, Services, ConfigMaps etc.).
  主机群集中只有某些命名空间资源可以同步并可用(例如Pod，Services，ConfigMaps等)。
• Users and pods that need to communicate with the virtual cluster (such as operators) now communicate only with the virtual Kubernetes API server instead of the host Kubernetes API server.
  现在，需要与虚拟集群通信的用户和Pod(例如操作员)仅与虚拟Kubernetes API服务器通信，而不是与主机Kubernetes API服务器通信。
• Pods cannot directly access host cluster resources.
  Pod无法直接访问主机群集资源。
• Virtual cluster resources don’t pollute the host cluster etcd.
  虚拟集群资源不会污染主机集群etcd。

Since the virtual cluster is a working Kubernetes cluster itself, it is also even possible to install virtual clusters within virtual clusters.

由于虚拟集群本身就是一个可用的Kubernetes集群，因此甚至可以在虚拟集群中安装虚拟集群。

如何在阁楼中创建vClusters (How to Create vClusters in loft)

To test how virtual clusters work and if they could fit your use case, you can use loft, a multi-tenancy manager for Kubernetes. Virtual clusters should work in most Kubernetes clusters above version v1.14 that support persistent volume claims.

为了测试虚拟集群如何工作，以及他们是否能满足你的使用情况，您可以使用阁楼 ，一个多租户经理Kubernetes。 虚拟集群应该可以在支持持久卷声明的版本v1.14以上的大多数Kubernetes集群中工作。

You can set up loft for free on your local Kubernetes cluster or in any cloud provider via the following helm commands (see the official docs for more information):

您可以通过以下helm命令在本地Kubernetes集群或任何云提供商上免费设置阁楼(有关更多信息，请参见官方文档 )：

# Install ingress controller in the cluster 
helm install nginx-ingress nginx-ingress \
--repo https://kubernetes-charts.storage.googleapis.com \ 
--namespace nginx-ingress \ 
--set-string controller.config.hsts=false \ 
--create-namespace \ 
--wait # Install loft with self signed certificate 
# Change loft.localhost to your desired url 
# and make sure the URL points to the ingress 
# controller LoadBalancer external ip 
helm install loft loft --repo https://charts.devspace.sh/ \ 
--namespace loft \ 
--create-namespace \ 
--set admin.username=admin \ 
--set admin.password=admin \ 
--set certIssuer.create=false \ 
--set ingress.host=loft.localhost \ 
--set ingress.tls.secret=loft-cert \ 
--set cluster.connect.local=true \ 
--wait

Wait until loft is running and make sure you install the loft CLI and log in to your loft instance:

等待直到loft运行，并确保安装loft CLI并登录到loft实例：

# Login in the UI with admin:admin 
loft login https://loft.localhost --insecure

You can now create a new virtual cluster via the loft CLI in any namespace you like:

现在，您可以通过loft CLI在喜欢的任何名称空间中创建新的虚拟集群：

# if the namespace does not exist, loft will create it for you 
loft create vcluster test --space mynamespace

If the command ends successfully, loft will also switch your current local kube-context to access the virtual cluster, which you can test via kubectl:

如果命令成功结束，loft还将切换您当前的本地kube-context来访问虚拟集群，您可以通过kubectl进行测试：

kubectl get namespaces 
NAME            STATUS AGE 
default         Active 37s 
kube-system     Active 37s 
kube-public     Active 37s 
kube-node-lease Active 37s

You can now use the vCluster in the same way as any other Kubernetes cluster.

现在，您可以像使用任何其他Kubernetes群集一样使用vCluster。

For more information about how to manage vClusters in loft, make sure to check out the official loft documentation.

有关如何在放样中管理vClusters的更多信息，请确保查看 官方放样文档 。

虚拟集群的优势和局限性 (Advantages and Limitations of Virtual Clusters)

We think virtual clusters are an interesting new technology that can drastically reduce cost and effort for several use cases, such as ephemeral environments. Compared to the approach of creating many small independent clusters, virtual clusters have multiple advantages:

我们认为虚拟集群是一项有趣的新技术，可以极大地减少临时场景等多个用例的成本和工作量。 与创建许多小型独立集群的方法相比，虚拟集群具有多个优点：

• Less cluster boilerplate (one k3s pod in a shared host cluster vs a complete standalone Kubernetes cluster)
  更少的集群样板(共享主机集群中的一个k3s pod与完整的独立Kubernetes集群)
• Easier to manage (helm deploy/delete vs custom terraform scripts)
  易于管理(帮助部署/删除与自定义terraform脚本)
• Less startup and teardown time (seconds vs minutes)
  更少的启动和拆卸时间(秒数与分钟数)

While virtual clusters seem promising, they also have some limitations that should be taken into consideration:

尽管虚拟集群看起来很有前途，但它们也有一些局限性应予以考虑：

• Not all Kubernetes features work in virtual clusters (e.g. virtual storage classes, virtual container runtimes, network plugins etc.).
  并非所有Kubernetes功能都可以在虚拟集群中使用(例如，虚拟存储类，虚拟容器运行时，网络插件等)。
• Isolation between standalone clusters is obviously still better than between virtual clusters having the same host cluster.
  独立群集之间的隔离显然仍然比具有相同主机群集的虚拟群集之间的隔离要好。

For a more detailed analysis of benefits and use cases of virtual clusters, take a look at this article.

有关虚拟集群的好处和用例的详细分析，请参阅 本文 。

结论 (Conclusion)

Virtual clusters have the potential to become an important component in the Kubernetes ecosystem. Being more cost-effective and easier to manage than many small clusters while at the same time being better isolated than namespaces makes virtual clusters a superior solution for many use cases. Examples for this are scenarios in which engineers require access to Kubernetes such as testing, experimentation or cloud-native development. Virtual clusters could so help to foster Kubernetes diffusion in many organizations.

虚拟集群有可能成为Kubernetes生态系统的重要组成部分。 与许多小型集群相比，它更具成本效益并且更易于管理，同时比名称空间具有更好的隔离性，这使得虚拟集群成为许多用例的出色解决方案。 例如，工程师需要访问Kubernetes的场景，例如测试，实验或云原生开发。 虚拟集群可以帮助促进Kubernetes在许多组织中的扩散。

Originally published at https://loft.sh.

最初发布在 https://loft.sh 。

翻译自: https://medium.com/@fabiankramm/introduction-to-virtual-clusters-in-kubernetes-eb6bc55a16ff