kubernetes云平台_建立内部kubernetes平台

kubernetes云平台

An internal Kubernetes platform is the next step to get the full benefits of Kubernetes for your organization.

内部Kubernetes平台是为您的组织充分利用Kubernetes的下一步。

The container orchestration technology Kubernetes has become the dominant solution for cloud infrastructure and as such, it is maturing at an unrivaled pace. Many companies have already adopted Kubernetes or are in the process of it. However simple Kubernetes adoption is not enough, but you need a broader diffusion of Kubernetes in your organization.

容器编排技术Kubernetes已成为云基础架构的主要解决方案，因此它正在以无与伦比的速度成熟。 许多公司已经或正在采用Kubernetes。 然而， 仅仅采用Kubernetes是不够的 ，但是您需要在组织中更广泛地传播Kubernetes。

Some early adopters show what next steps are needed for this deeper adoption of Kubernetes: An internal Kubernetes platform.

一些早期采用者表明，要更广泛地采用Kubernetes，需要采取哪些下一步措施： 内部Kubernetes平台。

Spotify, Datadog (page 25), and Box are just some publicly available examples of companies that have built such platform for their engineering teams.

Spotify ， Datadog (第25页)和Box只是为公司的工程团队构建了这样的平台的公司的一些公开示例。

什么是内部Kubernetes平台 (What is an internal Kubernetes platform)

An internal Kubernetes platform is a platform that allows engineers to get a direct access to a Kubernetes environment on-demand for company-internal use.

内部Kubernetes平台是允许工程师按需直接访问Kubernetes环境以供公司内部使用的平台。

From an engineer’s perspective, the platform must provide self-service namespaces or another Kubernetes environment on-demand. It also needs to be easy to use even for engineers that are not experienced in working with Kubernetes. And finally, an internal Kubernetes platform must provide the engineers a real Kubernetes access, which means that it is possible for them to interact with Kubernetes and use Kubernetes-native components and resources. For this, it is not sufficient to just use a Platform-as-a-Service system that runs on top of Kubernetes or provide a Kubernetes test environment behind a CI/CD pipeline.

从工程师的角度来看，该平台必须按需提供自助名称空间或另一个Kubernetes环境。 即使对于没有使用Kubernetes经验的工程师来说，它也必须易于使用。 最后，内部Kubernetes平台必须为工程师提供真正的Kubernetes访问权限，这意味着他们可以与Kubernetes交互并使用Kubernetes本地组件和资源。 为此，仅使用在Kubernetes之上运行的平台即服务系统或在CI / CD管道后提供Kubernetes测试环境是不够的。

From an administrator perspective, an internal Kubernetes platform needs to be centrally controllable, so that the admins can oversee the whole system because with a company-wide roll-out the usage of it will be much higher than with some first adoption experiments. It is thus also necessary that the platform allows to manage and limit the users efficiently, so that the total cost of running the internal Kubernetes platform does not get out of hand.

从管理员的角度来看，内部Kubernetes平台需要是可集中控制的，以便管理员可以监视整个系统，因为在整个公司范围内部署该系统的使用率将比某些首次采用实验高得多。 因此，该平台还必须允许有效地管理和限制用户，以使运行内部Kubernetes平台的总成本不会失控。

A good internal Kubernetes platform combines all of these features in a single solution that supports all stakeholders in their further Kubernetes adoption.

良好的内部Kubernetes平台将所有这些功能结合在一个解决方案中，为所有利益相关者进一步采用Kubernetes提供支持。

为什么要有一个内部Kubernetes平台 (Why should you have an internal Kubernetes platform)

With the increasing amount of applications running on top of Kubernetes, it is also necessary that the adoption of Kubernetes spreads within organizations, so that developers can directly interact with the technology underlying their applications. Only if this requirement is fulfilled, the Kubernetes advantages of faster development cycles and enhanced application stability can be realized.

随着在Kubernetes上运行的应用程序数量的增加，还必须在组织内部推广Kubernetes的采用，以便开发人员可以直接与其应用程序底层的技术进行交互。 只有满足此要求， 才能实现Kubernetes具有更快的开发周期和增强的应用程序稳定性的优势 。

As can be seen in the Stack Overflow Developer Survey 2020, developers are ready for this next step as Kubernetes is a highly “wanted” and “loved” technology, i.e. developers want to work with it and the ones already doing so like it.

从2020年Stack Overflow开发人员调查中可以看出，开发人员已准备好下一步，因为Kubernetes是一种高度“受欢迎”和“喜爱”的技术，即开发人员希望与之合作，并且已经在使用这种技术。

The first step to enable engineers to work with Kubernetes is to provide them access to it with individual Kubernetes work environments. And this is exactly what an internal Kubernetes platform does, which is why it is so fundamental in a successful further adoption.

使工程师能够使用Kubernetes的第一步是为他们提供使用单个Kubernetes工作环境进行访问的权限。 这正是内部Kubernetes平台所做的事情，这就是为什么它在成功地被进一步采用时如此重要的原因。

存在什么样的Kubernetes环境来构建内部平台 (What kind of Kubernetes environments exist to build an internal platform)

Theoretically, there are several options to provide engineers a direct Kubernetes access: Local Kubernetes clusters, individual clusters, or shared clusters.

从理论上讲，有多种选择可为工程师提供直接的Kubernetes访问权限：本地Kubernetes集群，单个集群或共享集群。

1. Local Clusters: Local clusters are special Kubernetes versions made for running on the engineers’ computers, such as minikube or kind. As such, they are necessarily limited to the locally available computing resources and do not have all Kubernetes features that exist in the “real” Kubernetes running in cloud environments. The local runtime environment further makes it impossible to streamline the setup process, so that it needs to be done by the engineers themselves, which requires some k8s expertise.

1.本地集群：本地集群是特殊的Kubernetes版本，可以在工程师的计算机上运行，​​例如minikube或kind 。 因此，它们必然限于本地可用的计算资源，并且不具有在云环境中运行的“真实” Kubernetes中存在的所有Kubernetes功能。 本地运行时环境进一步使得无法简化设置过程，因此需要工程师自己来完成，这需要一些k8s专业知识。

For this, local Kubernetes clusters are well suited and very popular for the initial adoption and experimentation phase but they are not the right solution for further adoption steps or to build an internal platform.

为此，本地Kubernetes集群非常适合在初始采用和试验阶段使用，并且非常受欢迎，但它们不是进一步采用步骤或构建内部平台的正确解决方案。

2. Individual Clusters: Individual clusters are clusters that are only used by one engineer. It is possible to build an internal platform that provides your engineers full individual clusters. In principle, EKS, AKS, and GKE are already “external” Kubernetes platforms and if you gave every engineer access to these public cloud provider solutions, you would have something like an “internal” Kubernetes platform. However, such a solution would be very expensive as computing resources are used extremely inefficiently (the cluster management fees without computing resources would already be $70 per month per engineer). It is also much harder for admins to oversee a system with a huge number of clusters and to find out what is still used and what could be deleted. Finally, most companies would not want to give every engineer direct access to their cloud provider account.

2.单独的群集：单独的群集是仅由一名工程师使用的群集。 可以构建一个内部平台，为您的工程师提供完整的个人集群。 原则上，EKS，AKS和GKE已经是“外部” Kubernetes平台，如果让每个工程师都可以使用这些公共云提供商解决方案，那么您将拥有“内部” Kubernetes平台。 但是，这种解决方案将非常昂贵，因为计算资源的使用效率极低(不带计算资源的群集管理费用已经是每个工程师每月70美元)。 对于管理员来说，要监视具有大量群集的系统并找出仍在使用的内容和可以删除的内容的难度也要大得多。 最后，大多数公司都不想让每个工程师直接访问其云提供商帐户。

So, building an internal Kubernetes platform with individual clusters for engineers is theoretically possible but it would be very expensive and inefficient, which is why it will hardly be ever done in reality.

因此，从理论上讲，可以为工程师构建带有单个集群的内部Kubernetes平台，但这将是非常昂贵且效率低下的，这就是为什么实际上很难做到这一点。

3. Shared Clusters: Shared clusters are multi-tenant clusters that are used and shared with several engineers or teams. Shared clusters are real cloud-based Kubernetes clusters and so have basically endless computing resources and can be configured in the same way as the production system. At the same time, only one cluster is needed for a whole engineering team, which makes it easy to manage and control for admins and keeps the resource utilization high.

3.共享群集：共享群集是多租户群集，已被多个工程师或团队使用并共享。 共享集群是真正的基于云的Kubernetes集群，因此基本上具有无穷无尽的计算资源，并且可以与生产系统相同的方式进行配置。 同时，整个工程团队只需要一个集群，这使得管理员的管理和控制变得容易，并保持较高的资源利用率。

Shared clusters are the preferred (and only feasible) way to build an internal Kubernetes platform.

共享集群是构建内部Kubernetes平台的首选方法(也是唯一可行的方法)。

To get a more detailed comparison of the different types of Kubernetes environments, take a look at the comparison of local and remote clusters and the comparison of individual and shared clusters.

为了更详细地比较不同类型的Kubernetes环境，请看一下 本地和远程集群 的 比较以及单个集群和共享集群 的 比较 。

命名空间与vClusters (Namespaces vs. vClusters)

If you use a shared cluster environment for your internal platform, engineers will often work with namespaces, which are used as basis to establish multi-tenancy in Kubernetes. For many use cases, the separation of users via namespaces will be enough but there also are virtual Kubernetes clusters (vClusters) that provide a harder form of multi-tenancy and so more stability for your platform. vClusters have the additional advantage that they give the engineers the feeling of working with an individual cluster, so that the engineers can freely configure everything as they need it, for example, they can even independently choose which Kubernetes version they want to use.

如果您在内部平台上使用共享集群环境，则工程师通常会使用名称空间，这些名称空间是在Kubernetes中建立多租户的基础。 在许多用例中，通过名称空间分隔用户就足够了，但是还有虚拟的Kubernetes集群(vClusters) ，它们提供了更难的多租户形式，因此为您的平台提供了更高的稳定性。 vClusters的另一个优势是，它们使工程师可以使用单个群集，从而使工程师可以根据需要自由配置所有内容，例如，他们甚至可以独立选择要使用的Kubernetes版本。

手动与自助 (Manual vs. Self-Service)

Now, the question remains how to provide engineers access to the namespaces or vClusters on a shared cluster. One simple solution for this would be that the cluster admin manually creates and distributes the namespaces and vClusters for the engineers. However, this would create a highly problematic bottleneck that would be a huge productivity killer for engineers, as can be seen in a VMware survey which found that “waiting for central IT to provide access to infrastructure” is the number 1 impediment for developer productivity.

现在，问题仍然是如何为工程师提供对共享集群上的名称空间或vClusters的访问。 一种简单的解决方案是，集群管理员手动为工程师创建并分配名称空间和vClusters。 但是，这将产生严重问题的瓶颈，对工程师来说，这将是一个巨大的生产力杀手，从VMware调查中可以看出，该调查发现“等待中央IT提供对基础结构的访问”是阻碍开发人员生产力的第一大障碍。

For this, you will need a self-service platform that is easily useable, so developers can start working productively with it from the start and do not have to learn much about the Kubernetes details first.

为此，您将需要一个易于使用的自助服务平台，因此开发人员可以从一开始就开始使用该平台进行高效的工作，而不必首先了解太多有关Kubernetes的细节。

您如何构建内部Kubernetes平台 (How can you build an internal Kubernetes platform)

1.选择Kubernetes平台 (1. Choose the Kubernetes platform)

As previously described, you want to build your internal platform with shared clusters. Still, you need to decide on which Kubernetes platform your internal platform will run.

如前所述，您想使用共享集群构建内部平台。 尽管如此，您仍需要确定内部平台将在哪个Kubernetes平台上运行。

Here, you have to choose between a public cloud and a private cloud environment. The most popular public cloud environments are EKS (Amazon Web Service), AKS (Microsoft Azure), and GKE (Google Cloud).

在这里，您必须在公共云环境和私有云环境之间进行选择。 最受欢迎的公共云环境是EKS(Amazon Web Service) ， AKS(Microsoft Azure)和GKE(Google Cloud) 。

Alternatively, it is also possible to use a multi-cloud or hybrid-cloud approach, which combines several cloud providers or even public and private clouds. Special tools such as Rancher and OpenShift can be very useful to run this type of system.

或者，也可以使用多云或混合云方法，该方法将多个云提供商甚至公共云和私有云结合在一起。 诸如Rancher和OpenShift之类的特殊工具对于运行此类系统非常有用。

A good starting point for your internal Kubernetes platform is to use just a single environment that reflects the environment of your production system best. For example, if you use EKS for your Kubernetes applications in production, it makes sense to also start with EKS for your internal platform for development, testing, CI/CD,… If you already have a multi-cloud system in production, you need to evaluate if it makes sense to recreate this setup or if a single-cloud system might be enough.

内部Kubernetes平台的一个很好的起点是仅使用一个最能反映您的生产系统环境的环境。 例如，如果将EKS用于生产中的Kubernetes应用程序，那么也可以从EKS开始用于内部平台进行开发，测试，CI / CD等。如果您已经在生产中使用多云系统，则需要评估重新创建此设置是否有意义，或者单云系统是否足够。

2.定义平台架构 (2. Define the platform architecture)

As a next step, you need to determine how your platform should work and what the architecture should look like.

下一步，您需要确定平台应如何工作以及体系结构应如何。

One question in this area is how the platform users’ authentication will work as they will need accounts to access it. There are several options to care for this part of the user management. The simplest is to let admins create the user accounts centrally. This is a good option if you have a relatively small team and not so much fluctuation. In larger teams, it makes sense to let users sign-on themselves, e.g. by allowing users with a specific email address to create new accounts, or by implementing a Single-Sign-on (SSO) system that uses the user management of another service you already have in place, such as GitHub. For the implementation of such a system, you might take a look at dex which is an OpenID Connect provider that connects to many different services, such as LDAP, GitHub, GitLab, Google, Microsoft, and many others.

该领域的一个问题是平台用户的身份验证将如何工作，因为他们需要使用帐户进行访问。 有几个选项可用于照顾用户管理的这一部分。 最简单的方法是让管理员集中创建用户帐户。 如果您的团队规模较小且波动不大，那么这是一个不错的选择。 在较大的团队中，让用户自行登录是有意义的，例如，允许具有特定电子邮件地址的用户创建新帐户，或实施使用其他服务的用户管理的单点登录(SSO)系统您已经存在，例如GitHub。 对于这种系统的实现，您可以看看dex ，它是一个OpenID Connect提供程序，可连接到许多不同的服务，例如LDAP，GitHub，GitLab，Google，Microsoft等。

Another question that you need to answer is how the users of the internal platform will interact with it. Here, you can decide between a graphical UI, which might be easiest to work with for beginners but is also hardest to build, a CLI, that can potentially be well integrated in engineers’ workflows, or Kubernetes Custom Resource Definitions (CRDs), which would be the implementation that is closest to fundamental Kubernetes. Of course, it is also possible to combine the different options, e.g. by providing a GUI that creates CRDs in the background.

您需要回答的另一个问题是内部平台的用户将如何与其交互。 在这里，您可以选择可能最适合初学者使用但也最难构建的图形用户界面，可以很好地集成到工程师的工作流程中的CLI或Kubernetes自定义资源定义(CRD)，将是最接近基本Kubernetes的实现。 当然，例如通过提供在后台创建CRD的GUI，也可以组合不同的选项。

3.搭建平台 (3. Set up your platform)

The third step in building a Kubernetes platform is the actual setup. Here, you face a make-or-buy decision.

构建Kubernetes平台的第三步是实际设置。 在这里，您将面临买或买的决定。

The main benefit to build your own platform from scratch is that such a platform is perfectly customizable as you can determine every part of it. At the same time, this means that the setup requires a lot of work and time. You should also expect that you constantly have to improve your platform as your users’ needs and Kubernetes itself evolve. For this, choosing to build an own platform is mostly suited for organizations with very special needs, e.g. as they are working in a regulated industry, or for companies that are so large that some customization benefits pay off compared to the huge effort. Spotify, as an example, decided to build an own platform just for internal use. If you decide to go this way, you may still build upon existing solutions such as previously mentioned dex or kiosk, which is a multi-tenancy extension for Kubernetes.

从头开始构建自己的平台的主要好处是，您可以确定平台的每个部分，因此可以完全自定义该平台。 同时，这意味着设置需要大量的工作和时间。 您还应该期望随着用户需求和Kubernetes本身的发展，您必须不断改进平台。 为此，选择构建自己的平台最适合有特殊需求的组织(例如，他们在受监管的行业中工作)或规模如此之大的公司，以至于付出了巨大的定制收益后，他们就选择了。 例如，Spotify决定构建自己的平台供内部使用。 如果您决定采用这种方式，则仍可以基于现有解决方案，例如前面提到的dex或kiosk ，它是Kubernetes的多租户扩展。

For most companies, however, just buying an existing solution may be the better option because the setup is much easier and faster, you do not have to dedicate permanent development resources to improve the platform and because you will automatically get some best practices. At the same time, such a solution has only limited options for customization, which is why it may not be feasible to use for very specialized companies. One advanced off-the-shelf internal Kubernetes platform solution is loft. loft works with any Kubernetes cluster, it provides a GUI, a CLI and is fully based on Kubernetes CRDs. It also takes care of the user management, allows SSO and has some additional features such as a sleep mode to save cost by shutting down unused namespaces and vClusters.

但是，对于大多数公司而言，购买现有解决方案可能是更好的选择，因为安装过程更加轻松快捷，您不必花费永久的开发资源来改善平台，并且可以自动获得一些最佳实践。 同时，这种解决方案只有很少的定制选项，这就是为什么将其用于非常专业的公司可能不可行的原因。 loft是一种先进的现成的内部Kubernetes平台解决方案。 loft可与任何Kubernetes集群一起使用，它提供GUI和CLI，并且完全基于Kubernetes CRD。 它还负责用户管理，允许SSO并具有一些其他功能，例如通过关闭未使用的名称空间和vClusters来节省成本的睡眠模式。

4.随您的工程师并提供开发工具 (4. Onboard your engineers and provide dev tooling)

After you have set up your internal Kubernetes platform, you need to prepare the roll-out in your organization. Here, you should keep in mind that for many engineers who have not worked with Kubernetes before not only the platform but also the workflows will be new. For this, you should prepare an extensive documentation for both the platform and the new development/testing/deployment workflows. To make the transition as easy as possible for the engineers, it makes sense to pre-configure and to implement smart defaults as much as possible.

设置内部Kubernetes平台后，您需要在组织中准备推出。 在这里，您应该记住，对于许多没有使用Kubernetes的工程师来说，不仅平台，而且工作流程都是新的。 为此，您应该为平台和新的开发/测试/部署工作流准备大量的文档。 为了使工程师尽可能轻松地进行过渡，有必要进行预配置并尽可能实施智能默认设置。

Having the right developer tooling for the engineers is very helpful at this stage, especially as most of these tools can be pre-configured for your typical use cases. Examples for tools specifically made for developers are DevSpace, which also has an optional loft integration if this is your chosen platform, Skaffold, and Tilt.

在这一阶段，为工程师配备合适的开发人员工具非常有帮助，尤其是因为其中大多数工具都可以针对您的典型用例进行预配置。 DevSpace是专门为开发人员设计的工具的示例，如果您选择的平台是Skaffold和Tilt ，那么它还具有可选的loft集成 。

As a roll-out strategy, it is often a good idea to start with a single team and progress gradually. This allows you to educate the teams individually and to get insights about their main challenges. If you additionally measure the actual adoption and usage, you can further uncover unexpected challenges and resistances. With this information, you can then improve your system and your documentation step-by-step, so that the adoption process in the teams becomes easier over time.

作为推出策略，从一个团队开始并逐步发展通常是一个好主意。 这使您可以分别培训团队并获得有关其主要挑战的见解。 如果另外衡量实际采用和使用情况，则可以进一步发现意外的挑战和阻力。 有了这些信息，您就可以逐步改进系统和文档，从而使团队中的采用过程随着时间的推移变得更加容易。

5.控制成本 (5. Control Cost)

With increasing diffusion of Kubernetes in your organization, cost will become a more important factor because now, every engineer is permanently using a cloud-based Kubernetes environment. Since you are using a shared environment, the usage of the cloud resources should be generally relatively efficient. Still there are some areas of improvement that can reduce the cloud cost significantly.

随着组织中Kubernetes的扩散不断增加，成本将成为一个更重要的因素，因为现在，每个工程师都永久使用基于云的Kubernetes环境。 由于您使用的是共享环境，因此云资源的使用通常应该相对高效。 仍然有一些改进领域可以显着降低云成本。

To find these areas and to generally get a better understanding of your cost structure, e.g. which team causes which cost, you should monitor the cost. For this, tools such as Kubecost or Replex can be very helpful.

为了找到这些领域并大致上了解您的成本结构，例如哪个团队导致了哪个成本，您应该监视成本。 为此，诸如Kubecost或Replex之类的工具可能会非常有帮助。

You should also start with cost control measures very early on: The most important measure is to limit engineers in their computing resource usage. This will prevent excessive cost from mistakes by engineers. You should also activate horizontal auto-scaling for your cluster as this will adapt your computing power to the current needs.

您还应该尽早开始采用成本控制措施：最重要的措施是限制工程师的计算资源使用量。 这样可以防止工程师错误地增加成本。 您还应该为集群激活水平自动缩放 ，因为这将使您的计算能力适应当前的需求。

It is also a requirement for the implementation of a “ sleep-mode “, an automatic system that shuts down unused namespaces or vClusters and so prevents idle resources. The savings from such a sleep-mode in combination with horizontal auto-scaling can be enormous (around 70%) if you imagine that engineers are not needing the computing resources at night, on week-ends, or during meetings but you still pay for them if they keep running.

这也是实现“ 睡眠模式 ”的要求，该系统会关闭未使用的名称空间或vCluster，从而防止空闲资源。 如果您认为工程师在夜间，周末或会议期间不需要计算资源，但仍需付费，则这种睡眠模式与水平自动缩放相结合所节省的费用可观(约70％)他们，如果他们继续运行。

结论 (Conclusion)

If you have successfully adopted Kubernetes and you now want to take the next step, providing further engineers in your organization with a Kubernetes access is inevitable. The best solution for this is an internal Kubernetes platform that allows engineers to create Kubernetes work environments on-demand. Such a platform allows to standardize the use of Kubernetes within your organization, so even non-Kubernetes experts can work with it, while admins keep central control and oversight.

如果您已成功采用Kubernetes，现在又想采取下一步，那么不可避免的是为组织中的其他工程师提供Kubernetes访问权限。 最好的解决方案是内部Kubernetes平台，允许工程师按需创建Kubernetes工作环境。 这样的平台可以在您的组织内标准化Kubernetes的使用，因此即使非Kubernetes专家也可以使用它，而管理员则可以保持集中控制和监督。

No matter if you decide to build such a platform from scratch or if you simply buy an existing platform solution, the investment is worth it because only with an internal Kubernetes platform and access for every engineer you will get the full Kubernetes benefits, e.g. in form of faster development cycles, improved stability, and ultimately better software.

无论您是决定从头开始构建这样的平台，还是只是购买现有的平台解决方案，投资都是值得的，因为只有使用内部Kubernetes平台并为每个工程师提供访问权，您才能获得Kubernetes的全部好处，例如形式更快的开发周期，更高的稳定性以及最终更好的软件。

Photo by Clint Adair on Unsplash

Clint Adair在Unsplash上拍摄的照片

Originally published at https://loft.sh.

最初发布在 https://loft.sh 。

翻译自: https://levelup.gitconnected.com/building-an-internal-kubernetes-platform-f714a0b5ef72

kubernetes云平台