aws集群重启_通过AWS CDK部署本地Kubernetes集群

aws集群重启

前言 (Preface)

It has been a year after I assumed AWS Cloud Support Engineer. Looking back on the past 365 days, I feel none of them was unreasonably wasted. Peoples I met and cases/issues I worked on made me grow — just as Amazon’s philosophy, “it’s always Day 1”.

我担任AWS Cloud Support工程师已经一年了。 回顾过去的365天，我感到没有一个人被浪费掉。 我遇到的人和我从事的案例/问题使我得以成长-正如亚马逊的理念“永远都是第一天”。

But I have to say that I really don’t have free time to do side-projects during this year. Challenges filled every single days — bleeding edge technology, monthly goals, anxious users and urgent response times. Working as a Support is really different from working as a Developer. At least under most of circumstance, pressures from users won’t directly apply to developer. I spent most of my time learning non-technical skills to handle user’s issues more smoothly. For instance, effective negotiate and build trust with users.

但是我不得不说，今年我真的没有空闲时间来做附带项目。 每天都有挑战，包括尖端技术，每月目标，急切的用户和紧急响应时间。 作为支持人员与作为开发人员确实不同。 至少在大多数情况下，来自用户的压力不会直接作用于开发人员。 我大部分时间都在学习非技术技能，以便更顺畅地处理用户问题。 例如，有效地与用户协商并建立信任。

Now I’m getting little more used to working as a Support, so I start to have some free time to do some interesting things I’d like to do. In this topic, I’ll describe the progress that how do I use the AWS CDK to design a script to deploy a native Kubernetes(K8s) cluster on AWS. In addition to CDK, I also use kubeadm as the core to automate the whole deploying process. If you’d like to check the final script directly, please check the repository on my GitHub, kubeadm-CDK.

现在，我已经越来越不习惯用作支持人员了，所以我开始有一些空闲时间来做一些我想做的有趣的事情。 在本主题中，我将描述如何使用AWS CDK设计脚本以在AWS上部署本机Kubernetes(K8s)集群的进度。 除了CDK，我还使用kubeadm作为核心来自动化整个部署过程。 如果您想直接检查最终脚本，请检查我的GitHub上的存储库kubeadm-CDK 。

关于这个话题 (About this topic)

包括哪些信息？ (What information are included?)

• How do I design CDK script and shell scripts to deploy native K8s cluster, and how to make them work.
  我如何设计CDK脚本和Shell脚本以部署本机K8s集群，以及如何使其工作。
• Some technical issues(bombs💣) when I implement them.
  我实施这些技术时遇到了一些技术问题。
• Future works.
  未来的作品。

不包括哪些信息？ (What information are NOT included?)

• Basic of AWS CDK and AWS services. If you’d like to know that, please consider refer to AWS official documents and samples. Otherwise, if you buy AWS Support plans, you could look for AWS BD/TAM/SA’s assistance or you could also consider create a support cases for guidance(may be I’ll be the one provide you assistance in the case 🤣 ).
  AWS CDK和AWS服务的基础。 如果您想知道这一点，请考虑参考AWS官方文档和示例。 否则，如果您购买了AWS Support计划，则可以寻求AWS BD / TAM / SA的帮助，或者也可以考虑创建一个支持案例以进行指导(也许我会是在案例provide中为您提供帮助的人)。
• Basic of K8s.If you’d like to know that, please consider refer to Kubernetes official document.
  K8的基础知识。如果您想知道这一点，请考虑参考Kubernetes官方文档。
• How to use CDK to deploy EKS. This topic is about deploy NATIVE K8s cluster 🤣.
  如何使用CDK部署EKS。 本主题与部署NATIVE K8s集群有关。

1.如何设计CDK脚本和Shell脚本，以及如何使其工作。 (1. How do I design CDK script and shell scripts, and how to make them work.)

In my opinion, if I’d like to deploy an application or a service via CDK, I need to figure out:

我认为，如果我想通过CDK部署应用程序或服务，则需要弄清楚：

• What AWS services are required as infrastructure?
  需要哪些AWS服务作为基础架构？
• How to automate deployment process based on these infrastructure?
  如何基于这些基础架构自动化部署过程？

基础设施 (Infrastructure)

Basically, for first question, to deploy a K8s cluster, the following AWS resources are required:

基本上，对于第一个问题，要部署K8s集群，需要以下AWS资源：

• EC2 instances to serve as master and worker nodes.
  EC2实例充当主节点和工作节点。
• Isolated VPC and subnets to ensure EC2 instances won’t be effected by existing VPC related configurations.
  隔离的VPC和子网，以确保EC2实例不会受到现有VPC相关配置的影响。

Based on resource above, I also want to:

基于以上资源，我还想：

• Secure Control Plane to make administrators could only access it privately.
  安全控制平面使管理员只能私下访问它。
• Open least ports for both control plane and worker node. They could work properly under the settings, and keep cluster way from unexpected traffics as far as possible.
  为控制平面和辅助节点打开最少的端口。 它们可以在设置下正常工作，并尽可能地避免意外通信产生群集方式。

Therefore, the following resources are additionally required:

因此，还需要以下资源：

• An additional EC2 instance to serve as a bastion host. This bastion host should be the ONLY host could access all ports of all host on VPC.
  另一个EC2实例用作堡垒主机。 此堡垒主机应为唯一主机，可以访问VPC上所有主机的所有端口。
• Security groups to satisfies network security requirement.
  安全组满足网络安全要求。

Once making sure what resources are required, then with CDK’s help, basic deployment scripts could be constructed.

一旦确定需要什么资源，然后在CDK的帮助下，可以构建基本的部署脚本。

K8s集群部署 (K8s cluster deployment)

Now the problem is, how to deploy K8s cluster on EC2 instance automatically?

现在的问题是，如何在EC2实例上自动部署K8s集群？

There are many existing K8s cluster deployment approaches. For instance, kubespray, Rancher or Ubuntu Juju. After spending many efforts on testing and surveying for long time, I finally decided to use kubeadm. The reasons are:

现有许多K8s集群部署方法。 例如， kubespray ， Rancher或Ubuntu Juju 。 经过长时间的测试和勘测，我终于决定使用kubeadm。 原因如下：

• This tool is officially developed, maintained and supported by Kubernetes community.
  该工具由Kubernetes社区正式开发，维护和支持。
• I could gain more control over installation than other tools, but it also doesn’t require me to manage all details as directly install everything by myself.
  与其他工具相比，我可以获得更多的安装控制权，但是由于我自己直接安装了所有内容，因此它也不需要我管理所有细节。
• Few dependencies and configuration for dependencies are required.
  很少需要依赖项，也不需要配置依赖项。

• I could simply use shell script to automate deployment process with kubeadm. To install K8s cluster via kubeadm, I just need to follow instructions in kubeadm installation guide and cluster creation guide. Put these instructions in shell script and inject to EC2 user data looks simple.

  我可以简单地使用shell脚本通过kubeadm自动化部署过程。 要通过kubeadm安装K8s集群，我只需要按照kubeadm安装指南和集群创建指南中的说明进行操作 。 将这些指令放在shell脚本中并注入EC2用户数据看起来很简单。

Based on kubeadm’s workflow, CDK scrips and shell script should follow the order below:

根据kubeadm的工作流程，CDK脚本和shell脚本应遵循以下顺序：

But things are always not as simple as I thought … .

但是事情总是不像我想的那么简单……。

2.问题 (2. Issues)

💣问题1.安全组规则。 (💣Issue 1. Security group rule.)

My first problem is, Pods on Kubernetes cluster created by my script cannot access the internet. In general, after bootstrapping a K8s cluster, the most important thing is making sure network connectivity. It will effects applications and services could work properly or not. The problem is, if I create Ubuntu Pods on cluster, and perform commands like ping, curl in these Pod via kubectl exec, I could observe that traffics below couldn't be successfully established:

我的第一个问题是，由我的脚本创建的Kubernetes集群上的Pods无法访问Internet。 通常，在引导K8s群集后，最重要的是确保网络连接。 这将影响应用程序和服务能否正常工作。 问题是，如果我在集群上创建Ubuntu Pods，并通过kubectl exec在这些Pod中执行ping ， curl等命令，我会发现无法成功建立以下流量：

• Pod <-> Internet [ping IP address was allowed, but ping specific domain name failed]
  Pod <-> Internet [允许ping IP地址，但无法ping特定域名]
• Pod <-> Pod [ping both cluster IP or Service/Pod DNS name failed]
  Pod <-> Pod [ping集群IP或Service / Pod DNS名称失败]
• Pods were unable to resolve DNS records or even access CoreDNS Pod.
  Pod无法解析DNS记录，甚至无法访问CoreDNS Pod。

The first thing came up to my mind was security group. Actually, when I planned security groups rules, I just simply followed the port tables in kubeadm’s document. But one thing the document didn’t mention was, for cross-worker node traffic, ports would randomly be used.

我想到的第一件事是安全小组。 实际上，当我计划安全组规则时，我只是遵循kubeadm文档中的端口表。 但是该文件未提及的一件事是，对于跨工作节点流量，将随机使用端口。

It relates to how K8s network works. When Pods sits on different worker nodes, if these Pod would like to communicate with each other, CNI plugin on these worker node will convert source and destination IP address between cluster IP and host IP to ensure packets could reach destination. Under the situation, port in packets will still be retained. Therefore, if worker nodes don’t expose all ports to each other, then cross worker node traffic will be blocked, and connections between Pods cannot be established. That is the reason I add the function, __attach_inter_worker_access_rule() in security group setting.

它与K8s网络的工作方式有关。 当Pod位于不同的工作程序节点上时，如果这些Pod希望彼此通信，则这些工作程序节点上的CNI插件将在群集IP和主机IP之间转换源IP地址和目标IP地址，以确保数据包可以到达目标地址。 在这种情况下，仍保留包中的端口。 因此，如果工作程序节点未将所有端口彼此公开，则跨工作程序节点的通信将被阻止，并且Pod之间的连接无法建立。 这就是我在安全组设置中添加函数 __attach_inter_worker_access_rule()的原因。

After attaching the rule, it looked all network connectivity issues were gone. But the happiness didn’t last too long — when I deploy the CDK script again, different issue occurred.

附加规则后，看起来所有网络连接问题都消失了。 但是这种快乐并没有持续太久-当我再次部署CDK脚本时，发生了另外的问题。

💣问题2.节点异味问题。 (💣Issue 2. Node taint issue.)

The new circumstance was, network connectivity became unstable. The syndromes were the same as the ones I encountered previously, but with different behavior:

新的情况是，网络连接变得不稳定。 这些症状与我之前遇到的症状相同，但是行为不同：

• For Pod <-> Pod connectivity, connection intermittently lost. It means, if some Pods could connect to each other, but some couldn’t.
  对于Pod <-> Pod连接，连接间歇性丢失。 这意味着，如果某些Pod可以彼此连接，但有些则不能。
• For DNS resolving, failure also occurred intermittently. Some Pods could resolve DNS record and connect to CoreDNS Pods, but some couldn’t.
  对于DNS解析，也间歇性地发生故障。 有些Pod可以解析DNS记录并连接到CoreDNS Pod，但有些则不能。

So I had 2 choices: use TCPDump to analyze packets, or figure out what configurations led the situation. Finally, I chose the latter one. The reason was, clusters were constructed from scratch, so it shall not suffer from network issues. Analyze progress and setting I used was a good starting point.

因此，我有两个选择：使用TCPDump分析数据包，或找出导致这种情况的配置。 最后，我选择了后者。 原因是，群集是从头开始构建的，因此它不会遭受网络问题的困扰。 分析进度和设置是一个很好的起点。

My first decision was trying other CNI Plugins. In the beginning, the one I used was Flannel, a simple and reliable solution. However, the issue forced me try other options such as Weave Net and Calico, but they were still unable to solve it. Therefore, I thought CNI Plugin might not be the problem.

我的第一个决定是尝试其他CNI插件。 一开始，我使用的是Flannel ，这是一种简单可靠的解决方案。 但是，该问题迫使我尝试其他选择，例如Weave Net和Calico ，但他们仍然无法解决。 因此，我认为CNI插件可能不是问题。

Then I started thinking OS-level issue. Basically, the one I used to create instance was Ubuntu 20.04(Focal Fossa). Thought I could successfully bootstrap K8s cluster, according to kubeadm installation instructions, it looked Xenial(Ubuntu 16.04) should be the one the APT packages source the kubeadm packed for. However, changing OS version was still unable to solve the issue. I also tried Ubuntu 18.04(Bionic Beaver) but problem did still persist.

然后我开始考虑操作系统级别的问题。 基本上，我用来创建实例的是Ubuntu 20.04(Focal Fossa) 。 以为我可以成功引导K8s集群，根据kubeadm的安装说明 ，它看起来Xenial(Ubuntu 16.04)应该是kubeadm包装的APT软件包之一。 但是，更改操作系统版本仍无法解决该问题。 我也尝试了Ubuntu 18.04(Bionic Beaver)，但问题仍然存在。

As a result, I did the following things:

结果，我做了以下事情：

• Tried using different OS and CNI Plugins combinations.
  尝试使用不同的OS和CNI插件组合。
• Tried adding different wait and delay condition in shell scripts for worker nodes and master nodes.
  尝试在shell脚本中为工作节点和主节点添加不同的等待和延迟条件。

These attempts took me almost all 1-week after after-work times! The main reason was, CDK is based on AWS CloudFormation. Using it to creating and deleting resources is significantly SLOW. In my case, it took me at most half of hour to create EC2 instances(only EC2 instances!!!!💀💀💀💀💀)(wait/delate condition time hadn’t been counted!!!!💀💀💀💀💀).

这些尝试使我几乎在下班后的1周内都花了！ 主要原因是，CDK基于AWS CloudFormation。 使用它创建和删除资源的速度非常慢。 就我而言，我最多花了半个小时来创建EC2实例(仅EC2实例！！！💀💀💀💀💀)(没有计算等待/延迟条件时间！！！💀💀💀💀💀 )。

When I started to decide to give up the kubeadm, it occurred to me that I should verify nodes the Pods with network connectivity issue, and I found the root cause — those Pods were created on master nodes. Based on my security group rules(according to kubeadm recommendation), master node only allows traffics from worker nodes to access its 6443 port. Therefore, if Pod were scheduled on master node, being unable to establish connection with CoreDNS and Pods on worker nodes was expected.

当我开始决定放弃kubeadm时，我想到应该验证节点Pods的网络连接问题，并且找到了根本原因-这些Pod是在主节点上创建的。 根据我的安全组规则(根据kubeadm建议 )，主节点仅允许来自工作节点的流量访问其6443端口。 因此，如果将Pod安排在主节点上，则无法与工作节点上的CoreDNS和Pod建立连接。

However, according to kubeadm troubleshooting guide, by default, the node-role.kubernetes.io/master:NoSchedule taint will be applied to control-plane nodes. For that reason, Pods SHALL NOT able to be scheduled on master nodes. To verify that, I added the following settings to kubaadm configuration yaml file:

但是，根据kubeadm故障排除指南 ，默认情况下， node-role.kubernetes.io/master:NoSchedule污染将应用于控制平面节点。 因此，无法在主节点上调度Pod。 为了验证这一点，我将以下设置添加到kubaadm配置yaml文件：

# ...snip
 nodeRegistration:
 # ...snip
   taints:
   - key: "kubeadmNode"
     value: "master"
     effect: "NoSchedule"
 # ...snip

And it WOKRED!!!! Such a great document!!!! Thanks, Kubernetes!!!!!

它搞砸了！！！ 如此出色的文档！！！ 谢谢，Kubernetes !!!!!

Issue其他问题。 鲁克·塞夫 (💣Additional Issue. Rook Ceph)

To provide cluster created by script a persistent storage, I also add instructions to deploy Rook Ceph in scripts. But the problem is, no matter how many disk space did I allocate to EC2 instances, the ceph status command always throws the error that no disk space could be used(using rook-toolbox).

为了向脚本创建的集群提供持久性存储，我还添加了在脚本中部署Rook Ceph的说明。 但是问题是，无论我分配给EC2实例多少磁盘空间， ceph status命令始终会引发以下错误：无法使用任何磁盘空间(使用rook-toolbox )。

After verifying logs of each Pod created by Rook Ceph, I found messages mentioned that no disk could be mounted. Therefore, I tried attach additional disk to nodes in CDK scripts, and it WORKED 💀. However, I think I just solve the issue by dumb luck. To use Rook Ceph as persistent storage for K8s cluster, great understanding to Ceph is required. It will be one of my future work.

在验证了Rook Ceph创建的每个Pod的日志后，我发现消息提到无法安装磁盘。 因此，我尝试将其他磁盘附加到CDK脚本中的节点上，并且工作WO。 但是，我认为我只是靠运气来解决问题。 要将Rook Ceph用作K8s集群的持久存储，需要对Ceph有深入的了解。 这将是我未来的工作之一。

3.未来的工作。 (3. Future work.)

As I mentioned in preface, you could see the final scripts in my personal GitHub, kubeadm-CDK. If you encounter issues while deploying, please feel free to let me know via GitHub issues. I’d be glad to provide assistance.

正如我在序言中提到的，您可以在我的个人GitHub kubeadm-CDK中看到最终脚本。 如果您在部署时遇到问题，请随时通过GitHub问题通知我。 我很乐意提供协助。

Besides, here are what could be improve for the project:

此外，以下是该项目可以改进的地方：

• Allow the script to deploy multiple master nodes control plane.
  允许脚本部署多个主节点控制平面。
• More CNI plugin options.
  更多CNI插件选项。
• More persistent storage option(based Rook).
  更持久的存储选项(基于Rook)。
• Concrete IAM permission list for deployment.
  用于部署的具体IAM权限列表。

If you’re also interested in the project, you also star it on my GitHub. Besides, any advice and feedback are welcomed!

如果您也对该项目感兴趣，也可以在我的GitHub上对其加注星标。 此外，欢迎任何建议和反馈！

翻译自: https://medium.com/hallblazzar-開發者日誌/deploy-native-kubernetes-cluster-via-aws-cdk-ea0a9430e648

aws集群重启