前言 (Preface)
Kubernetes is a lot of fun, has lots of features and usually supports most of one’s whims as a container orchestration tool in a straight-forward fashion.
Kubernetes很有趣，具有很多功能，通常以直截了当的方式支持大多数人的异想天开，作为容器编排工具。
However, one request from my direct manager had made me sweat during my attempts to achieve it: auto-scale pods according to a complicated logic criteria.
但是，我的直接经理提出的一个要求使我不知所措：根据复杂的逻辑标准自动缩放吊舱。
Trying to tackle the task, my online research yielded partial solutions, and I ran through so many brick walls trying to crack this one, that I had to write an article about it in order to avoid future confusion regarding this matter for all poor souls who might try to scale-up their micro-services on a criteria that’s not CPU/Mem.
为了解决这个问题，我的在线研究产生了部分解决方案，我遇到了很多砖墙，试图破解这个砖墙，因此我不得不写一篇关于它的文章，以避免将来对所有可怜的灵魂都感到困惑。可能会尝试以非CPU /内存的标准扩展其微服务。
挑战 (The Challenge)
It all started when we needed to scale one of our deployments according to the number of pending messages in a certain queue of RabbitMQ.That is a cool, not overly complicated task that can be achieved by utilizing Prometheus, Rabbitmq-exporter, and Prometheus-adapter together (hereinafter referred to as “the trio”).
一切都始于我们需要根据RabbitMQ的特定队列中待处理消息的数量来扩展我们的部署之一的时候，这是一项很酷的任务，但并不复杂，可以通过使用Prometheus，Rabbitmq-exporter和Prometheus-适配器一起(以下称为“三重奏”)。
With much enthusiasm and anticipation, I jumped right into the implementation only to later discover that one of my manager’s magic light-bulbs had switched on in his brain. It happens quite often, fortunately for him, and less fortunately for me as this usually means stretching the capabilities of the technology at hand with advanced and not-often-supported demands.
带着极大的热情和期望，我跳入了实现过程，后来才发现经理的一个神奇的灯泡已经在他的大脑中打开。 对于他而言，这种情况经常发生，这对他来说是幸运的，而对我而言，这不是幸运的，因为这通常意味着要扩展手头技术的功能，以满足先进且不受支持的需求。
He came up with a better, more accurate scaling criteria for our deployment. In a nutshell: measures how long a message has been waiting in queue “A” using the message’s timestamp, and then performs some logic to determine the final value of the metric, which is always returned as a positive integer.
他为我们的部署提出了更好，更准确的扩展标准。 简而言之：使用消息的时间戳测量消息在队列“ A”中等待了多长时间，然后执行一些逻辑以确定度量标准的最终值，该值始终以正整数形式返回。
Well, that’s nice and all, but as far as my knowledge extends, the trio mentioned above is not able to perform the advanced logic my manager desired. After all it relies solely on metrics that RabbitMQ exposes, so I was left to figure out a solution.
好吧，这很不错，但是就我所掌握的知识而言，上述三重奏无法执行我的经理想要的高级逻辑。 毕竟，它完全依赖RabbitMQ公开的指标，因此我不得不找出一个解决方案。
The experience from trying to implement the trio has helped me gain a better view on how the Horizontal Pod Autoscaler works and reads data from sources.
尝试实现三重奏的经验帮助我更好地了解了Horizontal Pod Autoscaler的工作原理以及如何从源读取数据。
As per the documentation, HPA works mainly against 3 APIs:
根据文档 ，HPA主要针对3个API：
- Metrics 指标
- Custom Metrics 自定义指标
- External Metrics 外部指标
My plan was to somehow harness the ‘custom metrics’ API and have it work against an internal application metrics API of our own, with the intention that the HPA would be able to read data from the internal API and scale accordingly.
我的计划是以某种方式利用“自定义指标” API，并使它与我们自己的内部应用程序指标API协同工作，以使HPA能够从内部API读取数据并据此进行扩展。
This API could, in the future, be extended and serve as an application-metric for other deployments that need scaling based on internal application metrics or any kind of metrics for that matter.
将来，可以扩展此API并将其用作其他部署的应用程序度量，这些其他部署需要基于内部应用程序度量或与此相关的任何类型的度量进行扩展。
This in essence involves the following tasks:
本质上，这涉及以下任务：
- Writing the code for our internal API 编写内部API的代码
- Creating a Kubernetes deployment and service for our internal API 为我们的内部API创建Kubernetes部署和服务
- Creating a Custom Metrics APIService in Kubernetes 在Kubernetes中创建自定义指标APIService
- Creating the HPA resource 创建HPA资源
And with that in mind, let’s get to work.
考虑到这一点，让我们开始工作。
Please note that for the sake of demonstration, I used the ‘custom-metrics’ namespace in all yaml definitions. However, it’s an arbitrary selection so feel free to deploy it anywhere you want.
请注意 ，为了演示起见，我在所有yaml定义中都使用了“ custom-metrics”命名空间。 但是，它是一个任意选择，因此可以随时将其部署到所需的任何位置。
编写内部API指标服务器 (Writing the Internal API Metrics Server)
Since web development isn’t a Devops specialty, I turned to the pros and asked my talented colleague Naama Yochai to help out with getting this part of the equation done. She’s the one in the team in charge of the deployment that we’re trying to scale, so she also had high-stakes in this project.
由于Web开发不是Devops的专长，因此我转向专业人士，并请我才华横溢的同事Naama Yochai帮助完成这部分工作。 她是我们正在尝试扩展的负责团队部署的团队中的一员，因此她在该项目中也占了上风。
Our basic requirement was a fast and simple web application, so we used JavaScript and the Express web-server, nothing fancy.
我们的基本要求是一个快速简单的Web应用程序，因此我们使用了JavaScript和Express Web服务器，这没什么花哨的。
For the sake of POC we only need to return some mock integer value and Naama would later on write the code performing the advanced logic behind the scaling criteria.
出于POC的考虑，我们只需要返回一些模拟整数值，Naama稍后便会编写代码，在缩放标准后面执行高级逻辑。
However, we had no idea how to return the results in a way that the HPA would understand. What’s the expected output when querying our API?We knew it was probably JSON, but had no idea of the structure nor the path on the web-server that HPA would search for when looking for that scale metric. Will it hit ‘/’ by default?
但是，我们不知道如何以HPA可以理解的方式返回结果。 查询API时的预期输出是什么？我们知道它可能是JSON，但不知道HPA在寻找该比例指标时将在Web服务器上搜索的结构或路径。 默认情况下会打“ /”吗？
To better understand how APIs in Kubernetes work, we needed to take a look at the ‘metrics’ API (implemented by ‘metrics-server’) that was already deployed in our cluster. This API gives you CPU/Mem metrics for pods and worker nodes.
为了更好地了解Kubernetes中的API如何工作，我们需要看一下已经在集群中部署的``指标''API(由``指标服务器''实现)。 该API为您提供Pod和辅助节点的CPU /内存指标。
When we described the service, we noticed the ‘Self Link’ value in the metadata of the service:
当我们描述服务时，我们注意到服务元数据中的“ Self Link”值：
This gave us a hint as to the path in which APIs are expected to work in Kubernetes.
这为我们提供了有关在Kubernetes中期望API起作用的路径的提示。
Once the root path was discovered, we created a small Docker container running NodeJS and ended up having a simple HTTP server running in port 6443.
发现根路径后，我们创建了一个运行NodeJS的小型Docker容器，最终在端口6443上运行了一个简单的HTTP服务器。
However, we soon found the first chink in the armor: the Kubernetes API works exclusively with HTTPS web-servers (namely port 443) and cannot use insecure HTTP.
但是，我们很快发现了第一个缺点：Kubernetes API仅与HTTPS Web服务器(即端口443)一起使用，并且不能使用不安全的HTTP。
A couple of code tweaks later, and viola! The server was working in a secure protocol and listening on the same port.
稍后修改了两个代码，中提琴！ 服务器正在以安全协议运行，并在同一端口上进行侦听。
Here’s the Dockerfile we used:
这是我们使用的Dockerfile：
FROM ubuntu:18.04# Install some debugging/editor toolsRUN apt-get update --fix-missing && \
apt-get install -y \
build-essential \
net-tools \
iputils-ping \
vim \
curl \
nginx \
wget -yRUN curl -sL https://deb.nodesource.com/setup_10.x | bash -
RUN apt-get install nodejs -yWORKDIR /metric-exporterCOPY . /metric-exporterRUN npm installEXPOSE 6443
ENTRYPOINT [“/usr/bin/node”]
CMD [“./index.js”]创建Kubernetes部署和服务 (Creating a Kubernetes Deployment & Service)
Now that the basic code outline for the internal API was ready, we proceeded with creating a deployment.
现在已经准备好内部API的基本代码大纲，我们继续创建部署。
The metrics-exporter.yaml deployment file is fairly simple and looks like this:
metrics-exporter.yaml部署文件非常简单，如下所示：
apiVersion: apps/v1
kind: Deployment
metadata:
name: metrics-exporter
spec:
replicas: 1
selector:
matchLabels:
app: metrics-exporter
template:
metadata:
labels:
app: metrics-exporter
spec:
containers:
- image: our-company/metrics-exporter
name: metrics-exporter
ports:
- containerPort: 6443Once the pod was up and running, it needed to be backed-up by a corresponding service, so we applied the following metrics-exporter-service.yaml file:
Pod启动并运行后，需要由相应的服务进行备份，因此我们应用了以下metrics-exporter-service.yaml文件：
apiVersion: v1
kind: Service
metadata:
name: metrics-exporter
spec:
ports:
- port: 443
protocol: TCP
targetPort: 6443
selector:
app: metrics-exporterNotice that the service is configured to run on port 443, but the container port is the aforementioned 6443 port.
请注意，该服务配置为在端口443上运行，但是容器端口是上述的6443端口。
Let’s test the internal API from within the pod. Since there’s no certificate to our web-server, we used the ‘-k’ switch:
让我们从Pod中测试内部API。 由于我们的网络服务器没有证书，因此我们使用了“ -k”开关：
curl -k https://localhost:6443/apis/custom.metrics.k8s.io/v1beta1/
curl -k https://localhost:6443/apis/custom.metrics.k8s.io/v1beta1/
And received the following output:
并收到以下输出：
{“status”:”healthy”}
{“status”:”healthy”}
This would serve later on as a “Liveness probe” that the web-server is healthy and accepting requests.
稍后，这将作为Web服务器运行状况良好且正在接受请求的“实时调查”。
So up until this point we had a working web-server on port 6443 that hopefully the APIService we were going to create would be able to communicate with. We had yet to figure out the path to retrieve metrics from, but more on that later.
因此，到目前为止，我们在端口6443上有一个正常工作的网络服务器，希望我们将要创建的APIService能够与之通信。 我们还没有找到从中检索指标的途径，但稍后会介绍更多。
创建自定义指标APIService (Creating the Custom Metrics APIService)
When applied, the below resource definition creates and registers the ‘v1beta1.custom.metrics.k8s.io’ APIService. Note that it has no namespace affinity and thus a global resource.
应用后，以下资源定义将创建并注册“ v1beta1.custom.metrics.k8s.io ” APIService。 请注意，它没有名称空间关联，因此没有全局资源。
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
name: v1beta1.custom.metrics.k8s.io
spec:
insecureSkipTLSVerify: true
group: custom.metrics.k8s.io
groupPriorityMinimum: 1000
versionPriority: 5
service:
name: metrics-exporter
namespace: custom-metrics
version: v1beta1The most interesting bit in this APIService resource definition is the ‘service’ stanza. It basically registers a new APIService object that will bind the new API path to the Kubernetes service that’s implementing it, in our case it is the ‘metrics-exporter’ service in the ‘custom-metrics’ namespace.
此APIService资源定义中最有趣的部分是“服务”节。 它基本上注册了一个新的APIService对象，该对象会将新的API路径绑定到实现它的Kubernetes服务，在我们的例子中，它是“ custom-metrics”命名空间中的“ metrics-exporter”服务。
This APIService would connect to our internal API Kubrenetes service in an attempt to retrieve our custom application metrics.
该APIService将连接到我们的内部API Kubrenetes服务，以尝试检索我们的自定义应用程序指标。
The chain of information looks like this:
信息链如下所示：
APIService → Metrics-Exporter Kubernetes-Service → Metrics-Exporter Pod
APIService→指标导出Kubernetes服务→指标导出Pod
If you’re using RBAC in your cluster, you’ll also need to create ServiceAccount, ClusterRole and ClusterRoleBindings for the custom metrics API. I’ve added all yaml definitions for the above in here.Without these you’ll get “Unauthorized” error messages since the API would fail to interact with Kubernetes resources.
如果您在集群中使用RBAC ，则还需要为自定义指标API创建ServiceAccount ， ClusterRole和ClusterRoleBindings 。 我在这里为上述内容添加了所有yaml定义。没有这些定义，您将收到“未经授权”错误消息，因为该API无法与Kubernetes资源进行交互。
Now, let’s examine the newly-created APIService:
现在，让我们检查一下新创建的APIService：
We can see that it’s available, let’s describe it:
我们可以看到它可用，让我们对其进行描述：
It reports back that “all checks passed”, which bodes well for now. It would not have passed if we omitted the “Insecure Skip TLS Verify: true” line (again, there’s no signed certificate for our web-server), or if the deployment/service underneath it were not in place.
它报告说“所有检查均已通过”，目前预示着良好的发展。 如果我们省略了“不安全的跳过TLS验证：是”行(同样，我们的Web服务器没有签名证书)，或者如果它下面的部署/服务没有到位，那么它就不会过去。
We’re almost done!
我们快完成了！
创建HPA资源 (Creating the HPA resource)
Now the last missing part of the puzzle is what I described earlier in this article and what consumed a lot of my time: the path from which the APIService would retrieve the metrics of our deployment. So far, our internal API returns a health check status in JSON format, which is merely enough to satisfy the APIService’s sanity checks.
现在，难题的最后遗漏的部分是我在本文前面所描述的内容，它消耗了我很多时间：APIService将从中检索我们部署指标的路径。 到目前为止，我们的内部API以JSON格式返回运行状况检查状态，这仅足以满足APIService的完整性检查要求。
We created the below HPA resource and started experimenting:
我们创建了以下HPA资源并开始进行实验：
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: binder-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: binder
minReplicas: 1
maxReplicas: 30
metrics:
- type: Object
object:
target:
kind: Service
name: metrics-exporter
metricName: seconds
targetValue: 100Note that the deployment we’re asking to scale is called “binder”, it’s one of our prominent micro-services that gets a lot of requests.
请注意，我们要求扩展的部署称为“绑定程序”，它是我们的重要微服务之一，它收到大量请求。
When describing the HPA resource we created, we get the following error that the HPA cannot get the value of a metric named “seconds” and as a result nothing would be scaled:
描述我们创建的HPA资源时，会出现以下错误，即HPA无法获得名为“ seconds”的度量值，结果将无法缩放：
At this stage we’re still ignorant and baffled as to the format that needs to be adhered to in regards to API calls done by Kubernetes.
在此阶段，对于Kubernetes进行的API调用所需要遵循的格式，我们仍然不知所措。
Since we use a managed Kubernetes cluster (EKS) on AWS, we turned to their premium support for help with troubleshooting (as we don’t have the master logs and can’t see what’s really going on under the hood). After several back-and-forth correspondences and failed attempts to help with the matter, the support engineer sent me off with:
由于我们在AWS上使用托管的Kubernetes集群(EKS)，因此我们寻求他们的高级支持来进行故障排除(因为我们没有主日志，也无法了解实际情况)。 经过数次来回通信并尝试解决此问题失败，支持工程师使我离开：
“Custom metrics API implementation or implementing any Kubernetes configurations falls outside of the scope of support for us Premium Support Engineers and is done on a best-effort basis”
“ 自定义指标API的实现或任何Kubernetes配置的实现都不属于我们高级支持工程师的支持范围，并且是在最大努力的基础上完成的”
So much for best effort :-/
尽最大努力：-/
With little new debug information we received from AWS, we carried on doing our own experimentation and research.
我们从AWS收到的新调试信息很少，因此我们继续进行了自己的实验和研究。
After lots of reading on how APIs interact in Kubernetes and hours of trial and error, we had managed to crack this one! We finally understood what the proper way of interacting with the API was, and where the APIService was expecting to find the “seconds” metric.
在大量阅读了有关API如何在Kubernetes中进行交互以及经过数小时的反复试验之后，我们设法破解了这一难题！ 我们最终了解了与API进行交互的正确方式是什么，以及APIService期望在何处找到“秒”度量。
It appears that the path convention is as follows:
似乎路径约定如下：
/apis/custom.metrics.k8s.io/v1beta1/namespaces/<NAMESPACE>/services/<NAME_OF_CUSTOM_METRICS_SERVICE>/<METRIC_NAME>’In our case, the full path was:
在我们的例子中，完整路径是：
/apis/custom.metrics.k8s.io/v1beta1/namespaces/custom-metrics/services/metrics-exporter/secondsAnd could be verified via:
可以通过以下方式验证：
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/custom-metrics/services/metrics-exporter/secondsHowever, that was not enough! We knew the path, but what was the expected output format when hitting that path?
但是，这还不够！ 我们知道路径，但是到达该路径时预期的输出格式是什么？
We were naive at first to think that a simple {“seconds”:”10”} would do. Needless to say it wasn’t that simple, so again we used the ‘metrics’ API as a reference and deduced the supported JSON output format.When piping it all into ‘jq’ for better readability, we get the following JSON output that originates from our internal API:
起初我们很天真地认为简单的{“seconds”:”10”}就能做到。 不用说它不是那么简单，所以我们再次使用'metrics'API作为参考并推导了受支持的JSON输出格式。当将它们全部管道化为'jq'以提高可读性时，我们得到了以下JSON输出从我们的内部API：
This, ladies and gentlemen, is the proper way of satisfying the APIService metric values. Your internal API code must output its metrics in the above way, or else no metric values would be processed by the HPA.
女士们，先生们，这是满足APIService指标值的正确方法。 您的内部API代码必须以上述方式输出其指标，否则HPA不会处理任何指标值。
Now the HPA is happy and can retrieve the current value of the ‘seconds’ metric:
现在，HPA感到很高兴，并且可以检索“秒”度量标准的当前值：
结论 (Conclusion)
Once again we’re witnessing the sheer brilliance and dynamics of Kubernetes, as it offers its users a way to scale pods based on whatever criteria they desire. This internal ‘metrics-server’ of ours is now being further developed to support any kind of deployment in our cluster in terms of custom metrics auto-scaling.
我们再次见证了Kubernetes的光彩和动态，因为它为用户提供了一种根据他们想要的标准缩放Pod的方法。 现在，我们的内部“指标服务器”正在进一步开发，以支持自定义指标自动扩展方面集群中的任何类型的部署。
Hope you enjoyed reading and please feel free to ask questions.
希望您喜欢阅读，请随时提问。
1263
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
