prometheus函数_具有prometheus适配器的自动缩放无核函数

最新推荐文章于 2024-08-10 07:39:39 发布

吴雄辉

最新推荐文章于 2024-08-10 07:39:39 发布

阅读量451

点赞数

文章标签： python

原文链接：https://itnext.io/autoscale-kubeless-functions-with-prometheus-adapter-f99507a23db6

版权

prometheus函数

As a part of the web ecosystem, our main concern is how to create an application that handles more, many more requests and how to keep it afloat. Am I wrong?

作为网络生态系统的一部分，我们的主要关注点是如何创建一个处理更多，更多请求的应用程序，以及如何使其永续运行。我错了吗？

Whatever you have, such as monolith, microservices, or functions you need to create ways to deal with the concern. You can add a new machine into your infrastructure (please don’t say, “ok boomer”, I’m not!) or watch the actions from your monitoring systems and click the red button: auto-deploy the new instance and split the traffic! Or, you can create your rules before the chaos, automate your scaling, lay back, and enjoy it!

无论您拥有什么(如整体式，微服务或功能)，都需要创建解决问题的方法。您可以在基础架构中添加新计算机(请不要说“ ok boomer” ，我不是！)，或从监视系统观看操作，然后单击红色按钮：自动部署新实例并拆分新实例。交通！或者，您可以在混乱之前创建规则，自动缩放，放松并享受它！

那Kubernetes宇宙呢？ (What about Kubernetes universe?)

If you live in Kubernetes, it gives a hand for the deals with HorizontalPodAutoscaler API — a.k.a. HPA — . You can easily say “hey! If the memory/CPU consumption of my application is more than X, please add a new Pod!” If you use metric-server cluster add-on, you could get these values with metrics.k8s.io API.

如果您住在Kubernetes，它可以帮助您使用HorizontalPodAutoscaler API(又名HPA)进行交易。您可以轻松地说“嘿！ 如果我的应用程序的内存/ CPU消耗超过X，请添加一个新的Pod！” 如果您使用metric-server集群附加组件，则可以通过metrics.k8s.io API获得这些值。

如果我的需求与CPU或内存消耗不同，该怎么办？ (What if my needs are different than CPU or memory consumption?)

For instance, I want to scale my application according to incoming requests per minute. It’s the most natural wish for the web, isn’t it? Or, it’s the main goal of the story, I would like to scale my function by the total call count.

例如，我想根据每分钟传入的请求扩展应用程序。这是网络上最自然的愿望，不是吗？或者，这是故事的主要目标，我想通过调用总数来扩展功能。

Let’s say hello to custom metric. You can create your own custom adapters to collect your own metrics or you could use known solutions. At the end of the road, you reach the values with custom.metrics.k8s.io API.

让我们向自定义指标问好。您可以创建自己的自定义适配器来收集自己的指标，也可以使用已知的解决方案。在这条路的尽头，您可以使用custom.metrics.k8s.io API来获取值。

Throughout the story, we’ll use Prometheus Adapter as a known solution to collect the Kubeless function metrics and auto-scale the functions.

在整个故事中，我们将使用Prometheus Adapter作为已知的解决方案来收集无Kube函数度量并自动缩放函数。

我们开始做吧！ (Let’s do it!)

Before the roll up our sleeves, I need to list the environment:

在卷起袖子之前，我需要列出环境：

Kubernetes version is 1.16.13

Kubernetes版本是1.16.13

Prometheus version is 2.20.1

Prometheus版本为2.20.1

Kubeless version is 1.0.7

Kubeless版本是1.0.7

Helm version is 2.16.1

头盔版本为2.16.1

Prometheus adapter version is 0.7.0

Prometheus适配器版本为0.7.0

I assume that you have a ready-in-use Kubernetes cluster and Prometheus server. If don’t, there are a lot of useful articles on the web about to setup them.

我假设您有一个现成的Kubernetes集群和Prometheus服务器。 如果不这样做，那么网络上将会有很多有用的文章来设置它们。

在集群上安装Kubeless，创建第一个功能 (Install the Kubeless on the cluster, create your first function)

Kubeless is the “Function-as-a-Service” — a.k.a FaaS — framework. You can build FaaS applications top of Kubernetes with it. It’s a strong alternative to KNative and OpenFaaS solutions.

Kubeless是“功能即服务”(也称为FaaS )框架。您可以使用它在Kubernetes之上构建FaaS应用程序。它是KNative和OpenFaaS解决方案的强大替代方案。

It’s the shortest part of the story. There’s no custom step to do it. You can follow the instructions, it’s on here. Also, you could use a basic UI to manage your functions with Kubeless-UI. kubectl create -f https://raw.githubusercontent.com/kubeless/kubeless-ui/master/k8s.yaml command can be used to install it.

这是故事中最短的部分。没有自定义步骤可以执行此操作。您可以按照此处的说明进行操作。另外，您可以使用基本UI来通过Kubeless-UI管理功能。可以使用kubectl create -f https://raw.githubusercontent.com/kubeless/kubeless-ui/master/k8s.yaml命令进行安装。

Image for post — Screenshot from Kubeless-UI

“How to create complex functions on the Kubeless?” is not a part of the story. So we don’t focus on it. You can follow the walkthrough to create your first function.

“如何在Kubeless上创建复杂的功能？” 不是故事的一部分。所以我们不关注它。您可以按照演练来创建您的第一个功能。

安装Prometheus适配器 (Install the Prometheus Adapter)

Prometheus adapter is using to collect custom metrics, with the custom rules, from your Kubernetes cluster. You can use the helm chart to install it with the default values. But;

Prometheus适配器用于从Kubernetes群集收集具有自定义规则的自定义指标。您可以使用头盔图以默认值安装它。但;

If your Prometheus server is not running on default namespace or is not using default port (9090).
如果您的Prometheus服务器未在默认名称空间上运行或未使用默认端口(9090)。
If you want to use it for only custom metrics (it can be used as an alternative to metric-server).
如果要仅将其用于自定义指标(可以用作指标服务器的替代方法)。
If you want to re-list the metrics in the different intervals (default value is 1 minute).
如果要以不同的间隔重新列出度量标准(默认值为1分钟)。
In order to define your custom rules (Prometheus queries to collect the metrics).
为了定义您的自定义规则(Prometheus查询以收集度量标准)。

You need to change some variables with your ones.

您需要更改一些变量。

There’s something wrong on the chart repository. When you use--set-file argument to pass your custom rule file to the helm install command, it puts the pipe symbol (|) and corrupts the generated template. So we’ll clone it ( git clone https://github.com/helm/charts.git ) and modify this file. You have to replace toYaml functions with tpl . For example; {{toYaml .Values.rules.custom | indent 4 }} must be {{tpl .Values.rules.custom . | indent 4 }} .

图表存储库中出了点问题。当使用--set-file参数将自定义规则文件传递给helm install命令时，它将放置管道符号(|)并破坏生成的模板。因此，我们将其克隆( git clone https://github.com/helm/charts.git )并修改此文件。您必须用tpl替换toYaml函数。例如; {{toYaml .Values.rules.custom | indent 4 }} {{toYaml .Values.rules.custom | indent 4 }}必须为{{tpl .Values.rules.custom . | indent 4 }} {{tpl .Values.rules.custom . | indent 4 }} {{tpl .Values.rules.custom . | indent 4 }} 。

Kubeless exposes the total call of the functions as a metric. It named as function_calls_total and it works as a counter. It starts to count when your first call comes. It’s presenting furthermore metrics of course but we’ll use it for the auto-scaling functionality.

Kubeless将函数的总调用公开为度量。它命名为function_calls_total ，并用作计数器。当您的第一个电话到来时，它开始计数。当然，它还提供了其他指标，但我们将其用于自动缩放功能。

The aim is to get the total call count at the last minute. The equivalent on the Prometheus side is: sum(increase(function_calls_total{kubernetes_namespace="kubeless-fn",function!="",kubernetes_pod_name!=""}[1m])) by (kubernetes_namespace,kubernetes_pod_name,function)

目的是获得最后一分钟的总通话次数。普罗米修斯方面的等效项是： sum(increase(function_calls_total{kubernetes_namespace="kubeless-fn",function!="",kubernetes_pod_name!=""}[1m])) by (kubernetes_namespace,kubernetes_pod_name,function)

In order to use the query in the adapter, let’s define our rule:

为了在适配器中使用查询，让我们定义规则：

- seriesQuery: '{__name__=~"function_.*",kubernetes_namespace="kubeless-fn",function!="",kubernetes_pod_name!=""}'
  resources:
    overrides:
      kubernetes_namespace: { resource: "namespace" }
      kubernetes_pod_name: { resource: "pod" }
  name:
    matches: "^function_(.*)"
    as: "${1}_per_minute"
  metricsQuery: sum(increase(<<.Series>>{<<.LabelMatchers>>}[1m]))
    by (<<.GroupBy>>)

and save it as custom-rule.yaml.

并将其另存为custom-rule.yaml 。

Now, your folder structure should be like that;

现在，您的文件夹结构应该像这样；

> charts
|_ .circleci
|_ .github
…
> custom-rule.yaml

>图表| _ .circleci | _ .github … > custom-rule.yaml

Run the below script in the folder and generate the yaml file. It will be used for the setup of Prometheus Adapter.

在文件夹中运行以下脚本并生成yaml文件。它将用于Prometheus适配器的设置。

helm template --name kubeless-adapter charts/stable/prometheus-adapter \
 --set prometheus.url=http://prometheus-service.monitoring.svc \
 --set prometheus.port=9090 \
 --set metricsRelistInterval=30s \
 --set rules.default=false \
 --set-file rules.custom=custom-rules.yaml > 1-install-prometheus-adapter.yaml

Our Prometheus Server is running on monitoring namespace as you see.

如您所见，我们的Prometheus Server在monitoring名称空间上运行。

Finally, run it: kubectl apply -f 1-install-prometheus-adapter.yaml

最后，运行它： kubectl apply -f 1-install-prometheus-adapter.yaml

After a couple of minutes, we check whether the adapter can reach the custom metric API or not: kubectl logs kubeless-adapter-prometheus-adapter-[RS_RANDOM_STRING]-[POD_ID] -c prometheus-adapter . If everything is OK, you would not see any errors.

几分钟后，我们检查适配器是否可以达到自定义指标API： kubectl logs kubeless-adapter-prometheus-adapter-[RS_RANDOM_STRING]-[POD_ID] -c prometheus-adapter 。如果一切正常，您将不会看到任何错误。

创建虚拟指标 (Create dummy metrics)

Before we’re collecting the metrics, we’ll call the function multiple times. You can expose your function as a service with Ingress object, but for the test scenario, we’ll use port-forward option: kubectl port-forward -n kubeless-fn get-python-[RS_RANDOM_STRING]-[POD_ID] 8080:8080

在收集指标之前，我们将多次调用该函数。您可以使用Ingress对象将函数作为服务公开，但是对于测试方案，我们将使用port-forward选项： kubectl port-forward -n kubeless-fn get-python-[RS_RANDOM_STRING]-[POD_ID] 8080:8080

After that you can use this script to call your function: while true;do sleep 0.3;curl "http://localhost:8080";done . Your function will be called approximately 90~100 times in a minute.

之后，您可以使用此脚本来调用您的函数： while true;do sleep 0.3;curl "http://localhost:8080";done 。一分钟内您的函数将被调用约90〜100次。

After a minute check the API result: kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/kubeless-fn/pods/*/calls_total_per_minute" > api-result.json . The output should be like that;

一分钟后，检查API结果： kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/kubeless-fn/pods/*/calls_total_per_minute" > api-result.json / kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/kubeless-fn/pods/*/calls_total_per_minute" > api-result.json 。输出应该是这样；

{
    "kind": "MetricValueList",
    "apiVersion": "custom.metrics.k8s.io/v1beta1",
    "metadata": {
        "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/kubeless-fn/pods/%2A/calls_total_per_minute"
    },
    "items": [
        {
            "describedObject": {
                "kind": "Pod",
                "namespace": "kubeless-fn",
                "name": "get-python-5588696cf9-spxt5",
                "apiVersion": "/v1"
            },
            "metricName": "calls_total_per_minute",
            "timestamp": "2020-09-20T21:19:52Z",
            "value": "91636m",
            "selector": null
        }
    ]
}

value field says that; get-python function was called 91.6 times per minute!

value栏指出： 每分钟调用get-python函数91.6次！

If you had more than one function, metrics for each would be shown there.

如果您有多个功能，则会在此处显示每个功能的指标。

定义HPA对象 (Define the HPA Object)

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: get-python-hpa
  namespace: kubeless-fn
  labels:
    function: get-python
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: get-python
  minReplicas: 1
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metricName: calls_total_per_minute
        targetAverageValue: "20"

I would like to say with this declaration; if the total calls per minute for get-python deployment is more than 20, scale up the application until the replica count will be 20.

我要对这个声明说：如果get-python部署的每分钟总调用次数超过20，请按比例扩大应用程序，直到副本数为20。

According to the above output (91.6) replica count should be 5 ( Math.round(91.6 / 20) ). Check the result with this command: kubectl describe hpa -n kubeless-fn get-python-hpa

根据以上输出(91.6)，副本数应为5( Math.round(91.6 / 20) )。使用以下命令检查结果： kubectl describe hpa -n kubeless-fn get-python-hpa

Now, stop your while… command and wait a couple of minutes. When you re-run describe command you will see “calls_total_per_minute” on pods: 0 / 20 output. Also, you will see the desired replica count is not changed!

现在，停止您的while…命令，然后等待几分钟。当您重新运行describe命令时，将在pod上看到“ calls_total_per_minute”：0/20输出。此外，您将看到所需的副本数未更改！

It’s about Stabilization Window. Kubernetes documentation said that;

关于稳定窗口。 Kubernetes文档说：

The stabilization window is used to restrict the flapping of replicas when the metrics used for scaling keep fluctuating. The stabilization window is used by the autoscaling algorithm to consider the computed desired state from the past to prevent scaling.

当用于缩放的指标持续波动时，稳定窗口用于限制副本的摆动。自动缩放算法使用稳定窗口来考虑过去计算的期望状态以防止缩放。

The default value is 300 seconds/5 minutes. If you run the describe command after 5 minutes, you’ll see the desired count is down:

默认值为300秒/ 5分钟。如果5分钟后运行describe命令，则会看到所需的计数已减少：