原理分析
prometheus+prometheus-adapter的工作原理
prometheus通过聚合层扩展kubenetes API原理
聚合层在 kube-apiserver 进程内运行。在扩展资源注册之前,聚合层不做任何事情。 要注册 API,用户必须添加一个 APIService 对象,用它来“申领” Kubernetes API 中的 URL 路径。 自此以后,聚合层将会把发给该 API 路径的所有内容 转发到已注册的 APIService。
总结原理如下:
- aggregatorServer 通过 APIServices 对象关联到某个 Service 来进行请求的转发,其关联的 Service 类型进一步决定了请求转发的形式。aggregatorServer 包括一个 GenericAPIServer 和维护自身状态的 Controller。其中 GenericAPIServer 主要处理 apiregistration.k8s.io 组下的 APIService 资源请求,而Controller包括:
-
- apiserviceRegistrationController:负责根据 APIService 定义的 aggregated server service 构建代理,将CR的请求转发给后端的 aggregated server
- availableConditionController:维护 APIServices 的可用状态,包括其引用 Service 是否可用等
-
- autoRegistrationController:用于保持 API 中存在的一组特定的 APIServices
- crdRegistrationController:负责将 CRD GroupVersions 自动注册到 APIServices 中
-
- openAPIAggregationController:将 APIServices 资源的变化同步至提供的 OpenAPI 文档
- apiserviceRegistrationController 负责根据 APIService 定义的 aggregated server service 构建代理,将CR的请求转发给后端的 aggregated server。apiService 有两种类型:Local(Service为空)以及Service(Service非空)。apiserviceRegistrationController 负责对这两种类型 apiService 设置代理:Local 类型会直接路由给 kube-apiserver 进行处理;而Service类型则会设置代理并将请求转化为对 aggregated Service 的请求(proxyPath := "/apis/" + apiService.Spec.Group + "/" + apiService.Spec.Version),而请求的负载均衡策略则是优先本地访问 kube-apiserver(如果service为kubernetes default apiserver service:443) =>通过 service ClusterIP:Port 访问(默认)或者通过随机选择 service endpoint backend 进行访问:
画图说明aggregator
prometheus-adapter源码分析
代码地址:https://github.com/kubernetes-sigs/prometheus-adapter
main函数入口
文件位置:cmd/adapter/adapter.go
func main() {
logs.InitLogs()
defer logs.FlushLogs()
// set up flags
cmd := &PrometheusAdapter{
PrometheusURL: "https://localhost",
MetricsRelistInterval: 10 * time.Minute,
}
cmd.Name = "prometheus-metrics-adapter"
cmd.OpenAPIConfig = genericapiserver.DefaultOpenAPIConfig(generatedopenapi.GetOpenAPIDefinitions, openapinamer.NewDefinitionNamer(api.Scheme, customexternalmetrics.Scheme))
cmd.OpenAPIConfig.Info.Title = "prometheus-metrics-adapter"
cmd.OpenAPIConfig.Info.Version = "1.0.0"
// 加载启动参数
cmd.addFlags()
cmd.Flags().AddGoFlagSet(flag.CommandLine) // make sure we get the klog flags
if err := cmd.Flags().Parse(os.Args); err != nil {
klog.Fatalf("unable to parse flags: %v", err)
}
// if --metrics-max-age is not set, make it equal to --metrics-relist-interval
if cmd.MetricsMaxAge == 0*time.Second {
cmd.MetricsMaxAge = cmd.MetricsRelistInterval
}
// 生成prometheus client,
promClient, err := cmd.makePromClient()
if err != nil {
klog.Fatalf("unable to construct Prometheus client: %v", err)
}
// 加载prometheus-adapter metric配置文件
if err := cmd.loadConfig(); err != nil {
klog.Fatalf("unable to load metrics discovery config: %v", err)
}
// stop channel closed on SIGTERM and SIGINT
stopCh := genericapiserver.SetupSignalHandler()
// 指标生产者,获取metric指标并缓存,runner会定时更新metric指标库
cmProvider, err := cmd.makeProvider(promClient, stopCh)
if err != nil {
klog.Fatalf("unable to construct custom metrics provider: %v", err)
}
// attach the provider to the server, if it's needed
if cmProvider != nil {
cmd.WithCustomMetrics(cmProvider)
}
// construct the external provider
emProvider, err := cmd.makeExternalProvider(promClient, stopCh)
if err != nil {
klog.Fatalf("unable to construct external metrics provider: %v", err)
}
// attach the provider to the server, if it's needed
if emProvider != nil {
cmd.WithExternalMetrics(emProvider)
}
// attach resource metrics support, if it's needed
if err := cmd.addResourceMetricsAPI(promClient, stopCh); err != nil {
klog.Fatalf("unable to install resource metrics API: %v", err)
}
// run the server,注册自定义api到kube-apiserver中
if err := cmd.Run(stopCh); err != nil {
klog.Fatalf("unable to run custom metrics adapter: %v", err)
}
}
- 运行参数
- prometheus规则配置
- 生成prometheus-client
- 根据规则生成指标
- 注册api接口到kube-apiserver
核心代码
prometheus-adapter指标规则
对prometheus的指标进行处理。查询规则
rules:
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
seriesFilters: []
resources:
overrides:
namespace:
resource: namespace
pod_name:
resource: pod
name:
matches: ^container_(.*)_seconds_total$
as: ""
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[1m])) by (<<.GroupBy>>)
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
seriesFilters:
- isNot: ^container_.*_seconds_total$
resources:
overrides:
namespace:
resource: namespace
pod_name:
resource: pod
name:
matches: ^container_(.*)_total$
as: ""
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[1m])) by (<<.GroupBy>>)
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
seriesFilters:
- isNot: ^container_.*_total$
resources:
overrides:
namespace:
resource: namespace
pod_name:
resource: pod
name:
matches: ^container_(.*)$
as: ""
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}) by (<<.GroupBy>>)
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
seriesFilters:
- isNot: .*_total$
resources:
template: <<.Resource>>
name:
matches: ""
as: ""
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
seriesFilters:
- isNot: .*_seconds_total
resources:
template: <<.Resource>>
name:
matches: ^(.*)_total$
as: ""
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
seriesFilters: []
resources:
template: <<.Resource>>
name:
matches: ^(.*)_seconds_total$
as: ""
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
resourceRules:
cpu:
containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
nodeQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>, id='/'}[1m])) by (<<.GroupBy>>)
resources:
overrides:
instance:
resource: node
namespace:
resource: namespace
pod_name:
resource: pod
containerLabel: container_name
memory:
containerQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>}) by (<<.GroupBy>>)
nodeQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>,id='/'}) by (<<.GroupBy>>)
resources:
overrides:
instance:
resource: node
namespace:
resource: namespace
pod_name:
resource: pod
containerLabel: container_name
window: 1m
生成指标
文件位置:prometheus-adapter/cmd/adapter/adapter.go
func (cmd *PrometheusAdapter) makeProvider(promClient prom.Client, stopCh <-chan struct{}) (provider.CustomMetricsProvider, error) {
if len(cmd.metricsConfig.Rules) == 0 {
return nil, nil
}
if cmd.MetricsMaxAge < cmd.MetricsRelistInterval {
return nil, fmt.Errorf("max age must not be less than relist interval")
}
// 获取资源对象
// grab the mapper and dynamic client
mapper, err := cmd.RESTMapper()
if err != nil {
return nil, fmt.Errorf("unable to construct RESTMapper: %v", err)
}
dynClient, err := cmd.DynamicClient()
if err != nil {
return nil, fmt.Errorf("unable to construct Kubernetes client: %v", err)
}
// 解析adapter规则配置
// extract the namers
namers, err := naming.NamersFromConfig(cmd.metricsConfig.Rules, mapper)
if err != nil {
return nil, fmt.Errorf("unable to construct naming scheme from metrics rules: %v", err)
}
// construct the provider and start it
cmProvider, runner := cmprov.NewPrometheusProvider(mapper, dynClient, promClient, namers, cmd.MetricsRelistInterval, cmd.MetricsMaxAge)
runner.RunUntil(stopCh)
return cmProvider, nil
}
func (l *cachingMetricsLister) RunUntil(stopChan <-chan struct{}) {
go wait.Until(func() {
if err := l.updateMetrics(); err != nil {
utilruntime.HandleError(err)
}
}, l.updateInterval, stopChan)
}
1. 生成adapter规则配置 2. 定时更新prometheus指标缓存
指标查询
指标查询接口
文件位置:/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.22.0/pkg/provider/interfaces.go
type CustomMetricsProvider interface {
// GetMetricByName fetches a particular metric for a particular object.
// The namespace will be empty if the metric is root-scoped.
GetMetricByName(ctx context.Context, name types.NamespacedName, info CustomMetricInfo, metricSelector labels.Selector) (*custom_metrics.MetricValue, error)
// GetMetricBySelector fetches a particular metric for a set of objects matching
// the given label selector. The namespace will be empty if the metric is root-scoped.
GetMetricBySelector(ctx context.Context, namespace string, selector labels.Selector, info CustomMetricInfo, metricSelector labels.Selector) (*custom_metrics.MetricValueList, error)
// ListAllMetrics provides a list of all available metrics at
// the current time. Note that this is not allowed to return
// an error, so it is reccomended that implementors cache and
// periodically update this list, instead of querying every time.
ListAllMetrics() []CustomMetricInfo
}
指标查询时,通过hpa查询对应指标,具体实现在prometheus-adapter
文件位置:prometheus-adapter/pkg/custom-provider/provider.go
func (p *prometheusProvider) GetMetricBySelector(ctx context.Context, namespace string, selector labels.Selector, info provider.CustomMetricInfo, metricSelector labels.Selector) (*custom_metrics.MetricValueList, error) {
// 获取先关资源
resourceNames, err := helpers.ListObjectNames(p.mapper, p.kubeClient, namespace, selector, info)
if err != nil {
klog.Errorf("unable to list matching resource names: %v", err)
// don't leak implementation details to the user
return nil, apierr.NewInternalError(fmt.Errorf("unable to list matching resources"))
}
// 向prometheus查询metric指标
queryResults, err := p.buildQuery(ctx, info, namespace, metricSelector, resourceNames...)
if err != nil {
return nil, err
}
// return the resulting metrics
return p.metricsFor(queryResults, namespace, resourceNames, info, metricSelector)
}