c++获取可用端口号_k8s springboot 高可用部署、基于qps动态扩缩容、prometheus监控

以下均为亲自实践并应用

概要

1、k8s sprintboot eureka高可用配置、部署、优化

2、k8s sprintboot 微服务高可用配置、部署、优化

3、springboot安装prometheus依赖并获取metric

4、helm安装prometheus operator、prometheus adapter(custom metric)

5、配置prometheus及adapter获取应用qps

6、配置k8s hpa(HorizontalPodAutoscaler)

7、配置grafana展示监控数据

配置

1、k8s sprintboot eureka高可用配置、部署、优化

简介：eureka做集群，需要eureka互相注册，为方便注册，使用statefulSet方式部署eureka server配置：

eureka:  server:    enable-self-preservation: false    eviction-interval-timer-in-ms: 5000  client:    fetch-registry: true      register-with-eureka: true #互相注册    serviceUrl:      defaultZone: ${EUREKA_URL}  instance:    prefer-ip-address: true    lease-renewal-interval-in-seconds: 3    lease-expiration-duration-in-seconds: 10

eureka_url:将多实例地址均填写，方便配置

EUREKA_URL="http://eureka-0.eureka.default.svc.cluster.local:8761/eureka/,http://eureka-1.eureka.default.svc.cluster.local:8761/eureka/"

eureka statefulSet配置：

apiVersion:  apps/v1kind: StatefulSetmetadata:  labels:    app: eureka  name: eureka  namespace: defaultspec:  replicas: 2  serviceName: eureka  podManagementPolicy: "Parallel"  revisionHistoryLimit: 3  selector:    matchLabels:      app: eureka  template:    metadata:      labels:        app: eureka      annotations:   #用于prometheus收集数据，需micrometer-registry-prometheus插件        prometheus.io/path: "/actuator/prometheus"        prometheus.io/port: "8761"        prometheus.io/scrape: "true"    spec:      containers:      - image: eureka:latest              #镜像名        imagePullPolicy: IfNotPresent        name: eureka        envFrom:        - configMapRef:            name: app-configmap           #环境变量        env:        - name: MEMORY          value: "800"        resources:          limits:            cpu: "0"            memory: 1000Mi          requests:            cpu: "0"            memory: 1000Mi        readinessProbe:          tcpSocket:            port: 8761          failureThreshold: 1          initialDelaySeconds: 20        livenessProbe:          tcpSocket:            port: 8761          failureThreshold: 1          initialDelaySeconds: 60        volumeMounts:        - mountPath: /logs          name: log-vol      volumes:      - name: log-vol        nfs:          path: /logs/eureka         #多实例日志可统一在此次查看          server: 127.0.0.1           #nfs地址      dnsPolicy: ClusterFirst      imagePullSecrets:      - name: registry-harbor      restartPolicy: Always

2、k8s sprintboot 微服务高可用配置、部署、优化

这边有思考点，client以什么地址注册eureka？a、以deployment直接部署，默认情况会将容器名称注册到eureka，k8s不识别deployment方式容器名称域名，导致无法访问b、使用statefulSet部署，通过pod名称注册，并且可以识别域名，以pod名称注册并使用，感觉完美，然而需要弹性扩缩容时候，pod缩容，注册信息不会立马从eureka server去除，有延迟，会导致请求到这个销毁的pod上，error！(如果不考虑hpa这方式ok)c、使用service注册，通过endpoint分发请求，pod销毁后pod ip会里面从endpoint里去除，不会出现缩容时候请求到销毁pod上，扩容时候，服务没完全起来请求转发到此pod上会出现问题，不过可以通过健康检查解决最终选择c方案：client配置如下：HOSTNAME即k8s service名称，注册时候会将service注册到eureka

#服务注册相关配置eureka:  client:    serviceUrl:      defaultZone: ${EUREKA_URL}  instance:    prefer-ip-address: true    instance-id: ${HOSTNAME}:${server.port}    ip-address: ${HOSTNAME}

deployment和service配置

apiVersion:  apps/v1kind: Deploymentmetadata:  labels:    app: client   name: client   namespace: defaultspec:  replicas: 2  revisionHistoryLimit: 3  selector:    matchLabels:      app: client   template:    metadata:      labels:        app: client       annotations:        prometheus.io/path: "/actuator/prometheus"        prometheus.io/port: "8024"        prometheus.io/scrape: "true"    spec:      containers:      - image: client:latest        imagePullPolicy: IfNotPresent        name: client         envFrom:        - configMapRef:            name: client-configmap        env:        - name: MEMORY          value: "1300"        - name: HOSTNAME          value: client         resources:          limits:            cpu: "0"            memory: 1500Mi          requests:            cpu: "0"            memory: 1500Mi        readinessProbe:          tcpSocket:            port: 8024          failureThreshold: 1          initialDelaySeconds: 60        livenessProbe:          tcpSocket:            port: 8024          failureThreshold: 1          initialDelaySeconds: 90      dnsPolicy: ClusterFirst      imagePullSecrets:      - name: registry-harbor      restartPolicy: Always---apiVersion: v1kind: Servicemetadata:  name: client   namespace: default  labels:    app: client  spec:  ports:  - name: http-8024    port: 8024    protocol: TCP    targetPort: 8024  selector:    app: client  

3、springboot安装prometheus依赖并获取metric

高可用部署完成后就需要弹性扩缩容， 需要获取qps数据，这里用的是prometheus，直接贴配置

增加pom依赖：

        io.micrometer            micrometer-registry-prometheus        

增加配置文件：

management:  endpoints:    web:      exposure:        include: '*'  metrics:    tags:      application: ${spring.application.name}

主函数增加bean

import io.micrometer.core.instrument.MeterRegistry;import org.springframework.beans.factory.annotation.Value;import org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryCustomizer;import org.springframework.context.annotation.Bean;public class Application {    public static void main(String[] args) {        SpringApplication.run(Application.class, args);    }    @Bean    MeterRegistryCustomizer configurer(            @Value("${spring.application.name}") String applicationName) {        return (registry) -> registry.config().commonTags("application", applicationName);    }}

配置完成后访问：http:localhost:8024/actuator/prometheus,返回数据如下(重点关注下http_server_requests_seconds_count,后面会用到)

# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management# TYPE jvm_memory_max_bytes gaugejvm_memory_max_bytes{application="portal-eureka",area="heap",id="Par Survivor Space",} 2.7918336E7jvm_memory_max_bytes{application="portal-eureka",area="nonheap",id="Compressed Class Space",} 1.073741824E9jvm_memory_max_bytes{application="portal-eureka",area="heap",id="Tenured Gen",} 5.59284224E8jvm_memory_max_bytes{application="portal-eureka",area="nonheap",id="Metaspace",} -1.0jvm_memory_max_bytes{application="portal-eureka",area="heap",id="Par Eden Space",} 2.23739904E8jvm_memory_max_bytes{application="portal-eureka",area="nonheap",id="Code Cache",} 2.5165824E8# HELP logback_events_total Number of error level events that made it to the logs# TYPE logback_events_total counterlogback_events_total{application="portal-eureka",level="info",} 804.0http_server_requests_seconds_count{application="portal-eureka",exception="None",method="POST",outcome="SUCCESS",status="204",uri="root",} 8.0http_server_requests_seconds_sum{application="portal-eureka",exception="None",method="POST",outcome="SUCCESS",status="204",uri="root",} 0.29997471http_server_requests_seconds_count{application="portal-eureka",exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 114.0...

4、helm安装prometheus operator、prometheus adapter(custom metric)

a、下载prometheus-operator

helm fetch stable/prometheus-operator --version 8.13.0

b、解压

tar -zxvf prometheus-operator-8.13.3.tgz

c、修改values.yaml文件,增加采集pod job(operator默认没有搜集kubernetes-pod的配置)

vim prometheus-operator/values.yaml

增加

    additionalScrapeConfigs:       - job_name: 'kubernetes-pods'        honor_labels: false        kubernetes_sd_configs:        - role: pod        tls_config:          insecure_skip_verify: true        relabel_configs:        - source_labels: [__meta_kubernetes_namespace]          action: replace          target_label: namespace        - source_labels: [__meta_kubernetes_pod_name]          action: replace          target_label: pod        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]          action: keep          regex: true        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]          action: replace          target_label: __metrics_path__          regex: (.+)        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]          action: replace          regex: ([^:]+)(?::d+)?;(d+)          replacement: $1:$2          target_label: __address__        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]          action: replace          target_label: __scheme__          regex: (.+)

d、安装

helm install -n prometheus-operator prometheus-operator --namespace monitor

e、暴露端口

kubectl edit svc prometheus-operator-prometheus -nmonitor #增加externalIPs或者nodeport

f、安装stable/prometheus-adapter

helm fetch stable/prometheus-adapter --version 2.3.1

解压

tar -zxvf prometheus-adapter-2.3.1.tgz

修改prometheus server配置

vim prometheus-adapter/values.yaml #将prometheus.url替换为http://prometheus-operator-prometheus.monitor.svc

安装

helm install -n prometheus-adapter prometheus-adapter   --namespace monitor

查看api

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1

5、配置prometheus及adapter获取应用qps

a、增加adapter规则

kubectl edit cm -nmonitor prometheus-adapter

增加规则，将http_server_requests_seconds_count以每5分钟为时间段计算为qps，metric名称为http_server_requests_seconds

    - seriesQuery: '{namespace!="",__name__!~"^container_.*"}'      seriesFilters:      - isNot: .*_seconds_total      resources:        template: <<.resource>>      name:        matches: ^(.*)_count$        as: ""      metricsQuery: sum(rate(<<.series>>{<<.labelmatchers>>}[5m])) by (<<.groupby>>)

b、重启adapter

kubectl -nmonitor delete pod prometheus-adapter-...

c、查看自定义metric qps

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_server_requests_seconds"

6、配置k8s hpa(HorizontalPodAutoscaler)

直接贴配置：

apiVersion: autoscaling/v2beta2kind: HorizontalPodAutoscalermetadata:  name: client  namespace: defaultspec:  scaleTargetRef:    apiVersion: apps/v1    kind: Deployment    name: client  minReplicas: 2  maxReplicas: 10  metrics:  - type: Pods    pods:      metric:        name: http_server_requests_seconds      target:        type: AverageValue        averageValue: 50

kubectl get hpa #查看hpa是否获取到metric

7、配置grafana展示监控数据

展示应用qps sql(不包含/actuator/prometheus地址)

sum(rate(http_server_requests_seconds_count{job="kubernetes-pods",uri!="/actuator/prometheus"  }[5m])) by (application)

image.png

dashboard模板地址：https://grafana.com/grafana/dashboards?dataSource=prometheus&direction=asc&orderBy=name&search=spring

推荐Spring Boot 2.1 Statistics模板，导入方法如下图，复制id或者json即可

image.png