etcd监控-prometheus+grafana

etcd metrics

etcd通过2379端口，metrics。可以访问http://etcd-ip:2379/metrics

prometheus-etcd metric采集

配置文件内如如下

promethrus通过etcd的2379端口，采集metric

cat > promethrus-etcd.yaml <<EOF
global:
  scrape_interval: 10s
scrape_configs:
  - job_name: etcd
    metrics_path: '/metrics'
    static_configs:
    - targets: ['10.240.0.32:2379','10.240.0.33:2379','10.240.0.34:2379']
EOF

k8s使用ConfigMap注入配置文件示例

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-conf-etcd
  namespace: prometheus
  labels:
    app: prometheus-etcd
data:
  prometheus.yml: |-
    global:
      scrape_interval:     10s
      evaluation_interval: 10s
    alerting:
      alertmanagers:
      - static_configs:
        - targets: ["alertmanager-service.prometheus:9093"]
    rule_files:
    - "/etc/prometheus/rules/custom.rule"
    scrape_configs:
      - job_name: etcd
        metrics_path: '/metrics'
        static_configs:
        - targets: ['172.24.31.25:2379','172.24.31.24:2379','172.24.31.22:2379']

配置完成重启prometheus后，即可看到新的job和指标

prometheus配置alert告警规则

 prometheus 2.x 规则如下

# These rules were manually synced from https://github.com/etcd-io/etcd/blob/master/contrib/mixin/mixin.libsonnet
groups:
- name: etcd
rules:
- alert: etcdInsufficientMembers
  annotations:
    message: 'etcd cluster "{{ $labels.job }}": insufficient members ({{ $value
      }}).'
  expr: |
    sum(up{job=~".*etcd.*"} == bool 1) by (job) < ((count(up{job=~".*etcd.*"}) by (job) + 1) / 2)
  for: 3m
  labels:
    severity: critical
- alert: etcdNoLeader
  annotations:
    message: 'etcd cluster "{{ $labels.job }}": member {{ $labels.instance }} has
      no leader.'
  expr: |
    etcd_server_has_leader{job=~".*etcd.*"} == 0
  for: 1m
  labels:
    severity: critical
- alert: etcdHighNumberOfLeaderChanges
  annotations:
    message: 'etcd cluster "{{ $labels.job }}": instance {{ $labels.instance }}
      has seen {{ $value }} leader changes within the last hour.'
  expr: |
    rate(etcd_server_leader_changes_seen_total{job=~".*etcd.*"}[15m]) > 3
  for: 15m
  labels:
    severity: warning
- alert: etcdHighNumberOfFailedGRPCRequests
  annotations:
    message: 'etcd cluster "{{ $labels.job }}": {{ $value }}% of requests for {{
      $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.'
  expr: |
    100 * sum(rate(grpc_server_handled_total{job=~".*etcd.*", grpc_code!="OK"}[5m])) BY (job, instance, grpc_service, grpc_method)
      /
    sum(rate(grpc_server_handled_total{job=~".*etcd.*"}[5m])) BY (job, instance, grpc_service, grpc_method)
      > 1
  for: 10m
  labels:
    severity: warning
- alert: etcdHighNumberOfFailedGRPCRequests
  annotations:
    message: 'etcd cluster "{{ $labels.job }}": {{ $value }}% of requests for {{
      $labels.grpc_method }} failed on etcd instance {{ $labels.instance }}.'
  expr: |
    100 * sum(rate(grpc_server_handled_total{job=~".*etcd.*", grpc_code!="OK"}[5m])) BY (job, instance, grpc_service, grpc_method)
      /
    sum(rate(grpc_server_handled_total{job=~".*etcd.*"}[5m])) BY (job, instance, grpc_service, grpc_method)
      > 5
  for: 5m
  labels:
    severity: critical
- alert: etcdGRPCRequestsSlow
  annotations:
    message: 'etcd cluster "{{ $labels.job }}": gRPC requests to {{ $labels.grpc_method
      }} are taking {{ $value }}s on etcd instance {{ $labels.instance }}.'
  expr: |
    histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{job=~".*etcd.*", grpc_type="unary"}[5m])) by (job, instance, grpc_service, grpc_method, le))
    > 0.15
  for: 10m
  labels:
    severity: critical
- alert: etcdMemberCommunicationSlow
  annotations:
    message: 'etcd cluster "{{ $labels.job }}": member communication with {{ $labels.To
      }} is taking {{ $value }}s on etcd instance {{ $labels.instance }}.'
  expr: |
    histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~".*etcd.*"}[5m]))
    > 0.15
  for: 10m
  labels:
    severity: warning
- alert: etcdHighNumberOfFailedProposals
  annotations:
    message: 'etcd cluster "{{ $labels.job }}": {{ $value }} proposal failures within
      the last hour on etcd instance {{ $labels.instance }}.'
  expr: |
    rate(etcd_server_proposals_failed_total{job=~".*etcd.*"}[15m]) > 5
  for: 15m
  labels:
    severity: warning
- alert: etcdHighFsyncDurations
  annotations:
    message: 'etcd cluster "{{ $labels.job }}": 99th percentile fync durations are
      {{ $value }}s on etcd instance {{ $labels.instance }}.'
  expr: |
    histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job=~".*etcd.*"}[5m]))
    > 0.5
  for: 10m
  labels:
    severity: warning
- alert: etcdHighCommitDurations
  annotations:
    message: 'etcd cluster "{{ $labels.job }}": 99th percentile commit durations
      {{ $value }}s on etcd instance {{ $labels.instance }}.'
  expr: |
    histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~".*etcd.*"}[5m]))
    > 0.25
  for: 10m
  labels:
    severity: warning
- alert: etcdHighNumberOfFailedHTTPRequests
  annotations:
    message: '{{ $value }}% of requests for {{ $labels.method }} failed on etcd
      instance {{ $labels.instance }}'
  expr: |
    sum(rate(etcd_http_failed_total{job=~".*etcd.*", code!="404"}[5m])) BY (method) / sum(rate(etcd_http_received_total{job=~".*etcd.*"}[5m]))
    BY (method) > 0.01
  for: 10m
  labels:
    severity: warning
- alert: etcdHighNumberOfFailedHTTPRequests
  annotations:
    message: '{{ $value }}% of requests for {{ $labels.method }} failed on etcd
      instance {{ $labels.instance }}.'
  expr: |
    sum(rate(etcd_http_failed_total{job=~".*etcd.*", code!="404"}[5m])) BY (method) / sum(rate(etcd_http_received_total{job=~".*etcd.*"}[5m]))
    BY (method) > 0.05
  for: 10m
  labels:
    severity: critical
- alert: etcdHTTPRequestsSlow
  annotations:
    message: etcd instance {{ $labels.instance }} HTTP requests to {{ $labels.method
      }} are slow.
  expr: |
    histogram_quantile(0.99, rate(etcd_http_successful_duration_seconds_bucket[5m]))
    > 0.15
  for: 10m
  labels:
    severity: warning

k8s ConfigMap注入配置示例

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-rules-etcd
  namespace: prometheus
  labels:
    instance: "etcd"
    severity: "critical"
    app: prometheus-etcd
data:
  custom.rule: |
    groups:
    - name: etcd
      rules:
      - alert: etcdInsufficientMembers
        annotations:
          message: 'etcd cluster "{{ $labels.job }}": insufficient members ({{ $value
            }}).'
        expr: |
          sum(up{job=~".*etcd.*"} == bool 1) by (job) < ((count(up{job=~".*etcd.*"}) by (job) + 1) / 2)
        for: 3m
        labels:
          severity: critical
      - alert: etcdNoLeader
      ......略......

prometheus重新加载配置文件后会在Alert页面出现这些新的规则

Grafana配置

 etcd3.4 官方提供的模板 grafana模板：（这个模板折腾半天搞不好，最后还是用了grafana官方的模板）

https://etcd.io/docs/v3.4/op-guide/grafana.json

我选择了3070号模板

加载模板

选择数据源

 生成后效果