jaeger使用clickhouse作为存储后端

之前使用elasticsearch作为jaeger存储后端，后续需对接其他的可观测平台（例如deepflow，signoz等支持opentelemetry的平台），发现es存储消耗过大，同时查询速度变慢，所以希望找到替换存储后端，调研后选择了clickhouse。

官方预计2024年底支持clickhouse作为存储后端，现在如果要用第三方的存储，可以通过实现官方的grpc接口实现，也可以通过插件实现。

查询文档发现存在jaeger-clickhouse插件，通过Sidecar的方式部署，实现功能。

https://github.com/jaegertracing/jaeger-clickhouse

最后一个版本是0.13.0,镜像地址

Package jaeger-clickhouse · GitHub

1.修改jaeger collector yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: jaeger-collector
  namespace: jaeger
  labels:
    app: jaeger-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger-collector
  template:
    metadata:
      labels:
        app: jaeger-collector
    spec:
      volumes:
        - name: plugin-config
          configMap:
            name: jaeger-ck-conf
        - name: binary
          emptyDir: {}
      initContainers:
        - name: jaeger-clickhouse
          image: 'ghcr.io/jaegertracing/jaeger-clickhouse:0.13.0'
          command:
            - cp
            - /go/bin/jaeger-clickhouse
            - /plugin/jaeger-clickhouse
          volumeMounts:
            - name: binary
              mountPath: /plugin
      containers:
        - name: dev-jaeger-collector
          image: 'jaegertracing/jaeger-collector:1.49'
          args:
            - >-
              --grpc-storage-plugin.binary=/plugin/jaeger-clickhouse/jaeger-clickhouse
            - >-
              --grpc-storage-plugin.configuration-file=/plugin-config/config.yaml
            - '--grpc-storage-plugin.log-level=debug'
          ports:
            - containerPort: 14250
              protocol: TCP
            - containerPort: 14268
              protocol: TCP
            - containerPort: 14269
              protocol: TCP
          env:
            - name: SPAN_STORAGE_TYPE
              value: grpc-plugin
          volumeMounts:
            - name: plugin-config
              mountPath: /plugin-config
            - name: binary
              mountPath: /plugin

2.修改jaeger query yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: jaeger-query
  namespace: jaeger
  labels:
    app: jaeger-query
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger-query
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: jaeger-query
    spec:
      volumes:
        - name: jaeger-configuration-volume
          configMap:
            name: jaeger-conf
            items:
              - key: query
                path: query.yaml
              - key: ui
                path: ui.json
            defaultMode: 420
        - name: plugin-config
          configMap:
            name: jaeger-ck-conf
        - name: binary
          emptyDir: {}
      initContainers:
        - name: jaeger-clickhouse
          image: 'ghcr.io/jaegertracing/jaeger-clickhouse:0.13.0'
          command:
            - cp
            - /go/bin/jaeger-clickhouse
            - /plugin/jaeger-clickhouse
          volumeMounts:
            - name: binary
              mountPath: /plugin
      containers:
        - name: jaeger-query
          image: 'jaegertracing/jaeger-query:1.57'
          args:
            - '--query.ui-config=/conf/ui.json'
            - >-
              --grpc-storage-plugin.binary=/plugin/jaeger-clickhouse/jaeger-clickhouse
            - >-
              --grpc-storage-plugin.configuration-file=/plugin-config/config.yaml
            - '--grpc-storage-plugin.log-level=debug'
          ports:
            - containerPort: 16686
              protocol: TCP
          env:
            - name: SPAN_STORAGE_TYPE
              value: grpc-plugin
          resources:
            limits:
              cpu: '4'
              memory: 8Gi
            requests:
              memory: 8Gi
              cpu: '2'
          volumeMounts:
            - name: jaeger-configuration-volume
              mountPath: /conf
            - name: plugin-config
              mountPath: /plugin-config
            - name: binary
              mountPath: /plugin

3.添加configmap yaml

jaeger-ck-conf

此处需配置你的clickhouse地址，账号密码等

kind: ConfigMap
apiVersion: v1
metadata:
  name: jaeger-ck-conf
  namespace: jaeger
  creationTimestamp: '2024-07-18T05:59:06Z'
  labels:
    app: jaeger-ck-conf
data:
  config.yaml: |
    address: clickhouse:9000
    username: default
    password: 
    init_sql_scripts_dir:
    init_tables:
    max_span_count:
    batch_write_size:
    batch_flush_interval:
    encoding:
    replication:
    spans_table:
    spans_index_table:
    operations_table:
    ttl:
    max_num_spans:

jaeger-conf

kind: ConfigMap
apiVersion: v1
metadata:
  name: jaeger-conf
  namespace: jaeger
  labels:
    app: jaeger-conf
data:
  ui: |
    {
      "dependencies": {
        "dagMaxNumServices": 200,
        "menuEnabled": true
      },
      "monitor": {
        "menuEnabled": true
      },
      "archiveEnabled": true,
      "tracking": {
        "gaID": "UA-000000-2",
        "trackErrors": true
      },
      "menu": [
        {
          "label": "About Jaeger",
          "items": [
            {
              "label": "GitHub",
              "url": "https://github.com/jaegertracing/jaeger"
            },
            {
              "label": "Docs",
              "url": "http://jaeger.readthedocs.io/en/latest/"
            }
          ]
        }
      ],
      "search": {
        "maxLookback": {
          "label": "2 Days",
          "value": "2d"
        },
        "maxLimit": 1500
      },
      "linkPatterns": [{
        "type": "process",
        "key": "jaeger.version",
        "url": "https://github.com/jaegertracing/jaeger-client-java/releases/tag/#{jaeger.version}",
        "text": "Information about Jaeger release #{jaeger.version}"
      }]
    }

4.部署至k8s，链路数据成功存入ck

后续jaeger官方支持ck后，可以直接对接，不需要通过插件的方式，预计24年底再进行一次改造。