k8s学习(三十八) 使用OpenTelemetry+jaeger实现链路追踪

---

前言

OpenTelemetry 可以用于从应用程序收集数据。它是一组工具、API 和 SDK 集合，我们可以使用它们来检测、生成、收集和导出遥测数据（指标、日志和追踪），以帮助分析应用的性能和行为。
这里基于K8S 1.20.2部署，其他版本可现在对应的operator和镜像即可。

---

提示：以下是本篇文章正文内容，下面案例可供参考

一、安装jaeger

下载镜像,并推送至私服

docker pull  docker.m.daocloud.io/jaegertracing/all-in-one:1.60.0
docker tag docker.m.daocloud.io/jaegertracing/all-in-one:1.60.0  172.16.10.160:88/library/jaegertracing/all-in-one:1.60.0
docker push 172.16.10.160:88/library/jaegertracing/all-in-one:1.60.0

修改jaeger.yaml，修改其中的镜像为172.16.10.160:88私服中的地址
jaeger.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
      - name: jaeger
        image: 172.16.10.160:88/library/jaegertracing/all-in-one:1.60.0
        env:
        - name: COLLECTOR_OTLP_ENABLED
          value: "true"
        ports:
        - containerPort: 16686
        - containerPort: 14268

---
apiVersion: v1
kind: Service
metadata:
  name: jaeger
spec:
  selector:
    app: jaeger
  type: ClusterIP
  ports:
    - name: ui
      port: 16686
      targetPort: 16686
      nodePort: 32186
    - name: collector
      port: 14268
      targetPort: 14268
    - name: http
      protocol: TCP
      port: 4318
      targetPort: 4318
    - name: grpc
      protocol: TCP
      port: 4317
      targetPort: 4317
  type: NodePort

执行：

apply -f jaeger.yaml

二、安装cert-manager

下载镜像并推送至私服

docker pull m.daocloud.io/quay.io/jetstack/cert-manager-cainjector:v1.15.2
docker tag m.daocloud.io/quay.io/jetstack/cert-manager-cainjector:v1.15.2 172.16.10.160:88/library/jetstack/cert-manager-cainjector:v1.15.2
docker push 172.16.10.160:88/library/jetstack/cert-manager-cainjector:v1.15.2

docker pull m.daocloud.io/quay.io/jetstack/cert-manager-controller:v1.15.2
docker tag m.daocloud.io/quay.io/jetstack/cert-manager-controller:v1.15.2 172.16.10.160:88/library/jetstack/cert-manager-controller:v1.15.2
docker push 172.16.10.160:88/library/jetstack/cert-manager-controller:v1.15.2

docker pull m.daocloud.io/quay.io/jetstack/cert-manager-acmesolver:v1.15.2
docker tag m.daocloud.io/quay.io/jetstack/cert-manager-acmesolver:v1.15.2 172.16.10.160:88/library/jetstack/cert-manager-acmesolver:v1.15.2
docker push 172.16.10.160:88/library/jetstack/cert-manager-acmesolver:v1.15.2

docker pull m.daocloud.io/quay.io/jetstack/cert-manager-webhook:v1.15.2
docker tag m.daocloud.io/quay.io/jetstack/cert-manager-webhook:v1.15.2 172.16.10.160:88/library/jetstack/cert-manager-webhook:v1.15.2
docker push 172.16.10.160:88/library/jetstack/cert-manager-webhook:v1.15.2

修改cert-manager中的镜像地址为172.16.10.160:88私服中对应的镜像地址（cert-manager下载地址：https://github.com/cert-manager/cert-manager/releases/download/v1.15.2/cert-manager.yaml）

cert-manager.yaml内容略，可直接从上面地址下载。
执行：

kubectl apply -f cert-manager.yaml

三、安装OpenTelemetry Operator

下载镜像并推送至私服

docker pull m.daocloud.io/ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.85.0
docker tag m.daocloud.io/ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.85.0 172.16.10.160:88/library/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.85.0
docker push 172.16.10.160:88/library/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.85.0

docker pull m.daocloud.io/gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1
docker tag m.daocloud.io/gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1 172.16.10.160:88/library/kubebuilder/kube-rbac-proxy:v0.13.1
docker push 172.16.10.160:88/library/kubebuilder/kube-rbac-proxy:v0.13.1

修改opentelemetry-operator.yaml中的镜像地址为172.16.10.160:88私服中对应的镜像地址（cert-manager下载地址：https://github.com/open-telemetry/opentelemetry-operator/releases/download/v0.85.0/opentelemetry-operator.yaml）

opentelemetry-operator.yaml的内容略

执行：

kubectl apply -f opentelemetry-operator.yaml

四、配置 OpenTelemetry Collector

通过创建 CR OpenTelemetryCollector 来配置 Otel 的采集器
open-telemetry-collector.yaml：

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s

    exporters:
      otlp/jaeger:
        endpoint: "jaeger.default:4317"
        tls:
          insecure: true

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [otlp/jaeger]

执行：

kubectl apply -f open-telemetry-collector.yaml

查看自定义的OpenTelemetryCollector

查看otel的pod

查看otel的service

五、配置 Instrumentation

Instrumentation 是 Otel Operator 的另一个 CRD，用于自动安装 Otel 探针和配置

open-telemetry-instrumentation.yaml：

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: instrumentation-sample
spec:
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1"
  env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: otel-collector.default:4318
  java:    
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://otel-collector.default:4317

执行：

kubectl apply -f open-telemetry-instrumentation.yaml

六、编写java示例程序并测试调用链跟踪

为 Pod 添加注解 instrumentation.opentelemetry.io/inject-java: "true" 通知 Otel Operator 该应用的类型以便注入正确的探针。
简单的示例为servicea -> serviceb

servicea和serviceb就是两个普通的java程序，servicea通过K8S服务名调用serviceb,没有其他额外配置

servera调用

package cn.ac.iscas.servicea.controller;

import cn.ac.iscas.servicea.util.CustomHttpClient;
import jakarta.annotation.PostConstruct;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.io.IOException;
import java.util.UUID;

/**
 * @author zhuquanwen
 * @version 1.0
 * @date 2024/8/9 9:50
 */
@RestController
public class TestController {
    @Value("${serverb.endpoint}")
    private String servicebEndpoint;

    private CustomHttpClient httpClient;

    @PostConstruct
    public void init() {
        httpClient = new CustomHttpClient(new CustomHttpClient.HttpClientProps());
    }

    @GetMapping("t1")
    public String t1() throws IOException, InterruptedException {
        return httpClient.doGet("http://" + servicebEndpoint + "/t1");
    }
}

servicea配置：

spring.application.name=service-a
server.port=4001
serverb.endpoint=localhost:4002

serviceb逻辑：

package cn.ac.iscas.serviceb.controller;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.UUID;

/**
 * @author zhuquanwen
 * @version 1.0
 * @date 2024/8/9 9:50
 */
@RestController
public class TestController {
    @GetMapping("t1")
    public String t1() {
        return "serverb-res-" + UUID.randomUUID();
    }
}

使用dockerfile将程序打包，并上传至私服

编写两个程序的k8s编排配置，并启动
deploy-servicea.yaml：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: servicea
spec:
  replicas: 1
  selector:
    matchLabels:
      app: servicea
  template:
    metadata:
      labels:
        app: servicea
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
      - name: servicea
        image: 172.16.10.160:88/library/test-micro/servicea:0.0.1
        ports:
        - containerPort: 4001


---
apiVersion: v1
kind: Service
metadata:
  name: servicea
spec:
  selector:
    app: servicea
  ports:
  - name: my-port
    protocol: TCP
    port: 4001
    targetPort: 4001
    nodePort: 32401
  type: NodePort

deploy-serviceb.yaml：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: serviceb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: serviceb
  template:
    metadata:
      labels:
        app: serviceb
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
      - name: serviceb
        image: 172.16.10.160:88/library/test-micro/serviceb:0.0.1
        ports:
        - containerPort: 4002


---
apiVersion: v1
kind: Service
metadata:
  name: serviceb
spec:
  selector:
    app: serviceb
  ports:
  - name: my-port
    protocol: TCP
    port: 4002
    targetPort: 4002
    nodePort: 32402
  type: NodePort

部署：

kubectl apply -f deploy-servicea.yaml
kubectl apply -f deploy-serviceb.yaml

通过describe pod 可以看到注入了一个initContainer和一些环境变量

通过浏览器访问servicea

访问jaeger,查看调用链