腾讯mini项目【指标监控服务重构】- 2023-07-14

今日已办

SigNoz

介绍

SigNoz 是一款开源应用程序性能监控工具，可帮助您监控应用程序并解决问题。 SigNoz 使用分布式跟踪来获取软件堆栈的可见性。

1. 监控应用程序指标，例如延迟、每秒请求数、错误率
2. 监控基础架构指标，例如 CPU 利用率或内存使用情况
3. 跨服务跟踪用户请求
4. 设置指标警报
5. 找到导致问题的根本原因
6. 可视化请求追踪

如何工作

SigNoz 使用 OpenTelemetry 收集数据。因此，SigNoz 支持 OpenTelemetry 支持的所有框架和语言。SigNoz 聚合所有收集的数据，可以使用仪表板查看与应用程序相关的所有指标和跟踪

架构

• OpenTelemetry Collector: Collects telemetry data from your services and applications.
• ClickHouse: An open-source, high performance columnar OLAP database management system.
• Query Service: The interface between the front-end and ClickHouse
• Frontend: The user interface, built in ReactJS and TypeScript.

安装

使用Docker Compose

git clone -b main https://github.com/SigNoz/signoz.git && cd signoz/deploy/
docker-compose -f docker/clickhouse-setup/docker-compose.yaml up -d

访问：http://localhost:3301/

容器signoz-otel-collector异常

Error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": code: 60, message: Table signoz_traces.distributed_signoz_index_v2 doesn't exist 2023/02/28 10:25:21 application run finished with error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": code: 60, message: Table signoz_traces.distributed_signoz_index_v2 doesn't exist

signoz-otel-collector issue

Fixing Permission problem with volume in wsl

问题解决正常启动

接入

Go OpenTelemetry Instrumentation | SigNoz

初始化OTEL-SDK的TracerProvider对象

// InitTracer
// @Description 初始化OTEL-SDK的TracerProvider对象
// @Author xzx 2023-07-14 21:23:10
// @Return func(context.Context) error
func InitTracer() func(context.Context) error {
	// 根据配置使用安全或非安全的连接方式
	secureOption := otlptracegrpc.WithTLSCredentials(credentials.NewClientTLSFromCert(nil, ""))
	if len(config.Insecure) > 0 {
		secureOption = otlptracegrpc.WithInsecure()
	}
	// 创建OTLP Exporter
	exporter, err := otlptrace.New(
		context.Background(),
		otlptracegrpc.NewClient(
			secureOption,
			otlptracegrpc.WithEndpoint(config.CollectorURL),
		),
	)
	if err != nil {
		log.Logger.Fatal(err.Error())
	}
	// 创建资源对象
	resources, err := resource.New(
		context.Background(),
		resource.WithAttributes(
			attribute.String("service.name", config.ServiceName),
			attribute.String("library.language", "go"),
		),
	)
	if err != nil {
		log.Logger.Error(fmt.Sprintf("Could not set resources: %s", err.Error()))
	}
	// 创建并注册TracerProvider对象
	otel.SetTracerProvider(
		sdktrace.NewTracerProvider(
			sdktrace.WithSampler(sdktrace.AlwaysSample()),
			sdktrace.WithBatcher(exporter),
			sdktrace.WithResource(resources),
		),
	)
	log.Logger.Info("SetTracerProvider successfully")
	return exporter.Shutdown
}

profile模拟接入

	r := gin.Default()
	r.Use(otelgin.Middleware(config.ServiceName))
	r.GET("/ping", func(c *gin.Context) {
		c.JSON(200, gin.H{
			"message": "pong",
		})
	})
	server := &http.Server{
		Addr:           ":8888",
		Handler:        r,
		ReadTimeout:    20 * time.Second,
		WriteTimeout:   20 * time.Second,
		MaxHeaderBytes: 1 << 20}

	err := server.ListenAndServe()

Getting Started

全局注册完Tracer后根据示例代码 Trace Instrumentation

func (profileCtx *ProfileContext) UnpackKafkaMessage(ctx context.Context) (needBreak bool, tpsStatus string, contextErr error) {
	ctx, span := otel.Tracer("UnpackKafkaMessage").Start(ctx, "UnpackKafkaMessage")
	//profileCtx.Ctx = newCtx
	// 反序列化，存入通用结构体
	if contextErr = json.Unmarshal(profileCtx.msg.Value, &profileCtx.Event); contextErr != nil {
		profileCtx.Status = state.StatusUnmarshalError
		span.End()
		return
	}

	log.Logger.Info("[UnpackKafkaItem] unpack kafka item success", zap.Any("event", profileCtx.Event))
	span.End()
	return
}

分析原因：上面的Trace是本地运行的（Receiver有端口映射），用docker compose运行需要考虑每个docker compose网络都是一个独立的网络命名空间，其内部的容器名称在其他网络中不可见

将容器添加到统一网络用容器名访问

明日待办

1. 继续接入SigNoz
2. 熟悉OTEL-SDK