微服务可观测之一站式性能监控

微服务可观测之一站式性能监控

前置概要

前几篇文章，发布了 Spring Boot Dubbo 微服务应用到 Kubernetes，并实现了 渐进式交付。
本篇实现微服务可观测，全面的性能监控，应用接入 Apache SkyWalking.

接入说明

之前 Redis,Zookeeper,MySQL 和 自己开发应用 都是通过 Helm 部署到 K8S,
本篇采用预发布的方式，在本机部署 SkyWalking，然后通过 KtConnect 预发布。

KtConnect

KtConnect 提供了本地和测试环境集群的双向互联能力。

• https://alibaba.github.io/kt-connect (文档)
• https://github.com/alibaba/kt-connect
• https://github.com/alibaba/kt-connect/releases/download/v0.3.7/ktctl_0.3.7_Windows_x86_64.zip

SkyWalking OAP

• https://archive.apache.org/dist/skywalking/9.4.0/

本地默认启动(H2)

SW_AGENT_COLLECTOR_BACKEND_SERVICES 11800
D:\SkyWalking\apache-skywalking-apm-bin\bin\oapService.bat

Dashboard 18080
D:\SkyWalking\apache-skywalking-apm-bin\webapp\application.yml 改监听端口
D:\SkyWalking\apache-skywalking-apm-bin\bin\webappService.bat

备注： Helm 部署方式

https://archive.apache.org/dist/skywalking/kubernetes/4.4.0/

把本地 SkyWalking 预发布到K8S，通过 ktctl.exe

PS C:\Users\Jazz\Desktop> ktctl preview skywalking-oap --skipPortChecking --expose 11800:11800
8:06PM INF Using cluster context kind-kind (kind-kind)
8:06PM INF KtConnect 0.3.7 start at 2472 (windows amd64)
8:06PM INF Fetching cluster time ...
8:06PM INF Successful create config map skywalking-oap-kt-ehtuf
8:06PM INF Deploying shadow pod skywalking-oap-kt-ehtuf in namespace default
8:06PM INF Waiting for pod skywalking-oap-kt-ehtuf ...
8:06PM INF Pod skywalking-oap-kt-ehtuf is ready
8:06PM INF Created shadow pod skywalking-oap-kt-ehtuf
8:06PM INF Forwarding pod skywalking-oap-kt-ehtuf to local via port 11800:11800
8:06PM INF Port forward local:59520 -> pod skywalking-oap-kt-ehtuf:22 established
8:06PM INF Reverse tunnel 0.0.0.0:11800 -> 127.0.0.1:11800 established
8:06PM INF Forward remote skywalking-oap-kt-ehtuf:11800:11800 -> 127.0.0.1:11800:11800
8:06PM INF ---------------------------------------------------------------
8:06PM INF  Now you can access your local service in cluster by name 'skywalking-oap'
8:06PM INF ---------------------------------------------------------------

PS C:\Users\Jazz> kubectl get svc skywalking-oap -o wide
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE   SELECTOR
skywalking-oap   ClusterIP   10.96.169.198   <none>        11800/TCP   58m   kt-role=shadow-preview,kt-target=LFpInAXXKEeXXoNXbCHv

SkyWalking Agent

接入文档
https://skywalking.apache.org/docs/skywalking-java/v8.15.0/en/setup/service-agent/java-agent/containerization/#kubernetes

demo-infra helm deployment.yaml 模板

sidecar 方式接入

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "svc.fullname" . }}
  labels:
    {{- include "svc.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "svc.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "svc.selectorLabels" . | nindent 8 }}
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      serviceAccountName: {{ include "svc.serviceAccountName" . }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      volumes:
        - name: skywalking-agent
          emptyDir: { }
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          volumeMounts:
            - name: skywalking-agent
              mountPath: /skywalking
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: {{ .Values.service.port }}
              protocol: TCP
          livenessProbe:
            initialDelaySeconds: 120
            periodSeconds: 10
            timeoutSeconds: 6
            successThreshold: 1
            failureThreshold: 6
            httpGet:
              path: /
              port: http
          readinessProbe:
            initialDelaySeconds: 120
            periodSeconds: 10
            timeoutSeconds: 6
            successThreshold: 1
            failureThreshold: 6
            httpGet:
              path: /
              port: http
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          env:
            - { name: TZ, value: Asia/Shanghai }
            - { name: SW_AGENT_NAME, value: demo-infra }
            - { name: SW_AGENT_COLLECTOR_BACKEND_SERVICES, value: skywalking-oap:11800 }
            - { name: JAVA_TOOL_OPTIONS, value: "-javaagent:/skywalking/agent/skywalking-agent.jar" }

      initContainers:
        - name: agent-container
          image: apache/skywalking-java-agent:8.15.0-java17
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - mountPath: /agent
              name: skywalking-agent
          command: [ "/bin/sh" ]
          args: [ "-c", "cp -R /skywalking/agent /agent/ " ]

      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}

关于 agent.config

文档：https://skywalking.apache.org/docs/skywalking-java/v8.15.0/en/setup/service-agent/java-agent/setting-override/
本篇采用环境变量方式覆盖了 SW_AGENT_NAME 和 SW_AGENT_COLLECTOR_BACKEND_SERVICES

调整 helm values.yaml

securityContext: {
  runAsUser: 0
}

root 用户启动，让 skywalking-api.log 有写入权限
/skywalking/agent/logs/skywalking-api.log

Grpc 发送数据到 OAP

接入文档
https://skywalking.apache.org/docs/skywalking-java/v8.15.0/en/setup/service-agent/java-agent/application-toolkit-logback-1.x/

gradle build.xml 添加依赖

implementation 'org.apache.skywalking:apm-toolkit-logback-1.x:8.15.0'

logback-spring.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>

    <springProperty scope="context" name="springAppName" source="spring.application.name"/>
    <property name="LOG_HOME" value="logs"/>

    <conversionRule conversionWord="ip" converterClass="com.lab.common.logback.LogIpConfig"/>

    <property name="LOG_PATTERN"
              value="%d{yyyy-MM-dd HH:mm:ss.SSS} | %ip | %-5level | ${PID:-} | %thread | %sw_ctx | %logger{5}#%method | %msg%n"/>

    <property name="LOG_PATTERN_COLOR"
              value="%d{yyyy-MM-dd HH:mm:ss.SSS} | %ip | %highlight(%-5level) | ${PID:-} | %boldYellow(%thread) | %sw_ctx | %boldBlue(%logger{5}#%method) | %msg%n"/>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
                <Pattern>${LOG_PATTERN_COLOR}</Pattern>
            </layout>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <appender name="grpc-log" class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.log.GRPCLogClientAppender">
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
                <Pattern>${LOG_PATTERN_COLOR}</Pattern>
            </layout>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <appender name="DAY_DEBUG" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>${LOG_HOME}/${springAppName}_debug-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <maxFileSize>50MB</maxFileSize>
            <maxHistory>5</maxHistory>
        </rollingPolicy>
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>DEBUG</level>
        </filter>
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
                <Pattern>${LOG_PATTERN_COLOR}</Pattern>
            </layout>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <appender name="DAY_INFO" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>${LOG_HOME}/${springAppName}_info-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <maxFileSize>50MB</maxFileSize>
            <maxHistory>10</maxHistory>
        </rollingPolicy>
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>INFO</level>
        </filter>
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
                <Pattern>${LOG_PATTERN_COLOR}</Pattern>
            </layout>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <appender name="DAY_ERROR" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>${LOG_HOME}/${springAppName}_error-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <maxFileSize>50MB</maxFileSize>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
                <Pattern>${LOG_PATTERN_COLOR}</Pattern>
            </layout>
            <charset>utf8</charset>
        </encoder>
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>ERROR</level>
        </filter>
    </appender>

    <logger name="org.springframework" level="ERROR"/>
    <logger name="org.apache.commons" level="ERROR"/>
    <logger name="org.apache.kafka" level="WARN"/>
    <logger name="org.apache.kafka.clients.producer.ProducerConfig" level="WARN"/>
    <logger name="org.apache.kafka.clients.consumer.ConsumerConfig" level="WARN"/>
    <logger name="com.alibaba.nacos.client.config.impl" level="WARN"/>
    <logger name="org.apache.kafka.clients.consumer.internals.AbstractCoordinator" level="WARN"/>
    <logger name="org.apache.kafka.clients.consumer.internals.ConsumerCoordinator" level="WARN"/>
    <logger name="org.apache.kafka.clients.FetchSessionHandler" level="WARN"/>
    <logger name="org.apache.kafka.clients.consumer.internals.Fetcher" level="WARN"/>
    <logger name="springfox.documentation.spring.web.readers.operation.CachingOperationNameGenerator" level="WARN"/>
    <logger name="org.hibernate.engine.QueryParameters" level="WARN"/>
    <logger name="org.springframework.jdbc.core.JdbcTemplate" level="WARN"/>
    <logger name="org.springframework.jdbc.core.StatementCreatorUtils" level="WARN"/>
    <logger name="com.alibaba.cloud.dubbo" level="WARN"/>
    <logger name="org.apache.dubbo" level="WARN"/>
    <logger name="com.alibaba.nacos.client" level="WARN"/>

    <logger name="jdbc.sqlonly" level="ERROR"/>
    <logger name="jdbc.audit" level="ERROR"/>
    <logger name="jdbc.resultset" level="ERROR"/>
    <logger name="jdbc.connection" level="ERROR"/>
    <logger name="jdbc.sqltiming" level="DEBUG" additivity="false">
        <appender-ref ref="DAY_DEBUG"/>
    </logger>
    <logger name="jdbc.resultsettable" level="OFF"/>

    <logger name="com.lab" level="DEBUG" additivity="false">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="DAY_ERROR"/>
        <appender-ref ref="DAY_INFO"/>
        <appender-ref ref="DAY_DEBUG"/>
        <appender-ref ref="grpc-log"/>
    </logger>

    <root level="INFO">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="DAY_ERROR"/>
        <appender-ref ref="DAY_INFO"/>
        <appender-ref ref="DAY_DEBUG"/>
        <appender-ref ref="grpc-log"/>
    </root>

</configuration>

打印 SkyWalking context

SW_CTX 包含：[$serviceName,$instanceName,$traceId,$traceSegmentId,$spanId]

至此接入完成，访问应用便会通过 Agent 和 Grpc 上报数据到 OAP，登录 Dashboard 可全面观测应用。
比如：服务实例，拓扑图，链路追踪，分布式日志，数据库性能/Slow SQL，Redis 性能 and so on ...