skywalking分布式追踪系统

可问春风

已于 2022-02-11 16:05:17 修改

阅读量452

点赞数

文章标签： skywalking

于 2021-09-14 17:08:47 首次发布

本文链接：https://blog.csdn.net/qq_44379042/article/details/120291856

版权

skywalking

官方文章
https://skywalking.apache.org/docs/main/v8.7.0/en/concepts-and-designs/overview/
配置告警
[root@zabbix_granfana config]# ll /opt/apache-skywalking/config/alarm-settings.yml
-rw-rw-r–. 1 1001 1002 3298 7月 30 20:32 /opt/apache-skywalking/config/alarm-settings.yml
官方文档参考

https://github.com/apache/skywalking/blob/v8.7.0/docs/en/setup/backend/backend-alarm.md

sky-php-agent --grpc ${SW_AGENT_COLLECTOR_BACKEND_SERVICES} --socket /tmp/sky-agent.sock > /var/log/sky-php.log 2>&1 &

while [[ true ]]; do
    sleep 1
done

skywalking 项目监控php显示日志
产生traceId ，ui显示日志等等
skywalking 项目监控java,web界面显示日志

官方文档
https://skywalking.apache.org/docs/main/v8.7.0/en/setup/service-agent/java-agent/application-toolkit-logback-1.x/

github java-agent配置

http://github.com/apache/skywalking/blob/master/docs/en/setup/service-agent/java-agent/README.md#table-of-agent-configuration-properties

[root@zabbix_granfana agent]# cp activations/apm-toolkit-logback-1.x-activation-8.7.0.jar /opt/ruoyi-admin/agent/plugins/
[root@zabbix_granfana plugins]# chmod g+w apm-toolkit-logback-1.x-activation-8.7.0.jar
[root@zabbix_granfana plugins]# chown 1001.1002 apm-toolkit-logback-1.x-activation-8.7.0.jar


# plugin.springannotation.classname_match_regex=${SW_SPRINGANNOTATION_CLASSNAME_MATCH_REGEX:}
plugin.toolkit.log.grpc.reporter.server_host=${SW_GRPC_LOG_SERVER_HOST:127.0.0.1}
plugin.toolkit.log.grpc.reporter.server_port=${SW_GRPC_LOG_SERVER_PORT:11800}
plugin.toolkit.log.grpc.reporter.max_message_size=${SW_GRPC_LOG_MAX_MESSAGE_SIZE:10485760}
plugin.toolkit.log.grpc.reporter.upstream_timeout=${SW_GRPC_LOG_GRPC_UPSTREAM_TIMEOUT:30}

配置项	说明	默认值
plugin.toolkit.log.transmit_formatted	是否以格式化或未格式化的格式传输记录的数据	true
plugin.toolkit.log.grpc.reporter.upstream_timeout	客户端向上游发送数据时将超时多长时间。单位是秒	30
plugin.toolkit.log.grpc.reporter.server_port	指定要向其报告日志数据的grpc服务器的端口	11800
plugin.toolkit.log.grpc.reporter.server_host	指定要向其报告日志数据的grpc服务器的主机	127.0.0.1
plugin.toolkit.log.grpc.reporter.max_message_size	指定grpc客户端要报告的日志数据的最大大小	10485760


查看agent/logs/skywalking-api.log报错
skywalking agent部署spring boot项目问题记录
WARN 2021-12-31 10:04:56:252 main AbstractClassEnhancePluginDefine : enhance class com.ruoyi.web.controller.system.SysDeptController by plugin org.apache.skywalking.apm.plugin.spring.mvc.v4.define.ControllerInstrumentation is not working. Because witness class org.springframework.cache.interceptor.DefaultKeyGenerator is not existed.
WARN 2021-12-31 10:04:56:252 main AbstractClassEnhancePluginDefine : enhance class com.ruoyi.web.controller.system.SysDeptController by plugin org.apache.skywalking.apm.plugin.spring.mvc.v4.define.ControllerForLowVersionInstrumentation is not working. Because witness class org.springframework.cache.interceptor.DefaultKeyGenerator is not existed.
WARN 2021-12-31 10:04:56:263 main AbstractClassEnhancePluginDefine : enhance class com.ruoyi.web.controller.system.SysDeptController by plugin org.apache.skywalking.apm.plugin.spring.mvc.v5.define.reactive.ReactiveControllerInstrumentation is not working. Because witness class org.springframework.web.reactive.result.method.InvocableHandlerMethod is not existed.
WARN 2021-12-31 10:04:56:263 main AbstractClassEnhancePluginDefine : enhance class com.ruoyi.web.controller.system.SysDeptController by plugin org.apache.skywalking.apm.plugin.spring.mvc.v3.define.ControllerInstrumentation is not working. Because witness class org.springframework.web.servlet.view.xslt.AbstractXsltView is not existed.
WARN 2021-12-31 10:04:56:303 main AbstractClassEnhancePluginDefine : enhance class com.ruoyi.web.controller.system.SysNoticeController by plugin org.apache.skywalking.apm.plugin.spring.mvc.v4.define.ControllerInstrumentation is not working. Because witness class org.springframework.cache.interceptor.DefaultKeyGenerator is not existed.
WARN 2021-12-31 10:04:56:304 main AbstractClassEnhancePluginDefine : enhance class com.ruoyi.web.controller.system.SysNoticeController by plugin org.apache.skywalking.apm.plugin.spring.mvc.v4.define.ControllerForLowVersionInstrumentation is not working. Because witness class org.springframework.cache.interceptor.DefaultKeyGenerator is not existed.
WARN 2021-12-31 10:04:56:304 main AbstractClassEnhancePluginDefine : enhance class com.ruoyi.web.controller.system.SysNoticeController by plugin org.apache.skywalking.apm.plugin.spring.mvc.v5.define.reactive.ReactiveControllerInstrumentation is not working. Because witness class org.springframework.web.reactive.result.method.InvocableHandlerMethod is not existed.
WARN 2021-12-31 10:04:56:304 main AbstractClassEnhancePluginDefine : enhance class com.ruoyi.web.controller.system.SysNoticeController by plugin org.apache.skywalking.apm.plugin.spring.mvc.v3.define.ControllerInstrumentation is not working. Because witness class org.springframework.web.servlet.view.xslt.AbstractXsltView is not existed.
WARN 2021-12-31 10:04:56:334 main AbstractClassEnhancePluginDefine : enhance class com.ruoyi.web.controller.system.SysConfigController by plugin org.apache.skywalking.apm.plugin.spring.mvc.v4.define.ControllerInstrumentation is not working. Because witness class org.springframework.cache.interceptor.DefaultKeyGenerator is not existed.
WARN 2021-12-31 10:04:56:334 main AbstractClassEnhancePluginDefine : enhance class com.ruoyi.web.controller.system.SysConfigController by plugin org.apache.skywalking.apm.plugin.spring.mvc.v4.define.ControllerForLowVersionInstrumentation is not working. Because witness class org.springframework.cache.interceptor.DefaultKeyGenerator is not existed.
WARN 2021-12-31 10:04:56:335 main AbstractClassEnhancePluginDefine : enhance class com.ruoyi.web.controller.system.SysConfigController by plugin org.apache.skywalking.apm.plugin.spring.mvc.v5.define.reactive.ReactiveControllerInstrumentation is not working. Because witness class org.springframework.web.reactive.result.method.InvocableHandlerMethod is not existed.
WARN 2021-12-31 10:04:56:335 main AbstractClassEnhancePluginDefine : enhance class com.ruoyi.web.controller.system.SysConfigController by plugin org.apache.skywalking.apm.plugin.spring.mvc.v3.define.ControllerInstrumentation is not working. Because witness class org.springframework.web.servlet.view.xslt.AbstractXsltView is not existed.
WARN 2021-12-31 10:04:56:367 main AbstractClassEnhancePluginDefine : enhance class com.ruoyi.web.controller.system.SysRegisterController by plugid


[root@zabbix_granfana plugins]# pwd
/opt/ruoyi-admin/agent/plugins
[root@zabbix_granfana plugins]# chmod g+w ./*

skywalking告警

https://www.cnblogs.com/heihaozi/p/apache-skywalking-alarm.html

[root@zabbix_granfana config]# cat alarm-settings.yml
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Sample alarm rules.
rules:
  # Rule unique name, must be ended with `_rule`.
  service_resp_time_rule:
    metrics-name: service_resp_time
    op: ">"
    threshold: 1000
    period: 10
    count: 3
    silence-period: 5
    message: Response time of service {name} is more than 1000ms in 3 minutes of last 10 minutes.
  service_sla_rule:
    # Metrics value need to be long, double or int
    metrics-name: service_sla
    op: "<"
    threshold: 8000
    # The length of time to evaluate the metrics
    period: 10
    # How many times after the metrics match the condition, will trigger alarm
    count: 2
    # How many times of checks, the alarm keeps silence after alarm triggered, default as same as period.
    silence-period: 3
    message: Successful rate of service {name} is lower than 80% in 2 minutes of last 10 minutes
  service_resp_time_percentile_rule:
    # Metrics value need to be long, double or int
    metrics-name: service_percentile
    op: ">"
    threshold: 1000,1000,1000,1000,1000
    period: 10
    count: 3
    silence-period: 5
    message: Percentile response time of service {name} alarm in 3 minutes of last 10 minutes, due to more than one condition of p50 > 1000, p75 > 1000, p90 > 1000, p95 > 1000, p99 > 1000
  service_instance_resp_time_rule:
    metrics-name: service_instance_resp_time
    op: ">"
    threshold: 1000
    period: 10
    count: 2
    silence-period: 5
    message: Response time of service instance {name} is more than 1000ms in 2 minutes of last 10 minutes
  database_access_resp_time_rule:
    metrics-name: database_access_resp_time
    threshold: 1000
    op: ">"
    period: 10
    count: 2
    message: Response time of database access {name} is more than 1000ms in 2 minutes of last 10 minutes
  endpoint_relation_resp_time_rule:
    metrics-name: endpoint_relation_resp_time
    threshold: 1000
    op: ">"
    period: 10
    count: 2
    message: Response time of endpoint relation {name} is more than 1000ms in 2 minutes of last 10 minutes
#  Active endpoint related metrics alarm will cost more memory than service and service instance metrics alarm.
#  Because the number of endpoint is much more than service and instance.
#
#  endpoint_avg_rule:
#    metrics-name: endpoint_avg
#    op: ">"
#    threshold: 1000
#    period: 10
#    count: 2
#    silence-period: 5
#    message: Response time of endpoint {name} is more than 1000ms in 2 minutes of last 10 minutes

webhooks:
#  - http://127.0.0.1/notify/
#  - http://127.0.0.1/go-wechat/

最近3分钟内服务的平均响应时间超过1秒
最近2分钟服务成功率低于80%
最近3分钟90%服务响应时间超过1秒
最近2分钟内服务实例的平均响应时间超过1秒规则中的参数属性如下

属性	含义
metrics-name	oal脚本中的度量名称
threshold	阈值，与metrics-name和下面的比较符号相匹配
op	比较操作符，可以设定>,<,=
period	多久检查一次当前的指标数据是否符合告警规则，单位分钟
count	达到多少次后，发送告警消息
silence-period	在多久之内，忽略相同的告警消息
message	告警消息内容
include-names	本规则告警生效的服务列表

webhook

wget https://bootstrap.pypa.io/pip/2.7/get-pip.py
python get-pip.py
#!/usr/bin/env python
# _*_ coding: utf-8 _*_
# Tile:
# Author:shy
#https://www.cnblogs.com/yy-cola/p/8066020.html
#https://www.cnblogs.com/qumogu/p/14317010.html
#https://www.cnblogs.com/leijiangtao/p/11757639.html
from flask import Flask,request
import smtplib
from email.mime.text import MIMEText
import time 
app = Flask(__name__)
@app.route("/send_mail",methods=["POST"])
def send_mail():
    info = request.json
    print(info,type(info))
    for i in info:
        alter_time = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(i["startTime"] / 1000))
        content = """
        警告时间:%s
        警告类型:%s
        服务名称:%s
        规则名称:%s
        详细内容:%s
        """ % (alter_time, i["scope"], i["name"], i["ruleName"], i["alarmMessage"])
        print(content)
    # 发送邮件
        mail_host="smtp.163.com"
        sender_user="user@163.com"
        sender_pwd = "password"
        sender_addr="smtp.163.com"
        rec_addr="接收信息的用户"
        message_mail = MIMEText(content,"plain",'utf-8')
        message_mail['Subject'] = 'Skywalking链路监控告警'
        message_mail['From'] = sender_addr
        message_mail['To'] = rec_addr
        server = smtplib.SMTP_SSL(mail_host)
        server.connect(mail_host,465)
        server.login(sender_user,sender_pwd)
        server.sendmail(sender_addr,rec_addr,message_mail.as_string())
    return "成功"
if __name__ == "__main__":
    app.run("127.0.0.1","5000")

skywalking的alarm-settings.yml发送邮件的接口
- http://127.0.0.1/:5000/send_mail

php docker安装以及日志显示traceid
https://github.com/SkyAPM/SkyAPM-php-sdk/blob/master/docker/Dockerfile
https://github.com/SkyAPM/SkyAPM-php-sdk/issues/458

tcpdump -i any -A -s0 -n -nn -l port 11800