OpenTracing
简介
OpenTracing 提供了一套平台无关、厂商无关的 API,这样不同的组织或者开发人员就能够更加方便的添加或更换追踪系统的实现。 OpenTracing API 中的一些概念和术语,在不同的语言环境下都是共享。
接口规范
- Tracer Interface
- Span Interface
- SpanContext Interface
- Global and No-op Tracers
jaeger install
# run jaeger
docker run -d \
--rm \
--name jaeger \
-p6831:6831/udp \
-p16686:16686 \
jaegertracing/all-in-one:1.16.0
# run demo
docker run -d \
--name hotrod\
--rm \
--link jaeger \
--env JAEGER_AGENT_HOST=jaeger \
--env JAEGER_AGENT_PORT=6831 \
-p8080-8083:8080-8083 \
jaegertracing/example-hotrod:1.16.0 \
all
jaeger 组件简介
jaeger-agent
jaeger client sdk 产生的数据由 jaeger-agent负责采集并发送到jaeger-collector
Port | Protocol | Function |
---|---|---|
6831 | UDP | accept jaeger.thrift in compact Thrift protocol used by most current Jaeger clients |
6832 | UDP | accept jaeger.thrift in binary Thrift protocol used by Node.js Jaeger client (because thriftrw npm package does not support compact protocol) |
5778 | HTTP | serve configs, sampling strategies |
5775 | UDP | accept zipkin.thrift in compact Thrift protocol (deprecated; only used by very old Jaeger clients, circa 2016) |
14271 | HTTP | Healthcheck at / and metrics at /metrics |
jaeger-collector
jaeger-collector 收集 agent发送过来的数据并分发给后端存储
Port | Protocol | Function |
---|---|---|
14267 | TChanel | used by jaeger-agent to send spans in jaeger.thrift format |
14250 | gRPC | used by jaeger-agent to send spans in model.proto format |
14268 | HTTP | can accept spans directly from clients in jaeger.thrift format over binary thrift protocol |
9411 | HTTP | can accept Zipkin spans in Thrift, JSON and Proto (disabled by default) |
14269 | HTTP | Healthcheck at / and metrics at /metrics |
Storage Backends
通过环境变量 SPAN_STORAGE_TYPE 来设置后端存储
- badger和Memory 仅在all-in-one安装模式下使用
- elasticsearch
- cassandra noSql数据库
- kafka 仅作为缓存,将数据分发到es或者Cassandra
Storage plugin
jaeger 支持 gRPC通信的后端存储
- InfluxDB
- Logz.io
jaeger-ingester
jaeger-ingester可以消费kafka内的数据分发到es或者cassandra
Port | Protocol | Function |
---|---|---|
14270 | HTTP | Healthcheck at / and metrics at /metrics |
jaeger-query
该服务是无状态应用,一般部署在负载均衡(nginx)后面
Port | Protocol | Function |
---|---|---|
16686 | HTTP | /api/* endpoints and Jaeger UI at / |
16687 | HTTP | Healthcheck at / and metrics at /metrics |
spark-dependencies
生产环境下需要部署spark-dependencies 分析调用链信息
验证Jaeger Client自动埋点
java-specialagent
java-specialagent是java应用自动埋点第三方引擎库,详细说明请点击github传送门
-
目前测试情况是springboot web应用和采用okhttp和Dropwizard框架编译的java应用都可以实现自动埋点,但是阿里的sofarpc无法自动埋点。
-
添加自动埋点的java应用会延长启动时间
本地运行 jaeger all-in-one应用
docker run -d --rm \
--name jaeger \
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-p 14250:14250 \
-p 9411:9411 \
jaegertracing/all-in-one:1.16.0
运行 springboot demo
验证可行,能够在jaeger ui中查询到服务追踪信息
java -javaagent:opentracing-specialagent-1.5.8.jar -Dsa.tracer=jaeger -Dsa.log.level=INFO -DJAEGER_SERVICE_NAME="springboot" -jar springboot-helloworld-1.0-SNAPSHOT.jar --server.port=9000
运行PRC demo
验证可行,能够在jaeger ui中查询到服务追踪信息
java -javaagent:opentracing-specialagent-1.5.8.jar -Dsa.tracer=jaeger -Dsa.log.level=INFO -DJAEGER_SERVICE_NAME="formatter" -jar jaeger-demo/formatter/target/formatter-1.0-SNAPSHOT.jar
java -javaagent:opentracing-specialagent-1.5.8.jar -Dsa.tracer=jaeger -Dsa.log.level=INFO -DJAEGER_SERVICE_NAME="publisher" -jar jaeger-demo/publisher/target/publisher-1.0-SNAPSHOT.jar
java -javaagent:opentracing-specialagent-1.5.8.jar -Dsa.tracer=jaeger -Dsa.log.level=INFO -DJAEGER_SERVICE_NAME="callserver" -jar jaeger-demo/callserver/target/callserver-1.0-SNAPSHOT.jar
运行sofarpc demo
验证失败,在jaeger ui中没有查询到追踪信息
- rpcclient 第一次发起请求会报错,后续请求可正常收到响应
- 报错信息如下
2020-03-03 14:45:50.730 INFO 34856 --- [ main] com.citicbank.RpcclientApplication : Started RpcclientApplication in 7.443 seconds (JVM running for 19.898)
com.alipay.sofa.rpc.context.RpcInvokeContext@5013c6bc{timeout=null, targetURL='null', targetGroup='null', responseCallback=null, future=null, map={}, requestBaggage={service=mosn}, responseBaggage={}}
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: com.alipay.sofa.rpc.core.exception.SofaTimeOutException: com.alipay.remoting.rpc.exception.InvokeTimeoutException: Rpc invocation timeout[responseCommand TIMEOUT]! the address is 127.0.0.1:12200
at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.convertToRpcException(BoltClientTransport.java:327)
at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.syncSend(BoltClientTransport.java:251)
at com.alipay.sofa.rpc.client.AbstractCluster.doSendMsg(AbstractCluster.java:535)
at com.alipay.sofa.rpc.client.AbstractCluster.sendMsg(AbstractCluster.java:506)
at com.alipay.sofa.rpc.filter.ConsumerInvoker.invoke(ConsumerInvoker.java:60)
at com.alipay.sofa.rpc.filter.sofatracer.ConsumerTracerFilter.invoke(ConsumerTracerFilter.java:66)
at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:98)
at com.alipay.sofa.rpc.filter.RpcReferenceContextFilter.invoke(RpcReferenceContextFilter.java:80)
at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:98)
at com.alipay.sofa.rpc.filter.ConsumerExceptionFilter.invoke(ConsumerExceptionFilter.java:37)
at com.alipay.sofa.rpc.filter.FilterInvoker.invoke(FilterInvoker.java:98)
at com.alipay.sofa.rpc.filter.FilterChain.invoke(FilterChain.java:262)
at com.alipay.sofa.rpc.client.AbstractCluster.filterChain(AbstractCluster.java:499)
at com.alipay.sofa.rpc.client.FailoverCluster.doInvoke(FailoverCluster.java:67)
at com.alipay.sofa.rpc.client.AbstractCluster.invoke(AbstractCluster.java:288)
at com.alipay.sofa.rpc.client.ClientProxyInvoker.invoke(ClientProxyInvoker.java:83)
at com.citicbank.service.RpcDemoService_proxy_0.sayHello(RpcDemoService_proxy_0.java)
at com.citicbank.service.RpcDemoClientImpl.sayClientAnnotation(RpcDemoClientImpl.java:19)
at com.citicbank.RpcclientApplication.main(RpcclientApplication.java:22)
... 8 more
Caused by: com.alipay.remoting.rpc.exception.InvokeTimeoutException: Rpc invocation timeout[responseCommand TIMEOUT]! the address is 127.0.0.1:12200
at com.alipay.remoting.rpc.RpcResponseResolver.preProcess(RpcResponseResolver.java:83)
at com.alipay.remoting.rpc.RpcResponseResolver.resolveResponseObject(RpcResponseResolver.java:54)
at com.alipay.remoting.rpc.RpcRemoting.invokeSync(RpcRemoting.java:186)
at com.alipay.remoting.rpc.RpcClientRemoting.invokeSync(RpcClientRemoting.java:64)
at com.alipay.remoting.rpc.RpcClient.invokeSync(RpcClient.java:355)
at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.doInvokeSync(BoltClientTransport.java:275)
at com.alipay.sofa.rpc.transport.bolt.BoltClientTransport.syncSend(BoltClientTransport.java:248)
... 25 more
Unknown instrumentable state for thread: DestroyJavaVM
运行命令
java -javaagent:opentracing-specialagent-1.5.8.jar -Dsa.tracer=jaeger -Dsa.log.level=INFO -DJAEGER_SERVICE_NAME="client" -jar rpcclient-0.0.1-SNAPSHOT.jar
java -javaagent:opentracing-specialagent-1.5.8.jar -Dsa.tracer=jaeger -Dsa.log.level=INFO -DJAEGER_SERVICE_NAME="server" -jar rpcserver-0.0.1-SNAPSHOT.jar
java-specialagent 支持的插件
OpenTracing Plugin (link to plugin implementation) | SpecialAgent Rule (link to rule implementation) | Min Version (min supported) | Max Version (max supported) |
---|---|---|---|
Akka Actor | [akka:actor ][akka-actor] | 2.5.0 | 2.6.1 |
Akka Http | [akka:http ][akka-http] | 10.1.0 | 10.1.10 |
Apache Camel | [camel ][camel] | 2.24.0 | 2.24.2 |
Apache HttpClient | [apache:httpclient ][apache-httpclient] | 4.4 | 4.5.9 |
Async Http Client | [asynchttpclient ][asynchttpclient] | 2.7.0 | 2.10.4 |
AWS SDK | [aws:sdk:1 ][aws-sdk-1] | 1.11.79 | 1.11.5.1 |
[aws:sdk:2 ][aws-sdk-2] | 2.1.4 | 2.7.15 | |
Cassandra Driver | [cassandra:driver:3 ][cassandra-driver-3] | 3.0.0 | 3.7.2 |
[cassandra:driver:4 ][cassandra-driver-4] | 4.0.0 | 4.2.0 | |
Couchbase Client | [couchbase-client ][couchbase-client] | 2.7.3 | 2.7.11 |
Dynamic | [dynamic (configurable)][dynamic] | * | * |
Elasticsearch Client | [elasticsearch:client-transport ][elasticsearch-7-transport-client][ elasticsearch:client-rest ][elasticsearch-7-rest-client] | 6.4.0 | 7.3.1 |
Feign | [feign ][feign] | 9.0.0 | 10.4.0 |
Google Http Client | [google-http-client ][google-http-client] | 1.19.0 | 1.33.0 |
Grizzly AsyncHttpClient | [grizzly:ahc ][grizzly-ahc] | 1.15 | 1.15 |
Grizzly HTTP Server | [grizzly:http-server ][grizzly-http-server] | 2.3.35 | 2.3.35 |
GRPC | [grpc ][grpc] | 1.6.0 | 1.23.0 |
Hazelcast | [hazelcast ][hazelcast] | 3.12 | 3.12.4 |
Java Concurrent API [java.util.concurrent ] | [concurrent ][concurrent] | 1.5 | 11 |
[Java JDBC API [java.sql ]][java-jdbc] | [jdbc (configurable)][jdbc] | 3.1 | 4.3 |
[Java JMS API [javax.jms ]][java-jms] | [jms:1 ][jms-1] | 1.1 | 1.1 |
[jms:2 ][jms-2] | 2.0 | 2.0a | |
Java Servlet API [javax.servlet ] | [servlet (configurable)][servlet] | 2.3 | 3.1 |
Jetty | 7.6.21.v20160908 | 9.2.15.v20160210 | |
Tomcat | 7.0.65 | 9.0.27 | |
Java Thread [java.lang.Thread ] | [thread ][thread] | 1.0 | 11 |
HttpURLConnection [java.net.HttpURLConnection ] | [httpurlconnection ][httpurlconnection] | 1.1 | 11 |
JAX-RS Client | [jax-rs ][jaxrs] | 2.0 | 2.1 |
Jedis Client | [jedis ][jedis] | 2.7.0 | 3.1.0 |
Kafka Client | [kafka:client ][kafka-client] | 1.1.0 | 2.3.0 |
Lettuce Client | [lettuce:5.0 ][lettuce-5.0] | 5.0.0.RELEASE | 5.0.5.RELEASE |
[lettuce:5.1 ][lettuce-5.1] | 5.1.0.M1 | 5.1.8.RELEASE | |
[lettuce:5.2 ][lettuce-5.2] | 5.2.0.RELEASE | 5.2.0.RELEASE | |
MongoDB Driver | [mongo:driver ][mongo-driver] | 3.9.0 | 3.11.0 |
Neo4j Driver | [neo4j:driver ][neo4j-driver] | 4.0.0 | 4.0.0 |
Netty | [netty ][netty] | 4.1.0 | 4.1.44 |
[OkHttp][java-okhttp] | [okhttp ][okhttp] | 3.5.0 | 4.2.2 |
Play Framework | [play ][play] | 2.6.0 | 2.7.3 |
Play WS | [play:ws ][play-ws] | 2.0.0 | 2.0.7 |
Pulsar Client | [pulsar:client ][pulsar-client] | 2.2.0 | 2.4.2 |
Pulsar Functions | [pulsar-functions (configurable)][pulsar-functions] | 2.2.0 | 2.4.2 |
RabbitMQ Client | [rabbitmq:client ][rabbitmq-client] | 5.0.0 | 5.7.3 |
Reactor | [reactor ][reactor] | 3.2.3.RELEASE | 3.2.3.RELEASE |
Redisson | [redisson ][redisson] | 3.11.0 | 3.11.5 |
RxJava 2 | [rxjava:2 ][rxjava-2] | 2.1.0 | 2.2.12 |
Spring JMS | [spring:jms ][spring-jms] | 5.0.0.RELEASE | 5.2.2.RELEASE |
Spring Kafka | [spring:kafka ][spring-kafka] | 2.2.0.RELEASE | 2.3.3.RELEASE |
Spring Messaging | [spring:messaging ][spring-messaging] | 5.1.0.RELEASE | 5.2.2.RELEASE |
Spring RabbitMQ | [spring:rabbitmq ][spring-rabbitmq] | 2.0.0.RELEASE | 2.2.1.RELEASE |
Spring WebFlux | [spring:webflux ][spring-webflux] | 5.1.0.RELEASE | 5.2.1.RELEASE |
Spring Boot WebSocket STOMP | [spring:websocket ][spring-websocket] | 2.1.0.RELEASE | 2.1.8.RELEASE |
Spring [@Async and @Scheduled ] | [spring:scheduling ][spring-scheduling] | 5.0.0.RELEASE | 5.2.1.RELEASE |
Spring Web | [spring:web:3 ][spring-web-3] | 3.0.3.RELEASE | 3.2.18.RELEASE |
[spring:web:4.0 ][spring-web-4.0] | 4.0.0.RELEASE | 4.0.9.RELEASE | |
[spring:web:4.x ][spring-web-4] | 4.1.0.RELEASE | 4.3.25.RELEASE | |
[spring:web:5 ][spring-web-5] | 5.0.0.RELEASE | 5.2.1.RELEASE | |
Spring Web MVC | [spring:webmvc:3 ][spring-webmvc-3] | 3.0.2.RELEASE | 3.2.18.RELEASE |
[spring:webmvc:4 ][spring-webmvc-4] | 4.0.0.RELEASE | 4.3.25.RELEASE | |
[spring:webmvc:5 ][spring-webmvc-5] | 5.0.0.RELEASE | 5.2.1.RELEASE | |
Spymemcached | [spymemcached ][spymemcached] | 2.11.0 | 2.12.3 |
Thrift | [thrift ][thrift] | 0.10.0 | 0.12.0 |
Zuul | [zuul ][zuul] | 1.0.0 | 1.3.1 |
关于skywalking
Skywalking 是一个APM系统,即应用性能监控系统,为微服务架构和云原生架构系统设计。完全支持opentracing接口,它通过探针自动收集所需的指标,并进行分布式追踪。通过这些调用链路以及指标,Skywalking APM会感知应用间关系和服务间关系,并进行相应的指标统计。目前支持链路追踪和监控应用组件如下,基本涵盖主流框架和容器,包括sofarpc、 dubbo等。详细列表如下
- HTTP Server
- Tomcat 7
- Tomcat 8
- Tomcat 9
- Spring Boot Web 4.x
- Spring MVC 3.x, 4.x 5.x with servlet 3.x
- Nutz Web Framework 1.x
- Struts2 MVC 2.3.x -> 2.5.x
- Resin 3 (Optional¹)
- Resin 4 (Optional¹)
- Jetty Server 9
- Spring WebFlux 5.x
- Undertow 1.3.0.Final -> 2.0.27.Final
- RESTEasy 3.1.0.Final -> 3.7.0.Final
- Play Framework 2.6.x -> 2.8.x
- Light4J Microservices Framework 1.6.x -> 2.x
- Netty SocketIO 1.x
- HTTP Client
- Feign 9.x
- Netflix Spring Cloud Feign 1.1.x, 1.2.x, 1.3.x
- Okhttp 3.x
- Apache httpcomponent HttpClient 2.0 -> 3.1, 4.2, 4.3
- Spring RestTemplete 4.x
- Jetty Client 9
- Apache httpcomponent AsyncClient 4.x
- HTTP Gateway
- Spring Cloud Gateway 2.1.x.RELEASE (Optional²)
- JDBC
- Mysql Driver 5.x, 6.x, 8.x
- Oracle Driver (Optional¹)
- H2 Driver 1.3.x -> 1.4.x
- Sharding-JDBC 1.5.x
- ShardingSphere 3.0.0, 4.0.0-RC1
- PostgreSQL Driver 8.x, 9.x, 42.x
- RPC Frameworks
- Dubbo 2.5.4 -> 2.6.0
- Dubbox 2.8.4
- Apache Dubbo 2.7.0
- Motan 0.2.x -> 1.1.0
- gRPC 1.x
- Apache ServiceComb Java Chassis 0.1 -> 0.5,1.x
- SOFARPC 5.4.0
- Armeria 0.63.0 -> 0.98.0
- Apache Avro 1.7.0 - 1.8.x
- MQ
- NoSQL
- Redis
- MongoDB Java Driver 2.13-2.14, 3.4.0-3.11.1
- Memcached Client
- Spymemcached 2.x
- Xmemcached 2.x
- Elasticsearch
- transport-client 5.2.x-5.6.x
- rest-high-level-client 6.7.1-6.8.4
- rest-high-level-client 7.0.0-7.5.2
- Solr
- SolrJ 7.x
- Cassandra 3.x
- cassandra-java-driver 3.7.0-3.7.2
- Service Discovery
- Distributed Coordination
- Zookeeper 3.4.x (Optional² & Except 3.4.4)
- Spring Ecosystem
- Spring Bean annotations(@Bean, @Service, @Component, @Repository) 3.x and 4.x (Optional²)
- Spring Core Async SuccessCallback/FailureCallback/ListenableFutureCallback 4.x
- Spring Transaction 4.x and 5.x (Optional²)
- Hystrix: Latency and Fault Tolerance for Distributed Systems 1.4.20 -> 1.5.12
- Scheduler
- Elastic Job 2.x
- OpenTracing community supported
- Canal: Alibaba mysql database binlog incremental subscription & consumer components 1.0.25 -> 1.1.2
- JSON
- GSON 2.8.x (Optional²)
- Vert.x Ecosystem
- Vert.x Eventbus 3.2+
- Vert.x Web 3.x
- Thread Schedule Framework
- Spring @Async 4.x and 5.x
- Cache
- Ehcache 2.x
- Kotlin
- Coroutine 1.0.1 -> 1.3.x (Optional²)
skywalking demo 实例
[北京服务器] http://122.112.182.72:8080/