基于kubernetes的jaeger 链路追踪部署
- jaeger的相关知识
- 介绍
- 架构图
- 对OpenTracing的原生支持
- 多个存储后端
- 基于k8s的jaeger安装
- 1.configmap 部署: jaeger-config.yaml
- 2. jaeger-query 部署安装: jaeger-query-deployment.yml
- 3.jaeger query service的安装:jaeger-query-service.yml
- 4.jaeger-collector 部署安装:jaeger-query-collector.yml
- 5. jaeger collector service的安装:jaeger-collector-service.yml
- 6. 测试用例hotrod 与代理jaeger-agent的部署
- 7.配置ingress 来访问服务,也可以使用NodePort的方式来访问
- 使用docker启动jaeger 测试环境 所有数据都是存储到内存中
- Docker部署jaeger并使用elasticsearch作为存储引擎
- docker + query安装
- docker + agent安装
jaeger的相关知识
https://blog.csdn.net/douzizuibang/article/details/83312674
https://yq.aliyun.com/articles/514488
官网:https://www.jaegertracing.io/docs/1.7/
github上的jaeger: https://github.com/jaegertracing,https://github.com/jaegertracing/jaeger-kubernetes
介绍
Jaeger是Uber Technologies用GO语言开发的分布式跟踪系统,现已开源。它用于监视和排除基于微服务的分布式系统,包括:
- 分布式上下文传播
- 分布式事务监控
- 根本原因分析
- 服务依赖性分析
- 性能/延迟优化
架构图
对OpenTracing的原生支持
Jaeger后端,Web UI和仪器库已经完全设计为支持OpenTracing标准。
- 通过跨度参考将跟踪表示为有向非循环图(不仅仅是树)
- 支持强类型span 标记和结构化日志
- 通过行李支持通用分布式上下文传播机制
多个存储后端
Jaeger支持两个流行的开源NoSQL数据库作为跟踪存储后端:Cassandra 3.4+和Elasticsearch 6.x / 7.x. 正在进行使用其他数据库的社区实验,例如ScyllaDB,InfluxDB,Amazon DynamoDB。Jaeger还提供了一个简单的内存存储器,用于测试设置。
jaeger-agent(代理)
Jaege-agent client 与 collector 中间的代理层,监听发送过来的 spans 数据并批量发送至 collector。
jaeger-collector
Jaeger-collector接收发送自 jaeger-agent 的 trace 数据(或者可直接发送 zipkin spans 至 collector),验证、索引、转换及存储 trace 数据。
Jaeger的存储是一个可插拔的组件,目前支持Cassandra,Elasticsearch和Kafka。
jaeger-query
jaeger-query从storage 中查询 trace 及前端 ui 展示。
hotrod
测试案例部署。
基于k8s的jaeger安装
PS:因为需要使用到es所有与es与kibana、fluentd放入同一namespace
1.configmap 部署: jaeger-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: jaeger-configuration
namespace: logging
labels:
app: jaeger
jaeger-infra: configuration
data:
span-storage-type: elasticsearch
collector: |
es:
server-urls: http://elasticsearch:9200
username: elastic
password: changeme
collector:
zipkin:
http-port: 9411
query: |
es:
server-urls: http://elasticsearch:9200
username: elastic
2. jaeger-query 部署安装: jaeger-query-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jaeger-query
namespace: logging
labels:
app: jaeger
jaeger-infra: query-deployment
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: jaeger
jaeger-infra: query-pod
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "16686"
spec:
containers:
- image: jaegertracing/jaeger-query:latest
name: jaeger-query
args: ["--config-file=/conf/query.yaml"]
ports:
- containerPort: 16686
protocol: TCP
readinessProbe:
httpGet:
path: "/"
port: 16687
volumeMounts:
- name: jaeger-configuration-volume
mountPath: /conf
env:
- name: SPAN_STORAGE_TYPE
valueFrom:
configMapKeyRef:
name: jaeger-configuration
key: span-storage-type
volumes:
- configMap:
name: jaeger-configuration
items:
- key: query
path: query.yaml
name: jaeger-configuration-volume
3.jaeger query service的安装:jaeger-query-service.yml
apiVersion: v1
kind: Service
metadata:
name: jaeger-query
namespace: logging
labels:
app: jaeger
jaeger-infra: query-service
spec:
ports:
- name: jaeger-query
port: 80
protocol: TCP
targetPort: 16686
selector:
jaeger-infra: query-pod
4.jaeger-collector 部署安装:jaeger-query-collector.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jaeger-collector
namespace: logging
labels:
app: jaeger
jaeger-infra: collector-deployment
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: jaeger
jaeger-infra: collector-pod
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "14268"
spec:
containers:
- image: jaegertracing/jaeger-collector:latest
name: jaeger-collector
args: ["--config-file=/conf/collector.yaml"]
ports:
- containerPort: 14267
protocol: TCP
- containerPort: 14268
protocol: TCP
- containerPort: 9411
protocol: TCP
readinessProbe:
httpGet:
path: "/"
port: 14269
volumeMounts:
- name: jaeger-configuration-volume
mountPath: /conf
env:
- name: SPAN_STORAGE_TYPE
valueFrom:
configMapKeyRef:
name: jaeger-configuration
key: span-storage-type
volumes:
- configMap:
name: jaeger-configuration
items:
- key: collector
path: collector.yaml
name: jaeger-configuration-volume
5. jaeger collector service的安装:jaeger-collector-service.yml
apiVersion: v1
kind: List
items:
- apiVersion: v1
kind: Service
metadata:
name: jaeger-collector
namespace: logging
labels:
app: jaeger
jaeger-infra: collector-service
spec:
ports:
- name: jaeger-collector-tchannel
port: 14267
protocol: TCP
targetPort: 14267
- name: jaeger-collector-http
port: 14268
protocol: TCP
targetPort: 14268
- name: jaeger-collector-zipkin
port: 9411
protocol: TCP
targetPort: 9411
selector:
jaeger-infra: collector-pod
type: ClusterIP
- apiVersion: v1
kind: Service
metadata:
name: zipkin
namespace: logging
labels:
app: jaeger
jaeger-infra: zipkin-service
spec:
ports:
- name: jaeger-collector-zipkin
port: 9411
protocol: TCP
targetPort: 9411
selector:
jaeger-infra: collector-pod
type: ClusterIP
6. 测试用例hotrod 与代理jaeger-agent的部署
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: hotrod
namespace: logging
spec:
template:
metadata:
labels:
app: hotrod
spec:
containers:
- image: jaegertracing/example-hotrod:latest
name: hotrod
ports:
- containerPort: 8080
- image: jaegertracing/jaeger-agent
name: jaeger-agent
ports:
- containerPort: 5775
protocol: UDP
- containerPort: 5778
- containerPort: 6831
protocol: UDP
- containerPort: 6832
protocol: UDP
command:
- "/go/bin/agent-linux"
- "--collector.host-port=jaeger-collector.logging:14267"
---
apiVersion: v1
kind: Service
metadata:
namespace: logging
labels:
app: hotrod
name: hotrod
spec:
ports:
- port: 8080
targetPort: 8080
selector:
app: hotrod
---
apiVersion: v1
kind: Service
metadata:
namespace: logging
labels:
app: hotrod
name: jaeger-agent
spec:
ports:
- port: 6831
targetPort: 6831
protocol: UDP
selector:
app: hotrod
7.配置ingress 来访问服务,也可以使用NodePort的方式来访问
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kibana-ingress
namespace: logging
annotations:
kubernets.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- kibana.xinchanedu.com
secretName: kibana-secret
- hosts:
- jaeger.xinchanedu.com
secretName: kibana-secret
- hosts:
- hotrod.xinchanedu.com
secretName: kibana-secret
rules:
- host: kibana.xinchanedu.com
http:
paths:
- path: /
backend:
serviceName: kibana
servicePort: 5601
- host: jaeger.xinchanedu.com
http:
paths:
- path: /
backend:
serviceName: jaeger-query
servicePort: 80
- host: hotrod.xinchanedu.com
http:
paths:
- path: /
backend:
serviceName: hotrod
servicePort: 8080
使用docker启动jaeger 测试环境 所有数据都是存储到内存中
docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14268:14268 -p 9411:9411 jaegertracing/all-in-one:latest
Docker部署jaeger并使用elasticsearch作为存储引擎
若你安装的collector和elasticsearch是在同一台机器上,使用docker容易的--link命令就可以将collector和elasticsearch关联上,安装命令如下:
docker run -d --name jaeger-collector --restart=always --link elasticsearch -e SPAN_STORAGE_TYPE=elasticsearch -e ES_SERVER_URLS=http://172.26.155.215:9200 -e ES_USERNAME=elastic -p 14267:14267 -p 14268:14268 -p 9411:9411 jaegertracing/jaeger-collector
注意:
--link elasticsearch,代表docker 关联,该名字必须和你安装elasticsearch —name的名字相同
--SPAN_STORAGE_TYPE=elasticsearch 代表安装jaeger选择elasticsearch作为存储
-e ES_SERVER_URLS=http://elasticsearch:9200次条目代表你选择容器安装的elasticsearch的9200端口
-e ES_USERNAME elasticsearch的用户名:默认elastic,下同
-e ES_PASSWORD elasticsearch的密码 没有设置密码不用填
-e 其实就是代表的环境变量,其他变量你可以使用以下语句查看:
当然,一般生产环境你肯定不会将collector和elasticsearch安装到同一台机器,至少你可能会安装多个collector,所以,如何跨机器的用collector连接此elasticsearch呢?
你可以用用以下命令:
docker run -d --name jaeger-collector --restart=always -e SPAN_STORAGE_TYPE=elasticsearch -e ES_SERVER_URLS=http://es部署机器的ip:9200 -e ES_USERNAME=elastic -p 14267:14267 -p 14268:14268 -p 9411:9411 jaegertracing/jaeger-collector
区别在于,你无需使用—link来进行容器互连,只需ES_SERVER_URLS填写对应的ip和port即可;
如出现以下错误:
"caller":"collector/main.go:102","msg":"Failed to init storage factory","error":"health check timeout: no Elasticsearch node available","errorVerbose":"no Elasticsearch node available
请检查elasticsearch地址是否正确
docker + query安装
同collector一样,若你安装的collector和elasticsearch是在同一台机器上,使用docker容易的–link命令就可以将query和elasticsearch关联上,安装命令如下:
docker run -d --name jaeger-query --restart=always --link elasticsearch -e SPAN_STORAGE_TYPE=elasticsearch -e ES_SERVER_URLS=http://172.26.155.215:9200 -p 16686:16686/tcp jaegertracing/jaeger-query
注意,ES_USERNAME、ES_PASSWORD这两个环境变量,当你的elasticsearch未设置账号密码时,你可以不填,也可以填上默认值,elasticsearch的默认ES_USERNAME=elastic,ES_PASSWORD=changeme
部署完成query之后,根据你暴露的端口号(-p 16686:16686/tcp),浏览器输入以下地址(将localhost换成你部署query的地址):
http://localhost:16686
注意:如果报elasticsearch以下错误
jaeger-query HTTP Error: search service failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]
删除之前的ja索引后重新收集即可
curl -XDELETE 'http://172.26.155.215:9200/ja*'
docker + agent安装
根据uber jaeger官网的架构,agent一般是和jaeger-client部署在一起,agent作为一个基础架构,每一台应用(接入jaeger-client的应用)所在的机器都需要部署一个agent;
根据数据采集原理,jaeger-client采集到数据之后,是通过UDP端口发送到agent的,jaeger-client和agent部署在一起的好处是UDP传输数据都在应用所在的机器,可避免UDP的跨网络传输,多一层安全保障。
当然,架构可能是多变的,你的agent可能不和jaeger-client所在的应用在一台机器,这个时候,jaeger-client就必须显示的指定其连接的agent的IP及port,具体做法后文jaeger-client对应模块会讲到。
前文提到,jaeger-client采集到数据之后,是通过UDP端口发送到agent的,agent接收到数据之后,使用Uber的Tchannel协议,将数据发送到collector,所以,agent是必须和collector相连的;
docker run -d --name jaeger-agent --restart=always -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778/tcp jaegertracing/jaeger-agent --reporter.grpc.host-port=172.26.155.215:14267
如前文所述,你可能不止一个collector,你可能需要这样:
docker run -d --name jaeger-agent --restart=always -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778/tcp jaegertracing/jaeger-agent --reporter.grpc.host-port=jaeger-collector-id1:14267,jaeger-collector-ip2:14267,jaeger-collector-ip3:14267
//--collector.host-port=collector jaeger-collector-id1:14267,jaeger-collector-ip2:14267,jaeger-collector-ip3:14267,用逗号分开,连接三个collector,这样的话,这三个collector只要一个存活,agent就可以吧数据传输完成,以避免单点故障