微服务链路监控-pinpoint 实践案例

      链路的跟踪系统的目的是：记录微服务组件的状态，为了快速的定位一个问题 ，排查问题，快速的定位问题，提升用户的体验！全链路监控的系统比较多，主流的是 zipkin ,skywalking ,pinpoint ;链路监控需要考虑的问题:

1：全链路监控系统的探针性能消耗

2：必须和开发确认代码的侵入性

3：监控维度，分析的颗粒度越小越好,能暴露多更多的指标出来

4:扩展性

一: 部署 pinpoint    # git clone https://github.com/naver/pinpoint-docker.git # cd pinpoint-docker [root@node-16 software]# docker-compose pull                     Pulling zoo1        ... done Pulling zoo2        ... done Pulling zoo3        ... done Pulling jobmanager  ... done Pulling taskmanager ... Done   # docker-compose up -d Creating network "pinpoint-docker-185_pinpoint" with driver "bridge" Creating volume "pinpoint-docker-185_data-volume" with default driver Creating volume "pinpoint-docker-185_mysql_data" with default driver Building pinpoint-hbase Step 1/12 : FROM java:8-jdk 8-jdk: Pulling from library/java 5040bd298390: Pull complete fce5728aad85: Pull complete 76610ec20bf5: Pull complete 60170fec2151: Pull complete e98f73de8f0d: Pull complete 11f7af24ed9c: Pull complete 49e2d6393f32: Pull complete bb9cdec9c7f3: Pull complete Digest: sha256:c1ff613e8ba25833d2e1940da0940c3824f03f802c449f3d1815a66b7f8c0e9d Status: Downloaded newer image for java:8-jdk  ---> d23bdf5b1b1b Step 2/12 : LABEL maintainer="Roy Kim <roy.kim@navercorp.com>"  ---> Running in 96fcfdf4a983 Removing intermediate container 96fcfdf4a983  ---> 6472c81218d4 Step 3/12 : ARG PINPOINT_VERSION=${PINPOINT_VERSION:-1.8.5}  ---> Running in dd04b7deb84e Removing intermediate container dd04b7deb84e  ---> b677f1a91218 Step 4/12 : ENV HBASE_REPOSITORY=http://apache.mirrors.pair.com/hbase  ---> Running in 2c75658a181b Removing intermediate container 2c75658a181b  ---> 29c4ddfe8d01 Step 5/12 : ENV HBASE_SUB_REPOSITORY=http://archive.apache.org/dist/hbase  ---> Running in 8dc5c970d913 Removing intermediate container 8dc5c970d913 inflating: /usr/local/tomcat/webapps/ROOT/WEB-INF/web.xml    inflating: /usr/local/tomcat/webapps/ROOT/WEB-INF/views/apis.jsp    inflating: /usr/local/tomcat/webapps/ROOT/META-INF/maven/com.navercorp.pinpoint/pinpoint-quickstart-testapp/pom.xml    inflating: /usr/local/tomcat/webapps/ROOT/META-INF/maven/com.navercorp.pinpoint/pinpoint-quickstart-testapp/pom.properties  Removing intermediate container f9a86ba61de4  ---> fafc6b47cd2a Step 5/5 : CMD [ "catalina.sh", "run" ]  ---> Running in 5f508cbd699f Removing intermediate container 5f508cbd699f  ---> 5e4ebc185be5   Successfully built 5e4ebc185be5 Successfully tagged pinpointdocker/pinpoint-quickstart:latest WARNING: Image for service pinpoint-quickstart was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`. Creating pinpoint-docker-185_zoo3_1 ... done Creating pinpoint-docker-185_zoo1_1 ... done Creating pinpoint-flink-jobmanager  ... done Creating pinpoint-docker-185_zoo2_1 ... done Creating pinpoint-hbase             ... done Creating pinpoint-mysql             ... done Creating pinpoint-flink-taskmanager ... done Creating pinpoint-web               ... done Creating pinpoint-collector         ... done Creating pinpoint-agent             ... done Creating pinpoint-quickstart        ... done # # docker-compose ps

二: Pinpoint Agent 部署 部署完服务端后，就要把微服务组件接入进来！   Agent 接入到微服务的案例:   前端服务接入到pinpoint 的agent :   # pwd xxxx/portal-service # cat Dockerfile FROM lizhenliang/java:8-jdk-alpine LABEL maintainer www.ctnrs.com ENV JAVA_ARGS="-Dfile.encoding=UTF8 -Duser.timezone=GMT+08" COPY ./target/portal-service.jar ./ COPY pinpoint /pinpoint EXPOSE 8080 CMD java -jar -javaagent:/pinpoint/pinpoint-bootstrap-1.8.3.jar -Dpinpoint.agentId=$(echo $HOSTNAME | awk -F- '{print "portal-"$NF}') -Dpinpoint.applicationName=ms-portal $JAVA_ARGS $JAVA_OPTS /portal-service.jar # 修改pinpoint 配置文件 # pinpoint]# ls pinpoint.config pinpoint.config pinpoint]# # cat pinpoint.config |more # # Pinpoint agent configuration # ########################################################### # Collector server                                        #  ########################################################### profiler.collector.ip=xxxxxxxxxxxxxxxxxxxxxxxxxxxx   只要修改IP 就行，，其它按默认的就可以了！   其它微服务组件方法是一样的，在Dockerfile 中增加调用pinpoint 的语句：

重新打包上传镜像，和拉取镜像   # ./docker_build.sh   ---------- Successfully built 016b31a83bfb Successfully tagged 10.83.35.11/microservice/portal:2020-06-04-20-18-03 The push refers to repository [10.83.35.11/microservice/portal] 2c34c16db671: Pushed 91f297c06201: Pushed 77628df807a8: Layer already exists a1e7033f082e: Layer already exists 78075328e0da: Layer already exists 9f8566ee5135: Layer already exists 2020-06-04-20-18-03: digest: sha256:8775732b6bfbfef36524c6d9e442b511dc9a9f36ea380657c874f619e362ab5f size: 1582 ingress.extensions/portal unchanged service/portal unchanged deployment.apps/portal configured #

检查pod # kubectl get pod -n ms NAME                       READY   STATUS    RESTARTS   AGE eureka-0                   1/1     Running   0          32h eureka-1                   0/1     Running   0          13s eureka-2                   1/1     Running   0          2m2s gateway-7947cd58c6-gzcjc   1/1     Running   0          2m29s order-85b574596-zkv7v      1/1     Running   0          2m22s portal-7b8dd7c8f4-t5brj    1/1     Running   0          115s product-95fbdf46-tfjs9     1/1     Running   0          2m14s stock-59d96965b6-8xfrz     1/1     Running   0          2m5s #     检查镜像           # kubectl get pods,svc,ing -n ms NAME                           READY   STATUS    RESTARTS   AGE pod/eureka-0                   0/1     Running   0          25s pod/eureka-1                   1/1     Running   0          2m17s pod/eureka-2                   1/1     Running   0          4m6s pod/gateway-7947cd58c6-gzcjc   1/1     Running   0          4m33s pod/order-85b574596-zkv7v      1/1     Running   0          4m26s pod/portal-7b8dd7c8f4-t5brj    1/1     Running   0          3m59s pod/product-95fbdf46-tfjs9     1/1     Running   0          4m18s pod/stock-59d96965b6-8xfrz     1/1     Running   0          4m9s   NAME              TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE service/eureka    ClusterIP   None         <none>        8888/TCP   3d service/gateway   ClusterIP   10.0.0.132   <none>        9999/TCP   32h service/portal    ClusterIP   10.0.0.18    <none>        8080/TCP   32h   NAME                         HOSTS               ADDRESS   PORTS   AGE ingress.extensions/eureka    eureka.ctnrs.com              80      3d ingress.extensions/gateway   gateway.ctnrs.com             80      32h ingress.extensions/portal    portal.ctnrs.com              80      32h #

检查pinpoint   # docker-compose ps            Name                         Command               State                                                             Ports                                                          ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- pinpoint-agent               /usr/local/bin/configure-a ...   Up                                                                                                                               pinpoint-collector           /usr/local/bin/start-colle ...   Up      8080/tcp, 0.0.0.0:9994->9994/tcp, 0.0.0.0:9995->9995/tcp, 0.0.0.0:9995->9995/udp, 0.0.0.0:9996->9996/tcp,                                                                                      0.0.0.0:9996->9996/udp                                                                                                   pinpoint-docker-185_zoo1_1   /docker-entrypoint.sh zkSe ...   Up      2181/tcp, 2888/tcp, 3888/tcp                                                                                             pinpoint-docker-185_zoo2_1   /docker-entrypoint.sh zkSe ...   Up      2181/tcp, 2888/tcp, 3888/tcp                                                                                             pinpoint-docker-185_zoo3_1   /docker-entrypoint.sh zkSe ...   Up      2181/tcp, 2888/tcp, 3888/tcp                                                                                             pinpoint-flink-jobmanager    /docker-entrypoint.sh jobm ...   Up      6123/tcp, 0.0.0.0:8081->8081/tcp                                                                                         pinpoint-flink-taskmanager   /docker-entrypoint.sh task ...   Up      0.0.0.0:19994->19994/tcp, 0.0.0.0:6121->6121/tcp, 0.0.0.0:6122->6122/tcp, 6123/tcp, 8081/tcp                             pinpoint-hbase               /bin/sh -c ${BASE_DIR}/hba ...   Up      0.0.0.0:16010->16010/tcp, 0.0.0.0:16030->16030/tcp, 0.0.0.0:2180->2181/tcp, 0.0.0.0:60000->60000/tcp,                                                                                          0.0.0.0:60020->60020/tcp                                                                                                 pinpoint-mysql               docker-entrypoint.sh mysqld      Up      0.0.0.0:13306->3306/tcp, 33060/tcp                                                                                       pinpoint-quickstart          catalina.sh run                  Up      0.0.0.0:8000->8080/tcp                                                                                                   pinpoint-web                 /usr/local/bin/start-web.sh      Up      0.0.0.0:8079->8080/tcp, 0.0.0.0:9997->9997/tcp                                                                           #   已发现微服务组件

访问微服务，检查Pinpoint的跟踪记录信息