htrace-zipkin与hadoop集成

最新推荐文章于 2022-10-05 22:27:06 发布
最新推荐文章于 2022-10-05 22:27:06 发布
阅读量4.1k 收藏
点赞数
分类专栏： Hadoop 文章标签： hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/yunlong34574/article/details/76396335
版权

这里坑更多

hadoop zipkin配置

hadoop的zipkin配置部分的文章就是shit呀，包括官方文档。
我这里使用的是

hadoop 2.7.1

https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/Tracing.html
来看看官方文档，照着配置，绝对不能成功，呵呵

为什么呢，咱们来看看这个jira

DFSClient should use hdfs.client.htrace HTrace configuration prefix rather than hadoop.htrace
https://issues.apache.org/jira/browse/HDFS-8213

要不要这么搞笑。

幸好stackoverflow上有个人说

Exactly, in the code, I see the configuration key's prefix is dfs.htrace, not the hadoop.htrace. And in dfsclient, it's dfs.client.htrace. You can change the prefix to dfs.htrace, then restart the cluster and it take effect. The code is in class org.apache.hadoop.tracing.SpanReceiverHost. Hope this help!

我觉得世界就是崩溃的呀，花了半天时间，你告诉我一切都是个误会。

<property>
    <name>hdfs.client.htrace.sampler</name>
    <value>AlwaysSampler</value>
  </property>
<property>
    <name>hdfs.client.htrace.spanreceiver.classes</name>
    <value>ZipkinSpanReceiver</value>
  </property>
  <property>
    <name>hdfs.client.htrace.zipkin.collector-hostname</name>
    <value>localhost</value>
  </property>
  <property>
    <name>hdfs.client.htrace.zipkin.collector-port</name>
    <value>9410</value>
  </property>

  <property>
    <name>dfs.htrace.sampler</name>
    <value>AlwaysSampler</value>
  </property>
<property>
    <name>dfs.htrace.spanreceiver.classes</name>
    <value>ZipkinSpanReceiver</value>
  </property>
  <property>
    <name>dfs.htrace.zipkin.collector-hostname</name>
    <value>localhost</value>
  </property>
  <property>
    <name>dfs.htrace.zipkin.collector-port</name>
    <value>9410</value>
  </property>

  <property>
    <name>dfs.client.htrace.sampler</name>
    <value>AlwaysSampler</value>
  </property>
<property>
    <name>dfs.client.htrace.spanreceiver.classes</name>
    <value>ZipkinSpanReceiver</value>
  </property>
  <property>
    <name>dfs.client.htrace.zipkin.collector-hostname</name>
    <value>localhost</value>
  </property>
  <property>
    <name>dfs.client.htrace.zipkin.collector-port</name>
    <value>9410</value>
  </property>

我采取的配置是十分激进的，把这些都配置上吧。没有具体检查是哪个，看看源码就知道了。先这样的，已经浪费很多时间了。

还有个很有意思的命令可以记忆一下

hdfs dfs -Dfs.shell.htrace.span.receiver.classes=org.apache.htrace.impl.ZipkinSpanReceiver \
           -Dfs.shell.htrace.sampler.classes=AlwaysSampler \
           -Dhadoop.htrace.zipkin.collector-hostname=localhost \
           -Dhadoop.htrace.zipkin.collector-port=9410 \
           -ls /
$ javac -cp `hadoop classpath` TracingFsShell.java
$ HADOOP_CLASSPATH=. hdfs TracingFsShell -put sample.txt /tmp/

这里的执行方法也很好，我之前都是自己统计classpath，太二了。

hadoop trace 测试

所谓出师未捷身先死

-sh-4.1$ hdfs dfs -ls /
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/thrift/protocol/TProtocolFactory
    at java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
    at java.lang.Class.getConstructor0(Class.java:3075)
    at java.lang.Class.getConstructor(Class.java:1825)
    at org.apache.htrace.SpanReceiverBuilder.build(SpanReceiverBuilder.java:107)
    at org.apache.hadoop.tracing.SpanReceiverHost.loadInstance(SpanReceiverHost.java:169)
    at org.apache.hadoop.tracing.SpanReceiverHost.loadSpanReceivers(SpanReceiverHost.java:154)
    at org.apache.hadoop.tracing.SpanReceiverHost.get(SpanReceiverHost.java:78)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:634)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:150)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:355)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
    at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
    at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)
    at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)
    at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
    at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
    at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
    at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
Caused by: java.lang.ClassNotFoundException: org.apache.thrift.protocol.TProtocolFactory
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 28 more
-sh-4.1$ ll

嗯，这个时候我觉得又可以看看官方文档了

$ git clone https://github.com/cloudera/htrace
  $ cd htrace/htrace-zipkin
  $ mvn compile assembly:single
  $ cp target/htrace-zipkin-*-jar-with-dependencies.jar $HADOOP_HOME/share/hadoop/hdfs/lib/

这是官方的介绍，需要一个fat的jar包，那么就干吧。打个包而已。我之前上传的是thin包，不包括依赖

htrace-zipkin编译打包

打包这个事情做过好多次了
首先呢，下面的命令是执行不了的，因为我下载的代码根本在注释里面说了，skip assembly，所以要修改pom.xml

mvn compile assembly:single
  1. 只是删除或者注释掉skip assembly那一段，没有效果，然后一直报错说
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.5.3:single (default) on project htrace-zipkin: Error reading assemblies: Error locating assembly descriptor: src/main/assembly/src.xml

这个嘛，既然你找不到，那我就给你一个

<?xml version="1.0" encoding="utf-8"?>
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor
license agreements. See the NOTICE file distributed with this work for additional
information regarding copyright ownership. The ASF licenses this file to
You under the Apache License, Version 2.0 (the "License"); you may not use
this file except in compliance with the License. You may obtain a copy of
the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required
by applicable law or agreed to in writing, software distributed under the
License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
OF ANY KIND, either express or implied. See the License for the specific
language governing permissions and limitations under the License. -->
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
  <id>jar-with-dependencies</id>
  <formats>
    <format>jar</format>
  </formats>
  <includeBaseDirectory>false</includeBaseDirectory>
  <dependencySets>
    <dependencySet>
      <unpack>true</unpack>
      <scope>runtime</scope>
    </dependencySet>
  </dependencySets>
  <fileSets>
    <fileSet>
      <directory>${project.build.outputDirectory}</directory>
    </fileSet>
  </fileSets>
</assembly>

这样还是不行的

[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (default) on project htrace-zipkin: Too many files with unapproved license: 1 See RAT report in: /Users/xxx/code/incubator-htrace/htrace-zipkin/target/rat.txt -> [Help 1]

这是个什么鬼呀，看了下报错信息

1 Unknown Licenses

*******************************

Unapproved licenses:

  src/main/assembly/src.xml

*******************************

原来是我自己写的xml不支持apache license，所以不给编译。这么高级。。。。
好吧，我就给你个license

<?xml version="1.0" encoding="utf-8"?>
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor
license agreements. See the NOTICE file distributed with this work for additional
information regarding copyright ownership. The ASF licenses this file to
You under the Apache License, Version 2.0 (the "License"); you may not use
this file except in compliance with the License. You may obtain a copy of
the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required
by applicable law or agreed to in writing, software distributed under the
License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
OF ANY KIND, either express or implied. See the License for the specific
language governing permissions and limitations under the License. -->
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http
://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
  <id>jar-with-dependencies</id>
  <formats>
    <format>jar</format>
  </formats>
  <includeBaseDirectory>false</includeBaseDirectory>
  <dependencySets>
    <dependencySet>
      <unpack>true</unpack>
      <scope>runtime</scope>
    </dependencySet>
  </dependencySets>
  <fileSets>
    <fileSet>
      <directory>${project.build.outputDirectory}</directory>
    </fileSet>
  </fileSets>
</assembly>

从随便一个pom里面拷贝了一份apache license声明放在文件头上，就可以了。

另外这个过程还出现了java doc的编译问题，没有解决，只是单纯的删除了pom中关于java doc的plugin部分跳过。

最终编译成功。

苏然Vincent
关注
专栏目录
htrace-zipkin-3.0.jar
05-31
HTrace Zipkin org.htrace/htrace-zipkin/3.0/htrace-zipkin-3.0.jar
java使用hbase、hadoop报错举例
刘大猫
06-03 1655
摘要 总结自己springboot项目使用hbase、hadoop中出现的莫名奇怪的依赖使用报错 hbase 2.0.2版本 hadoop 3.1.1版本 注意：本地依赖版本最好和服务器上版本一致，避免发生莫名奇怪错误 情况1：NoClassDefFoundError java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHost at org.apache.hadoop.hdfs.DFSClient.<.
「zeppelin」： java.lang.NoSuchMethodError: org.apache.hadoop.tracing.SpanReceiverHost.get(Lorg/a
图特摩斯科技-博客
06-27 5819
运行zeppelin 时遇到问题：  java.lang.NoSuchMethodError: org.apache.hadoop.tracing.SpanReceiverHost.get(Lorg/apache/hadoop/conf/  说明： 版本不匹配导致 解决：  cd zeppelin-0.7.2-bin-all/lib
maven 解决依赖冲突
weixin_49761581的博客
01-18 2591
依赖冲突的表现 ClassNotFound // class 名字会随着冲突的不同而不同 Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tracing.SpanReceiverHost at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) a
Too many files with unapproved license: 2 See RAT report
Gblfy_Blog
04-07 1167
解决方案 mvn -Prelease-nacos -Dmaven.test.skip=true -Dpmd.skip=true -Dcheckstyle.skip=true -Drat.numUnapprovedLicenses=100 clean install -U 或者 mvn -Prelease-nacos -Dmaven.test.skip=true -Drat.numUnapprovedLicenses=100 clean install -U
明明配置了CLASSPATH，但是依然提示java.lang.ClassNotFoundException错误的一种可能
天烬的专栏
05-20 1886
背景 有一个C++项目会调用Hadoop，该项目在部署时会利用脚本来配置所需要的CLASSPATH和其他环境变量。 在启动时发现报如下错误： Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration Caused by: java.lang.ClassNotFoun...
htrace-zipkin-3.0.4.jar
05-31
HTrace Zipkin org.htrace/htrace-zipkin/3.0.4/htrace-zipkin-3.0.4.jar
htrace-zipkin-3.0.3.jar
05-31
HTrace Zipkin org.htrace/htrace-zipkin/3.0.3/htrace-zipkin-3.0.3.jar
最新版-zipkin-server-2.23.2-exec.jar.rar
09-02
在给定的文件“最新版-zipkin-server-2.23.2-exec.jar.rar”中，包含的是Zipkin Server的执行版本2.23.2。这个版本的Zipkin Server是一个可执行的JAR文件，用于运行和管理Zipkin服务。 Zipkin的核心功能包括： 1. *...
egg-opentracing-zipkin:向Egg.js中的zipkin服务器报告跟踪信息
05-09
鸡蛋开Kong拉链 向Egg.js中的zipkin服务器报告跟踪信息。安装$ npm i egg-opentracing-zipkin --save用法egg-opentracing-zipkin取决于 ，您应该启用这些插件。 // {app_root}/config/plugin.jsexports . open...
分布式跟踪系统Zipkin.zip
07-16
Zipkin 是 Twitter 的一个开源项目，允许开发者收集 Twitter 各个服务上的监控数据，并提供查询接口。该系统让开发者可通过一个 Web 前端轻松的收集和分析数据，例如用户每次请求服务的处理时间等，可方便的监测系统中存在的瓶颈。Zipkin 使用 Apache Cassandra 做为数据存储系统，使用 Apache ZooKeeper – the Hadoop configuration management software – for coordination, and Facebook's Scribe data aggregation system as the logging framework to transport the trace data. 标签：Zipkin
[hive-4.0 ] Too many files with unapproved license: 2 See RAT report in: target/rat.txt
houzhizhen的专栏
08-30 588
用以下命令编译 hive-4.0（和之前的 hive 版本同样的命令）输出以下错误信息。
Hive On LLAP搭建&常见问题
pengpengsays
01-24 1186
Hive On LLAP搭建&常见问题 基本概述 Hive在2.0之后，推出一个新特性LLAP（Live Long And Process），可以显著提高查询效率。 LLAP是一个常驻于Yarn的进程，并不是一个执行引擎，它将DataNode数据预先缓存到内存中，然后交由DAG引擎进行查询、处理任务使用。部分查询、权限控制将由LLAP执行，短查询任务的结果会很快的返回。 相对于Hive 1.x，提升大约25倍的性能。 安装准备 Apache Slider安装 LLAP需要常驻于Yarn之上，所以需
Maven Error: Too many files with unapproved license
Blossom
05-15 5569
错误信息 在使用如下maven命令编译Flink源码时， mvn clean install -DskipTests 出现错误： [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.12:check (default) on project flink-parent: Too many files with...
sparkSQL连接hive失败案例和解决办法
最新发布
qq_46183404的博客
10-05 3064
sparkSQL连接hive失败案例和解决办法
Maven搭建hadoop环境之pom.xml基本配置
唤不醒的梦
10-29 780
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.
Too many files with unapproved license
xzh_blog
07-01 4585
Failed to execute goal org.apache.rat:apache-rat-plugin:0.12:check (default) on project nacos-console: Too many files with unapproved license: 1 See RAT report in: D:\program\IDEA_workSpace\alibaba\Nacos\console\target\rat.txt -> [Help 1]
HBase链路监控：HTrace+Zipkin
寂寞的博客
07-06 963
HTrace是曾经apache的一个孵化项目，在18年孵化失败，停止开发了：https://incubator.apache.org/projects/htrace.html。但在早期HBase和Hadoop版本中使用了HTrace，可以试试。在hbase的lib下已经自带了htrace的依赖： 启用trace功能...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值