搭建Hive On Spark 编译Hive源码错误解决方法（spark2.1.0，hadoop2.7.2）

最新推荐文章于 2024-05-15 23:19:27 发布
最新推荐文章于 2024-05-15 23:19:27 发布
阅读量2.7k 收藏
点赞数
分类专栏： Spark Hive
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/dax1n/article/details/58594789
版权
Spark 同时被 2 个专栏收录
70 篇文章 0 订阅
订阅专栏
Hive
15 篇文章 2 订阅
订阅专栏

Hive On Spark 的搭建需要自行编译Spark 源码  without Hive 然后部署。

编译成功之后部署Spark集群，此处不描述如何部署集群了。

然后本人自行从Hive官网下载Hive2.1.1安装包进行安装，之后启动运行Hive发现出现Class不兼容异常，经过Google得知版本冲突，导致自行编译Hive源码。


本人第一次是在archive.apache.org官方下载hive源码进行编译得到如下错误：（由于Spark升级到2.x版本，移除了JavaSparkListener类）

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project spark-client: Compilation failure: Compilation failure:

[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java:[46,24] cannot find symbol
[ERROR] symbol:   class JavaSparkListener
[ERROR] location: package org.apache.spark
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java:[444,40] cannot find symbol
[ERROR] symbol:   class JavaSparkListener
[ERROR] location: class org.apache.hive.spark.client.RemoteDriver
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java:[46,24] cannot find symbol
[ERROR] symbol:   class JavaSparkListener
[ERROR] location: package org.apache.spark
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java:[444,40] cannot find symbol
[ERROR] symbol:   class JavaSparkListener
[ERROR] location: class org.apache.hive.spark.client.RemoteDriver
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java:[63,38] cannot find symbol
[ERROR] symbol:   method get()
[ERROR] location: class org.apache.spark.executor.ShuffleReadMetrics
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java:[64,35] cannot find symbol
[ERROR] symbol:   method get()
[ERROR] location: class org.apache.spark.executor.ShuffleReadMetrics
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java:[65,35] cannot find symbol
[ERROR] symbol:   method get()
[ERROR] location: class org.apache.spark.executor.ShuffleReadMetrics
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java:[66,35] cannot find symbol
[ERROR] symbol:   method get()
[ERROR] location: class org.apache.spark.executor.ShuffleReadMetrics
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java:[50,39] cannot find symbol
[ERROR] symbol:   method get()
[ERROR] location: class org.apache.spark.executor.ShuffleWriteMetrics
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java:[51,36] cannot find symbol
[ERROR] symbol:   method get()
[ERROR] location: class org.apache.spark.executor.ShuffleWriteMetrics
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java:[102,34] cannot find symbol
[ERROR] symbol:   method isDefined()
[ERROR] location: class org.apache.spark.executor.InputMetrics
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java:[106,40] cannot find symbol
[ERROR] symbol:   method isDefined()
[ERROR] location: class org.apache.spark.executor.ShuffleReadMetrics
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java:[110,41] cannot find symbol
[ERROR] symbol:   method isDefined()
[ERROR] location: class org.apache.spark.executor.ShuffleWriteMetrics
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java:[48,55] cannot find symbol
[ERROR] symbol:   method get()
[ERROR] location: class org.apache.spark.executor.InputMetrics
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java:[49,29] cannot find symbol
[ERROR] symbol:   method get()
[ERROR] location: class org.apache.spark.executor.InputMetrics
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java:[158,14] method addSparkListener in class org.apache.spark.SparkContext cannot be applied to given types;
[ERROR] required: org.apache.spark.scheduler.SparkListenerInterface
[ERROR] found: org.apache.hive.spark.client.RemoteDriver.ClientListener
[ERROR] reason: actual argument org.apache.hive.spark.client.RemoteDriver.ClientListener cannot be converted to org.apache.spark.scheduler.SparkListenerInterface by method invocation conversion
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java:[448,5] method does not override or implement a method from a supertype
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java:[457,5] method does not override or implement a method from a supertype
[ERROR] /home/daxin/installData/source/apache-hive-2.1.1-src/spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java:[475,5] method does not override or implement a method from a supertype
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :spark-client



由于Spark升级导致接口和类的不兼容，后来去Hive的github上去查看Hive源码发现，Github上已经更新此错误，最后在Github上clone一份代码进行本地编译。

具体编译过程查看Hive官方文档即可，官方文档如下：

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

  $ git clone https://git-wip-us.apache.org/repos/asf/hive.git
  $ cd hive
  $ mvn clean package -Pdist
  $ cd packaging/target/apache-hive-{version}-SNAPSHOT-bin/apache-hive-{version}-SNAPSHOT-bin
  $ ls
  LICENSE
  NOTICE
  README.txt
  RELEASE_NOTES.txt
  bin/ (all the shell scripts)
  lib/ (required jar files)
  conf/ (configuration files)
  examples/ (sample input and query files)
  hcatalog / (hcatalog installation)
  scripts / (upgrade scripts for hive-metastore)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


最后输出目录：

packaging/target/apache-hive-{version}-SNAPSHOT-bin/apache-hive-{version}-SNAPSHOT-bin


例如我的输出目录：
/home/daxin/installData/source/hive-2.1.1/packaging/target/apache-hive-2.2.0-SNAPSHOT-bin/apache-hive-2.2.0-SNAPSHOT-bin


备注：编译过程中测试遇到问题，可以直接跳过maven测试：

mvn clean package -Pdist -Dmaven.test.skip=true



如果想指定Spark、Hadoop版本的话，在父目录的pom.xml修改依赖版本即可。，修改JDK,Scala版本同理。



经验：以后下载源码还是去github上下载为好，那里代码更新快，少走弯路。



-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
javartisan
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
hive源碼編譯(失敗記錄)
微电子学与固体电子学-俞驰
12-20 662
编译hive 源码的命令: mvn clean install -T 4C-DskinpTests出现如下报错: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project metastore-tools-common: Execution default-compile of goal org.apache.maven..
org.spark-project.hive 源码
Ms_WangH
11-12 726
源码下载：hive-release-1.2.1-spark2 其他：酷码派 - KuMaPai.com 开发资源分享下载平台： Maven仓库 > org.spark-project.hive
HiveOnSpark 报错：java.lang.IllegalStateException(Connection to remote Spark driver was lost)‘ Last kno
最新发布
JIE的博客 --- moon_coder
05-15 395
这里拷贝完成了还不行，因为我们在部署 Hive On Spark 时，通常会上传一份 Spark 的。包，因为该包属于 Hive，所以需要从 Hive 中拷贝一份到 Spark 中。我在 HDFS 上存储 Spark jars 目录的路径是。修改完成后，重启 Hive 元数据服务，再次测试，问题解决。刚刚搭建完 Hive On Spark，测试时出现该错误。目录到 HDFS 上，故该包同样需要上传。，请注意将其修改成你自己的。在 Spark 中找不到。
hive on spark 编译时遇到的问题
anjieqing1102的博客
12-31 268
1.官方网站下载spark 1.5.0的源码 2.根据官方编译即可。 export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -DskipTests clean package ...
java spark on hive_Hive on Spark编程入门指南
weixin_42100188的博客
02-12 452
先说明一下，这里说的从Hive 1.1版本开始，Hive on Spark已经成为Hive代码的一部分了，并且在spark分支上面，可以看这里https://github.com/apache/hive/tree/spark，并会定期的移到master分支上面去。关于Hive on Spark的讨论和进度，可以看这里https://issues.apache.org/jira/browse/HIV...
spark结合hive使用
yunmeng879的博客
03-18 167
idea:spark结合hive使用 Exception in thread "main" java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder': at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionStat
基于CDH 6.3.0 搭建 Hive on Spark 及相关配置和调优
05-25
CDH6针对hive on spark的调优文档，这个是生产的实战经验
Hive3.1.2编译源码
05-20
使用hive3.1.2和spark3.0.0配置hive on spark的时候，发现官方下载的hive3.1.2和spark3.0.0不兼容，hive3.1.2对应的版本是spark2.3.0，而spark3.0.0对应的hadoop版本是hadoop2.6或hadoop2.7。 所以，如果想要使用高...
hive on spark mr 数据开发常见问题解决
11-20
一种解决方法是增加Hiveserver2中的`hive.spark.client.server.connect.timeout`参数，将其设为更长的时间，如5分钟，以允许更长的等待时间。 4. **Java heap空间溢出** `java.lang.OutOfMemoryError: Java heap ...
大数据Hadoop+HBase+Spark+Hive集群搭建教程(七月在线)1
08-03
在构建大数据处理环境时，Hadoop、HBase、Spark和Hive是四个核心组件，它们协同工作以实现高效的数据存储、处理和分析。本教程将详细介绍如何在Ubuntu系统上搭建这些组件的集群。 1. **Hadoop**：Hadoop是Apache...
Hive on Spark源码分析DOC
12-18
Hive on Spark源码分析 Hive on Spark 源码分析是指将 Hive 默认的执行...通过对 Hive on Spark 的源码分析，我们可以更好地理解 Hive on Spark 的运行机理和实现原理，从而更好地应用 Hive on Spark 解决实际问题。
hive on spark hql 插入数据报错 Failed to create Spark client for Spark session Error code 30041
想练武，就得下功夫
07-29 6813
Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 50cec71c-2636-4d99-8de2-a580ae3f1c58)' FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.
hive on spark 利用maven重新编译spark
lukabruce的博客
12-21 1532
缘由：使用hive on spark 进行hivesql操作的时候报以下错误： Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)' FAILED: Execution Error, retur...
hive on spark 执行sql报错
cclovezbf的博客
09-20 3151
报错信息ERROR : FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed due to: Job 3 cancelled。这个是因为我们之前设置的excutor.memory=6G，其中有一些reseverd啥的。这个是啥，看过我其他文章的我都会提到这个，这个vector叫矢量化，也就是看你开启矢量化。如下图 这个文件格式如下 是真的垃圾。
Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin
L
07-06 4022
idea编译项目时，出现如下报错：[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.0.0:check (validate) on project devops-brain: Failed during checkstyle configuration: Cannot set property 'cacheFile' to 'F:\Copote\Code\devops-brain\devops-
spark on yarn 报错日志分析
coffeecolor的博客
07-22 493
日志： Application application_1589026118201_459822 failed 1 times due to AM Container for appattempt_1589026118201_459822_000001 exited with exitCode: 15 For more detailed output, check application tracking page:http://hadoop02:8088/proxy/application_158902
Hive3.1.2编译
docsz的博客
05-24 1661
1、修改pom文件 <spark.version>3.1.1</spark.version> <scala.binary.version>2.12</scala.binary.version> <scala.version>2.12.10</scala.version> <hadoop.version>3.2.2</hadoop.version> <guava.version&g
配置hive on spark后启动报错
newhandnew413的博客
08-14 1735
因为hive底层用的执行引擎是MapReduce，所以导致了运行起来非常慢，所以我们要更换执行引擎，就有了hive on spark，基于hive的基础上将执行引擎换成spark 版本：hive 3.1.2 spark3.0.0 需要配置的信息 在spark的配置文件spark-defaults.conf中 spark.master yarn spark.eventLog.enabled true spark.e
Mark ：Hive使用Spark on Yarn作为执行引擎
tianyeshiye
05-07 7271
原文：http://lxw1234.com/archives/2016/05/673.htmHive从1.1之后，支持使用Spark作为执行引擎，配置使用Spark On Yarn作为Hive的执行引擎，首先需要注意以下两个问题：Hive的版本和Spark的版本要匹配；具体来说，你使用的Hive版本编译时候用的哪个版本的Spark，那么就需要使用相同版本的Spark，可以在Hive的pom.xml...
大数据集群实战：Hadoop-HBase-Spark-Hive搭建步骤
"该资源是一个关于大数据处理技术的教程，主要涵盖了Hadoop、HBase、Spark和Hive的集群搭建过程。由七月在线提供，旨在帮助用户了解和实践大数据环境的构建。" 在这个教程中，你将学习如何一步步构建一个高效运行...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值