Spark 报错 ： Error: bad symbolic reference. A signature in SparkContext.class refers to term conf

最新推荐文章于 2022-10-13 17:36:59 发布
最新推荐文章于 2022-10-13 17:36:59 发布
阅读量979 收藏
点赞数
分类专栏： spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/zhangshenghang/article/details/99287508
版权

报错如下： 

Error:scalac: Error: bad symbolic reference. A signature in SparkContext.class refers to term conf
in package org.apache.hadoop which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling SparkContext.class.
scala.reflect.internal.Types$TypeError: bad symbolic reference. A signature in SparkContext.class refers to term conf
in package org.apache.hadoop which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling SparkContext.class.

	at scala.reflect.internal.pickling.UnPickler$Scan.toTypeError(UnPickler.scala:847)

	at scala.reflect.internal.pickling.UnPickler$Scan$LazyTypeRef.complete(UnPickler.scala:854)

	at scala.reflect.internal.pickling.UnPickler$Scan$LazyTypeRef.load(UnPickler.scala:863)

	at scala.reflect.internal.Symbols$Symbol.typeParams(Symbols.scala:1489)

	at scala.tools.nsc.transform.SpecializeTypes$$anonfun$scala$tools$nsc$transform$SpecializeTypes$$normalizeMember$1.apply(SpecializeTypes.scala:798)

	at scala.tools.nsc.transform.SpecializeTypes$$anonfun$scala$tools$nsc$transform$SpecializeTypes$$normalizeMember$1.apply(SpecializeTypes.scala:798)

	at scala.reflect.internal.SymbolTable.atPhase(SymbolTable.scala:207)

	at scala.reflect.internal.SymbolTable.beforePhase(SymbolTable.scala:215)

	at scala.tools.nsc.transform.SpecializeTypes.scala$tools$nsc$transform$SpecializeTypes$$normalizeMember(SpecializeTypes.scala:797)

	at scala.tools.nsc.transform.SpecializeTypes$$anonfun$22.apply(SpecializeTypes.scala:751)

	at scala.tools.nsc.transform.SpecializeTypes$$anonfun$22.apply(SpecializeTypes.scala:749)

	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)

	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)

	at scala.collection.immutable.List.foreach(List.scala:318)

	at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)

	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)

	at scala.tools.nsc.transform.SpecializeTypes.specializeClass(SpecializeTypes.scala:749)

	at scala.tools.nsc.transform.SpecializeTypes.transformInfo(SpecializeTypes.scala:1172)

	at scala.tools.nsc.transform.InfoTransform$Phase$$anon$1.transform(InfoTransform.scala:38)

	at scala.reflect.internal.Symbols$Symbol.rawInfo(Symbols.scala:1321)

	at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1241)

	at scala.reflect.internal.Symbols$Symbol.isDerivedValueClass(Symbols.scala:658)

	at scala.reflect.internal.transform.Erasure$ErasureMap.apply(Erasure.scala:131)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer$$anon$1.transform(Erasure.scala:1283)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer$$anon$1.transform(Erasure.scala:1030)

	at scala.reflect.internal.Trees$$anonfun$itransform$1.apply(Trees.scala:1229)

	at scala.reflect.internal.Trees$$anonfun$itransform$1.apply(Trees.scala:1228)

	at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2936)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:34)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:28)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:19)

	at scala.reflect.internal.Trees$class.itransform(Trees.scala:1227)

	at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:13)

	at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:13)

	at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2897)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:48)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer$$anon$1.transform(Erasure.scala:1292)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer$$anon$1.transform(Erasure.scala:1030)

	at scala.reflect.api.Trees$Transformer$$anonfun$transformStats$1.apply(Trees.scala:2927)

	at scala.reflect.api.Trees$Transformer$$anonfun$transformStats$1.apply(Trees.scala:2925)

	at scala.collection.immutable.List.loop$1(List.scala:170)

	at scala.collection.immutable.List.mapConserve(List.scala:186)

	at scala.reflect.api.Trees$Transformer.transformStats(Trees.scala:2925)

	at scala.reflect.internal.Trees$class.itransform(Trees.scala:1238)

	at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:13)

	at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:13)

	at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2897)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:48)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer$$anon$1.transform(Erasure.scala:1292)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer$$anon$1.transform(Erasure.scala:1030)

	at scala.reflect.internal.Trees$$anonfun$itransform$2.apply(Trees.scala:1235)

	at scala.reflect.internal.Trees$$anonfun$itransform$2.apply(Trees.scala:1233)

	at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2936)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:34)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:28)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:19)

	at scala.reflect.internal.Trees$class.itransform(Trees.scala:1232)

	at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:13)

	at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:13)

	at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2897)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:48)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer$$anon$1.transform(Erasure.scala:1288)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer$$anon$1.transform(Erasure.scala:1030)

	at scala.reflect.api.Trees$Transformer$$anonfun$transformStats$1.apply(Trees.scala:2927)

	at scala.reflect.api.Trees$Transformer$$anonfun$transformStats$1.apply(Trees.scala:2925)

	at scala.collection.immutable.List.loop$1(List.scala:170)

	at scala.collection.immutable.List.mapConserve(List.scala:186)

	at scala.reflect.api.Trees$Transformer.transformStats(Trees.scala:2925)

	at scala.reflect.internal.Trees$class.itransform(Trees.scala:1276)

	at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:13)

	at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:13)

	at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2897)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.scala$tools$nsc$transform$TypingTransformers$TypingTransformer$$super$transform(TypingTransformers.scala:44)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer$$anonfun$transform$1.apply(TypingTransformers.scala:44)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer$$anonfun$transform$1.apply(TypingTransformers.scala:44)

	at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2936)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:34)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:44)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer$$anon$1.transform(Erasure.scala:1292)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer$$anon$1.transform(Erasure.scala:1030)

	at scala.reflect.api.Trees$Transformer.transformTemplate(Trees.scala:2904)

	at scala.reflect.internal.Trees$$anonfun$itransform$4.apply(Trees.scala:1280)

	at scala.reflect.internal.Trees$$anonfun$itransform$4.apply(Trees.scala:1279)

	at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2936)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:34)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:28)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:19)

	at scala.reflect.internal.Trees$class.itransform(Trees.scala:1278)

	at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:13)

	at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:13)

	at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2897)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:48)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer$$anon$1.transform(Erasure.scala:1292)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer$$anon$1.transform(Erasure.scala:1030)

	at scala.reflect.api.Trees$Transformer$$anonfun$transformStats$1.apply(Trees.scala:2927)

	at scala.reflect.api.Trees$Transformer$$anonfun$transformStats$1.apply(Trees.scala:2925)

	at scala.collection.immutable.List.loop$1(List.scala:170)

	at scala.collection.immutable.List.mapConserve(List.scala:186)

	at scala.reflect.api.Trees$Transformer.transformStats(Trees.scala:2925)

	at scala.reflect.internal.Trees$$anonfun$itransform$7.apply(Trees.scala:1298)

	at scala.reflect.internal.Trees$$anonfun$itransform$7.apply(Trees.scala:1298)

	at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2936)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:34)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:28)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:19)

	at scala.reflect.internal.Trees$class.itransform(Trees.scala:1297)

	at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:13)

	at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:13)

	at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2897)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.scala$tools$nsc$transform$TypingTransformers$TypingTransformer$$super$transform(TypingTransformers.scala:44)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer$$anonfun$transform$2.apply(TypingTransformers.scala:46)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer$$anonfun$transform$2.apply(TypingTransformers.scala:46)

	at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2936)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:34)

	at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:46)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer$$anon$1.transform(Erasure.scala:1292)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer.transform(Erasure.scala:1302)

	at scala.tools.nsc.transform.Erasure$ErasureTransformer.transform(Erasure.scala:888)

	at scala.tools.nsc.ast.Trees$Transformer.transformUnit(Trees.scala:227)

	at scala.tools.nsc.transform.Transform$Phase.apply(Transform.scala:30)

	at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:464)

	at scala.tools.nsc.Global$GlobalPhase$$anonfun$run$1.apply(Global.scala:431)

	at scala.tools.nsc.Global$GlobalPhase$$anonfun$run$1.apply(Global.scala:431)

	at scala.collection.Iterator$class.foreach(Iterator.scala:727)

	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)

	at scala.tools.nsc.Global$GlobalPhase.run(Global.scala:431)

	at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1583)

	at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1557)

	at scala.tools.nsc.Global$Run.compileSources(Global.scala:1553)

	at scala.tools.nsc.Global$Run.compile(Global.scala:1662)

	at xsbt.CachedCompiler0.run(CompilerInterface.scala:115)

	at xsbt.CachedCompiler0.run(CompilerInterface.scala:94)

	at xsbt.CompilerInterface.run(CompilerInterface.scala:22)

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at java.lang.reflect.Method.invoke(Method.java:498)

	at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:101)

	at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:47)

	at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:41)

	at org.jetbrains.jps.incremental.scala.local.IdeaIncrementalCompiler.compile(IdeaIncrementalCompiler.scala:32)

	at org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:26)

	at org.jetbrains.jps.incremental.scala.remote.Main$.make(Main.scala:68)

	at org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:25)

	at org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at java.lang.reflect.Method.invoke(Method.java:498)

	at com.martiansoftware.nailgun.NGSession.run(NGSession.java:319)

问题原因：少引入了hadoop-client包

pom.xml中加入

   <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>2.6.0</version>
    </dependency>

 

 

jast_zsh
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
scala奇怪的bug
cheng9139的博客
04-19 5998
今天在用spark的graphx写程序的时候编译器报了一个奇怪的错 Error:scalac: Error: object VolatileIntRef does not have a member create scala.reflect.internal.FatalError: object VolatileIntRef does not have a member create at
Error:scalac: bad symbolic reference. A signature in package.class refers to type
weixin_45955039的博客
03-02 876
scala代码运行时报错Error:scalac: bad symbolic reference. A signature in package.class refers to type 找了好多办法，都只是说版本不兼容 解决方案是 自己的pom文件中，自己用到的actor类相关的jar包指定的版本是2.10.6，而自己的全局scala版本是2.11.8，修改后解决问题 在pom文件中添加以下依...
在scala中利用org.json4s 操作json
热门推荐
leehbing的博客
07-05 1万+
https://github.com/json4s/json4s/tree/v.3.2.0_scala2.10 SON4S  At this moment there are at least 6 json libraries for scala, not counting the java json libraries. All these libraries
记一次使用Phoenix-Spark时的错误：bad symbolic reference. A signature in package.class refers to term
Lee-Think的博客
12-05 1242
编译时抛出异常 ``` error: bad symbolic reference. A signature in package.class refers to term sql in package org.apache.spark which is not available. It may be completely missing from the current classpath, or the ver
flink中关于scala版本报错的解决办法
空-白_格的博客
03-22 1627
Error:scalac: Error: scala.collection.mutable.Set$.apply(Lscala/collection/Seq;)Lscala/collection/GenTraversable; java.lang.NoSuchMethodError: scala.collection.mutable.Set$.apply(Lscala/collection/Seq;)Lscala/collection/GenTraversable; at org.apache.fl...
Missing dependency ‘object scala.native in compiler mirror‘
L13763338360的博客
10-13 1184
编译seatunnel时，遇到一些问题：error: error while loading Object, Missing dependency 'object scala.native in compiler mirror', required by /modules/java.base/java/lang/Object.class
论文研究-Star direct product of symbolized vertex shift matrices: angle decomposition method in symbolic dynamics.pdf
08-18
符号化点转移矩阵的星花直积：符号动力学中的角分解方法，许传云，彭守礼，本文以矩阵形式提出了一维任意多峰映射符号动力学中的一种星花直积，这种直积基于符号化的点转移矩阵形式的角分解技术。...
MATLAB课件：ch7_symbolic_computation.pdf
06-18
MATLAB课件：ch7_symbolic_computation.pdf
muSE:自动从code.google.compmuvee-symbolic-expressions导出
05-16
muSE​​（“ muvee符号表达式”的缩写）是一种计划方言，旨在用作嵌入式语言。 它具有一些独特的语言功能，例如一流的宏，简单的常规读取时评估语法，可用作功能的向量和哈希表，以及模式匹配绑定的广泛使用。...
使用Maven编译Scala时报：bad symbolic reference. A signature in package.class refers to type compileTimeOnly
一杯咖啡。的博客
07-02 2528
在使用maven编译scala的时候，报错：    Error:scalac: bad symbolic reference. A signature in package.class refers to type compileTimeOnly    in package scala.annotation which is not available.    It may be complete...
scala安装及启动Scala报空指针异常的解决方法
cc
07-23 2820
1.在官网下载对应的安装包https://www.scala-lang.org/download/ 2.安装 选项位置保持默认即可,path环境建议手动添加,在安装中吧path勾掉 3.配置环境变量 SCALA_HOME: 变量值一栏输入：C:\Program Files（x86）\scala 也就是 Scala 的安装目录 path里面添加%SCALA_HOME%\bin 4.安装成功 ...
scala反射详解
齐梦星空
08-29 2548
概述 Scala 的反射分为两个范畴: 运行时反射：通常意义上的反射 编译时反射：宏，或者元编程 这两者之间的区别在于Environment, 而Environment又是由universe决定的. 反射的另一个重要的部分就是一个实体集合,而这个实体集合被称为mirror,有了这个实体集合我们就可以实现对需要反射的类进行对应的操作,如属性的获取,属性值得设置,以及对反射类方法的调用(其实就是成员函数的入口地址, 但请注意, 这只是个地址)! 可能有点绕, 说的直白点就是要操作类方法或者属性就需要获得指定的
idea中scala版本错乱导致的问题Error:scalac: bad symbolic reference. A signature in package.class refers to type
m0_37984367的博客
11-15 4709
Error:scalac: bad symbolic reference. A signature in package.class refers to type compileTimeOnly in package scala.annotation which is not available. It may be completely missing from the current cla...
scala报错
Mr Huo
01-11 2955
报错信息如下 Error:scalac: bad symbolic reference. A signature in package.class refers to type compileTimeOnly in package scala.annotation which is not available. It may be completely missing from ...
spark项目scala版本依赖问题
天龙八部
12-01 1858
今天从github上项目，运行一致报错，后来自己写了一个测试程序，竟然也是同样的错，最后发现竟然是scala版本不对 def main(args: Array[String]): Unit = { val conf=new SparkConf().setAppName("AppConf").setMaster("local[4]") val sc=new SparkCont...
spark开发环境搭建
喜上眉梢
07-05 1419
kafka_spark集成 kafka创建话题、消费者、生产者 1、话题 bin/kafka-topics.sh --create --zookeeper 192.168.20.10:2181 --replication-factor 1 --partitions 1 --topic test  2、生产者  bin/kafka-console-producer.sh --broke
本来能跑通但装了XGBoost后报错：NotImplementedError: Cannot convert a symbolic Tensor (lstm/strided_slice:0) to a numpy array.是怎么回事
最新发布
06-03
这个错误通常是由于尝试将一个TensorFlow符号张量转换为NumPy数组引起的。这通常发生在使用TensorFlow 2.x版本的动态图模式（Eager Execution）时，因为在动态图模式下，TensorFlow张量可以在Python对象和TensorFlow操作之间自由转换。 XGBoost依赖于NumPy，并且可能会更改TensorFlow的默认行为，从而导致此类错误。为了解决这个问题，您可以尝试在XGBoost之前禁用Eager Execution模式，可以通过以下代码实现： ```python import tensorflow as tf tf.compat.v1.disable_eager_execution() ``` 这将禁用动态图模式，并将TensorFlow操作转换为静态图模式。这应该可以解决您遇到的问题。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值