scala升级问题

最新推荐文章于 2024-08-29 09:27:42 发布

anddyhua

最新推荐文章于 2024-08-29 09:27:42 发布

阅读量1.3k

点赞数

分类专栏： Note for Bigdata

本文链接：https://blog.csdn.net/anddyhua/article/details/96748562

版权

本文介绍了在尝试将Scala版本从2.11升级到2.12时遇到的问题，SPARK3不再支持该升级。文章提到了需要删除2.11相关的jar包，并替换为2.12的版本。此外，还讨论了在standalone模式下使用spark-shell时出现的错误，错误表现为standalone spark webui cores为0，原因是Slave启动时遇到`java.net.ConnectException: Connection refused`。为解决此问题，建议在`spark-env.sh`中配置`SPARK_MASTER_HOST`，并使用指定端口的`start-master.sh`和`start-slave.sh`脚本。

摘要由CSDN通过智能技术生成

SPARK3 2.11 不再支持 scala从2.11升级到2.12，替换掉2.11包和scala 以及json4s等包
ls ./jars|egrep “(^scala|^hadoop|^json4s)”|xargs rm -f
ls …/jars|egrep “(^scala|^hadoop|^json4s)”|xargs -i cp {} .
ls …/jars|grep 2.11

19/07/21 18:33:33 INFO yarn.Client: Deleted staging directory hdfs://172.16.119.188:9000/user/spark/.sparkStaging/application_1563641923608_0003
19/07/21 18:33:33 ERROR yarn.Client: Application diagnostics message: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
        at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$$anonfun$2.apply(BeanIntrospector.scala:170)
        at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$$anonfun$2.apply(BeanIntrospector.scala:169)
        at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:244)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at scala.collection.TraversableLike.flatMap(TraversableLike.scala:244)
        at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:241)
        at scala.collection.immutable.List.flatMap(List.scala:355)
        at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.apply(BeanIntrospector.scala:169)
        at com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$._descriptorFor(ScalaAnnotationIntrospectorModule.scala:22)
        at com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospector$.fieldName(ScalaAnnotationIntrospectorModule.scala:30)
        at com.fasterxml.jackson.module.scala.introspect.ScalaAnn