首先是错误输出:
19/09/20 14:12:40 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.101.1, 56307, None)
19/09/20 14:12:40 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.101.1:56307 with 894.3 MB RAM, BlockManagerId(driver, 192.168.101.1, 56307, None)
19/09/20 14:12:40 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.101.1, 56307, None)
19/09/20 14:12:40 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.101.1, 56307, None)
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.internal.config.package$.SHUFFLE_SPILL_NUM_ELEMENTS_FORCE_SPILL_THRESHOLD()Lorg/apache/spark/internal/config/ConfigEntry;
at org.apache.spark.sql.internal.SQLConf$.<init>(SQLConf.scala:1003)
at org.apache.spark.sql.internal.SQLConf$.<clinit>(SQLConf.scala)
at org.apache.spark.sql.internal.StaticSQLConf$.<init>(StaticSQLConf.scala:31)
at org.apache.spark.sql.internal.StaticSQLConf$.<clinit>(StaticSQLConf.scala)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
at test.App$.main(App.scala:27)
at test.App.main(App.scala)
错误所在的代码行是:
val sparkSession:SparkSession= SparkSession.builder().
master("local")
.appName("spark session example")
.getOrCreate()
这个还能有错?我也是一脸懵逼。
一顿百度,google,最后的错误的解决方法在这里记录一下:
主要是版本问题,SparkSession在sparksql中,而sparksql依赖于spark-core,所以应该保持spark-sql和spark-core版本一致,这里说一下pom文件中的依赖的写法,
原来的pom.xml中的一点写法:
< dependency>
< groupId>org.apache.spark</ groupId>
< artifactId>spark-core_2.11</ artifactId>
< version>2.2.0</ version>
</ dependency>
< dependency>
< groupId>org.apache.spark< /groupId>
< artifactId>spark-sql_2.11</ artifactId>
< version>2.3.0< /version>
< /dependency>
这里可以看见spark-core是基于scala的2.11版本的,使用的版本是2.2.0,spark-sql基于scala的2.11版本,使用的版本是2.3.0,错误就在2.3.0和2.2.0不一致导致的。所以改好之后重新import change一下就好了。
改过的pom.xml文件片段:
< dependency>
< groupId>org.apache.spark< /groupId>
< artifactId>spark-core_2.11< /artifactId>
< version>2.2.0< /version>
</ dependency>
< dependency>
< groupId>org.apache.spark< /groupId>
< artifactId>spark-sql_2.11</ artifactId>
< version>2.2.0< /version>
< /dependency>