在本地调试运行spark程序时,报错Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/spark/SparkConf,这个错误就是程序在运行时找不到类
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
at cn.jin.spark.JavaLambdaWordCount.main(JavaLambdaWordCount.java:25)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
这是由于我在pom文件中加入了scope标签,导致程序在编译时忽略了加入scope标签的依赖,所以找不到依赖,只要把对应的scope标签注释掉就可了
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>${spark.version}</version>
<!-- 将下面这条注释掉就可以了 -->
<!-- <scope>provided</scope> -->
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
<!-- <scope>provided</scope> -->
</dependency>
修改后运行成功
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/09/18 13:06:06 INFO SparkContext: Running Spark version 3.0.0
21/09/18 13:06:07 INFO ResourceUtils: ==============================================================
21/09/18 13:06:07 INFO ResourceUtils: Resources for spark.driver:
21/09/18 13:06:07 INFO ResourceUtils: ==============================================================
21/09/18 13:06:07 INFO SparkContext: Submitted application: JavaLambdaWordCount