本地运行spark测试程序,报错如下:
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/broadcast/Broadcast
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.broadcast.Broadcast
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
一般报NoClassDefFoundError通常是自定义的包名或路径写错了,但这里Broadcast是spark自带的类,之所以会找不到,是因为在pom.xml中compile.scope我定义的是provided,也就是本地不会编译这些包,导致本地调试的时候找不到。所以需要调试定义Application配置时需要把include dependencies with “Provided” scope勾选上。
<properties>
<java.version>1.8</java.version>
<scala.version>2.11.8</scala.version>
<scala.binary.version>2.11</scala.binary.version>
<scala.main.verison>2.10</scala.main.verison>
<compile.scope>provided</compile.scope>
</properties>