Spark运行jar包时出现的报错及应对措施汇总

此文章是博主自己在集群上使用spark运行jar包出现的报错，以及采取的解决办法，在寻找解决办法时借助了网络（可能参考了某些大神的文章），仅供学习参考使用，不作他用

报错

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/10/08 21:02:10 INFO SparkContext: Running Spark version 3.0.0
22/10/08 21:02:10 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:380)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:120)
	at test.wyh.wordcount.TestWordCount$.main(TestWordCount.scala:10)
	at test.wyh.wordcount.TestWordCount.main(TestWordCount.scala)
22/10/08 21:02:10 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:380)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:120)
	at test.wyh.wordcount.TestWordCount$.main(TestWordCount.scala:10)
	at test.wyh.wordcount.TestWordCount.main(TestWordCount.scala)
 
Process finished with exit code 1

解决

在VM options 配置如下参数：

-Dspark.master=local[*]

报错：端口错误

解决：

查看端口

 

报错：

Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
	at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:314)
	at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:268)
	at java.util.jar.JarVerifier.processEntry(JarVerifier.java:316)
	at java.util.jar.JarVerifier.update(JarVerifier.java:228)
	at java.util.jar.JarFile.initializeVerifier(JarFile.java:383)
	at java.util.jar.JarFile.getInputStream(JarFile.java:450)
	at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:977)
	at sun.misc.Resource.cachedInputStream(Resource.java:77)
	at sun.misc.Resource.getByteBuffer(Resource.java:160)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.util.Utils$.classForName(Utils.scala:235)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:836)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

解决：手动删除

删除后缀名为*.RSA ，*.DSA ，*.SF的文件

报错：版本冲突

解决：换成和集群上一样的版本，博主使用的集群的scala版本为2.12