idea scala 跑_intellij idea scala 怎么运行

最新推荐文章于 2024-03-28 10:27:27 发布

weixin_39520013

最新推荐文章于 2024-03-28 10:27:27 发布

阅读量916

点赞数 1

文章标签： idea scala 跑

本文链接：https://blog.csdn.net/weixin_39520013/article/details/111850076

版权

打成Jar包，直接用spark-submit 方式运行就可以成功，直接用IntelliJ就总是失败，不知道怎么回事，求大神指点。程序是

object HelloSpark {

def main(args:Array[String]): Unit ={

if(args.length!=2){

System.err.println("Usage:HelloSpark ")

System.exit(1)

}

val conf = new SparkConf().setAppName("helloSpark").setMaster("spark://master:7077").set("spark.executor.memory", "1g")

.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")

val sc = new SparkContext(conf)

sc.textFile(args(0)).map(_.split("\t")).filter(_.length==6).map(x=>(x(1),1)).reduceByKey(_+_).map(x=>(x._2,x._1)).

sortByKey(false).map(x=>(x._2,x._1)).saveAsTextFile(args(1))

sc.stop()

}

这是网上的一个实例，步骤都是一步一步来的。

16/02/23 16:39:53 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, worker1): java.lang.ClassNotFoundException: com.spark.firstApp.HelloSpark$$anonfun$2

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:68)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)

......

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

16/02/23 16:39:53 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 2, worker0, partition 0,NODE_LOCAL, 2134 bytes)

16/02/23 16:39:53 INFO TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1) on executor worker2: java.lang.ClassNotFoundException (com.spark.firstApp.HelloSpark$$anonfun$2) [duplicate 1]

16/02/23 16:39:53 INFO TaskSetManager: Starting task 1.1 in stage 0.0 (TID 3, worker2, partition 1,NODE_LOCAL, 2134 bytes)

16/02/23 16:39:53 INFO TaskSetManager: Lost task 1.1 in stage 0.0 (TID 3) on executor worker2: java.lang.ClassNotFoundException (com.spark.firstApp.HelloSpark$$anonfun$2) [duplicate 2]

16/02/23 16:39:53 INFO TaskSetManager: Starting task 1.2 in stage 0.0 (TID 4, worker1, partition 1,NODE_LOCAL, 2134 bytes)

16/02/23 16:39:53 INFO TaskSetManager: Lost task 1.2 in stage 0.0 (TID 4) on executor worker1: java.lang.ClassNotFoundException (com.spark.firstApp.HelloSpark$$anonfun$2) [duplicate 3]

16/02/23 16:39:53 INFO TaskSetManager: Starting task 1.3 in stage 0.0 (TID 5, worker1, partition 1,NODE_LOCAL, 2134 bytes)

16/02/23 16:39:53 INFO TaskSetManager: Lost task 1.3 in stage 0.0 (TID 5) on executor worker1: java.lang.ClassNotFoundException (com.spark.firstApp.HelloSpark$$anonfun$2) [duplicate 4]

16/02/23 16:39:53 ERROR TaskSetManager: Task 1 in stage 0.0 failed 4 times; aborting job

16/02/23 16:39:53 INFO TaskSchedulerImpl: Cancelling stage 0

16/02/23 16:39:53 INFO TaskSchedulerImpl: Stage 0 was cancelled

16/02/23 16:39:53 INFO DAGScheduler: ShuffleMapStage 0 (map at HelloSpark.scala:20) failed in 10.015 s

16/02/23 16:39:53 INFO DAGScheduler: Job 0 failed: sortByKey at HelloSpark.scala:21, took 10.156984 s

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 5, worker1): java.lang.ClassNotFoundException: com.spark.firstApp.HelloSpark$$anonfun$2