spark运行项目error集锦

error
Exception in thread “main” java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef

$$less$colon$less; at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:156) at org.apache.spark.streaming.kafka.KafkaUtils.createStream(KafkaUtils.scala) at com.geo.testSpark.CollectorStreamV2.main(CollectorStreamV2.java:51) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$ runMain(SparkSubmit.scala:665)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/09/23 11:47:40 INFO spark.SparkContext: Invoking stop() from shutdown hook

Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 73, datanode3): ExecutorLostFailure (executor 3 lost)

15/09/23 15:00:39 ERROR scheduler.ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - java.lang.NoSuchMethodError: org.apache.spark.util.Utils$.newDaemonFixedThreadPool(ILjava/lang/String;)Ljava/util/concurrent/ThreadPoolExecutor; at org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:114) at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:125) at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:109) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher

$$anonfun$8.apply(ReceiverTracker.scala:308) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$ anonfun $8.apply(ReceiverTracker.scala:300) at org.apache.spark.SparkContext$ $anonfun$runJob $5.apply(SparkContext.scala:1767) at org.apache.spark.SparkContext$ $anonfun$runJob $5.apply(SparkContext.scala:1767) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) at org.apache.spark.scheduler.Task.run(Task.scala:70) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 15/09/23 15:00:39 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 2.0 (TID 70, datanode1): java.lang.NoSuchMethodError: org.apache.spark.util.Utils$.newDaemonFixedThreadPool(ILjava/lang/String;)Ljava/util/concurrent/ThreadPoolExecutor;
at org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:114)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:125)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:109)
at org.apache.spark.streaming.scheduler.ReceiverTracker $ReceiverLauncher$ $anonfun$8.apply(ReceiverTracker.scala:308)
at org.apache.spark.streaming.scheduler.ReceiverTracker $ReceiverLauncher$ $anonfun$8.apply(ReceiverTracker.scala:300)
at org.apache.spark.SparkContext $$anonfun$runJob$5.apply(SparkContext.scala:1767) at org.apache.spark.SparkContext$$ anonfun $runJob$5.apply(SparkContext.scala:1767)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor $TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
解决办法：
代码用到的spark的版本、kafka的版本、Scala的版本（如果有用到Scala，Spark 1.4.1匹配的版本Scala 2.10.4.）需一致，并且最好与线上服务器的各个版本一致。

error
org.apache.spark.SparkException: Couldn’t find leader offsets for Set()
解决办法：
创建新的topic和groupid
【error】
15/09/23 17:15:00 ERROR kafka.DirectKafkaInputDStream: ArrayBuffer(org.apache.spark.SparkException: Couldn’t find leaders for Set([rtb-collector-time-v04,0]))
15/09/23 17:15:00 ERROR scheduler.JobScheduler: Error generating jobs for time 1442999700000 ms
org.apache.spark.SparkException: ArrayBuffer(org.apache.spark.SparkException: Couldn’t find leaders for Set([rtb-collector-time-v04,0]))
at org.apache.spark.streaming.kafka.DirectKafkaInputDStream.latestLeaderOffsets(DirectKafkaInputDStream.scala:94)
at org.apache.spark.streaming.kafka.DirectKafkaInputDStream.compute(DirectKafkaInputDStream.scala:116)
at org.apache.spark.streaming.dstream.DStream

$$anonfun$getOrCompute$1$$ anonfun $1$$anonfun$apply $7.apply(DStream.scala:350) at org.apache.spark.streaming.dstream.DStream$ $anonfun$getOrCompute $1$$anonfun$1 $$anonfun$apply$7.apply(DStream.scala:350) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.apache.spark.streaming.dstream.DStream$$ anonfun $getOrCompute$1 $$anonfun$1.apply(DStream.scala:349) at org.apache.spark.streaming.dstream.DStream$$ anonfun $getOrCompute$1$$anonfun$1.apply(DStream.scala:349)
解决办法：
spark集群中/etc/hosts 里加配置上所有kafka节点的ip