自定义Spark application 监听器

最新推荐文章于 2021-12-23 16:20:53 发布
最新推荐文章于 2021-12-23 16:20:53 发布
阅读量6k 收藏 3
点赞数
分类专栏： spark 文章标签： spark 大数据
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u011291159/article/details/51160343
版权

Spark application 支持自定义listener，用户可以实时获取任务状态给自己的监控系统，可以获取以下几个状态：

trait SparkListener {
  /**
    * 当一个state执行成功或者失败的时候调用，包含了已完成stage的信息
   * Called when a stage completes successfully or fails, with information on the completed stage.
   */
  def onStageCompleted(stageCompleted: SparkListenerStageCompleted) { }

  /**
    * 当一个state提交的时候的时候调用
   * Called when a stage is submitted
   */
  def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted) { }

  /**
    * 当一个task任务开始时候调用
   * Called when a task starts
   */
  def onTaskStart(taskStart: SparkListenerTaskStart) { }

  /**
    * 当一个task执行成功或者失败的时候调用，包含了已完成task的信息
   * Called when a task begins remotely fetching its result (will not be called for tasks that do
   * not need to fetch the result remotely).
   */
  def onTaskGettingResult(taskGettingResult: SparkListenerTaskGettingResult) { }

  /**
    * 当一个task结束开始时候调用
   * Called when a task ends
   */
  def onTaskEnd(taskEnd: SparkListenerTaskEnd) { }

  /**
    * 当一个job启动开始调用
   * Called when a job starts
   */
  def onJobStart(jobStart: SparkListenerJobStart) { }

  /**
    *  当一个job执行成功或者失败的时候调用，包含了已完成job的信息
   * Called when a job ends
   */
  def onJobEnd(jobEnd: SparkListenerJobEnd) { }

  /**
    * 当一个环境变量改变的时候开始调用
   * Called when environment properties have been updated
   */
  def onEnvironmentUpdate(environmentUpdate: SparkListenerEnvironmentUpdate) { }

  /**
   * Called when a new block manager has joined
   */
  def onBlockManagerAdded(blockManagerAdded: SparkListenerBlockManagerAdded) { }

  /**
   * Called when an existing block manager has been removed
   */
  def onBlockManagerRemoved(blockManagerRemoved: SparkListenerBlockManagerRemoved) { }

  /**
   * Called when an RDD is manually unpersisted by the application
   */
  def onUnpersistRDD(unpersistRDD: SparkListenerUnpersistRDD) { }

  /**
   * Called when the application starts
   */
  def onApplicationStart(applicationStart: SparkListenerApplicationStart) { }

  /**
   * Called when the application ends
   */
  def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd) { }

  /**
   * Called when the driver receives task metrics from an executor in a heartbeat.
   */
  def onExecutorMetricsUpdate(executorMetricsUpdate: SparkListenerExecutorMetricsUpdate) { }

  /**
   * Called when the driver registers a new executor.
   */
  def onExecutorAdded(executorAdded: SparkListenerExecutorAdded) { }

  /**
   * Called when the driver removes an executor.
   */
  def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved) { }

  /**
    * 
   * Called when the driver receives a block update info.
   */
  
  def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated) { }
}

只需要extends SparkListener，然后注册到sparkContext 既可以实现自定义listener，代码逻辑如下： 

package com.suning.spark

import org.apache.spark.scheduler.MySparkListener
import org.apache.spark.{SparkConf, SparkContext}

/**
  * Created by Ricky on 2016/4/14 0014.
  */
object JobProcesser {
  def main(args: Array[String]) {
    val sparkConf = new SparkConf().setAppName("KafkaWordCountProducer").setMaster("local")
    val sc = new SparkContext(sparkConf)
    /*  sc.setJobGroup("test1","testdesc")
      val completedJobs= sc.jobProgressListener*/
    sc.addSparkListener(new MySparkListener)
    val rdd1 = sc.parallelize(List(('a', 'c', 1), ('b', 'a', 1), ('b', 'd', 8)))
    val rdd2 = sc.parallelize(List(('a', 'c', 2), ('b', 'c', 5), ('b', 'd', 6)))
    val rdd3 = rdd1.union(rdd2).map {
      x => {
        Thread.sleep(500)
        x
      }
    }.count()
    rdd1.map(x => 0.2).map(x => 0).map {
      x => {
        if (x == 0) {
          throw new Exception("my exeception")
        }
      }
        x
    }.reduce(_ + _)
    println(rdd3)
    sc.stop()
  }
}



package org.apache.spark.scheduler


/**
  * Created by Ricky on 2016/4/14 0014.
  */
class MySparkListener extends SparkListener {
  override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd) {
    println("*************************************************")
    println("app:end")
    println("*************************************************")
  }

  override def onJobEnd(jobEnd: SparkListenerJobEnd) {
    println("*************************************************")
    println("job:end")
    jobEnd.jobResult match {
      case JobSucceeded =>
        println("job:end:JobSucceeded")
      case JobFailed(exception) =>
        println("job:end:file")
        exception.printStackTrace()
    }
    println("*************************************************")
  }
}


执行日志： 

16/04/15 11:12:29 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.24.74.95:52662]
16/04/15 11:12:29 INFO Utils: Successfully started service 'sparkDriver' on port 52662.
16/04/15 11:12:29 INFO SparkEnv: Registering MapOutputTracker
16/04/15 11:12:29 INFO SparkEnv: Registering BlockManagerMaster
16/04/15 11:12:29 INFO DiskBlockManager: Created local directory at C:\Users\14070345\AppData\Local\Temp\blockmgr-5573ab97-dea7-45e5-ae99-01fc689ae0e4
16/04/15 11:12:29 INFO MemoryStore: MemoryStore started with capacity 950.4 MB
16/04/15 11:12:29 INFO HttpFileServer: HTTP File server directory is C:\Users\14070345\AppData\Local\Temp\spark-6c67028e-c0e3-412d-8eab-31962a1b0cbe\httpd-a348c669-8194-41d1-9be2-2921b4a3da56
16/04/15 11:12:29 INFO HttpServer: Starting HTTP Server
16/04/15 11:12:29 INFO Utils: Successfully started service 'HTTP file server' on port 52663.
16/04/15 11:12:29 INFO SparkEnv: Registering OutputCommitCoordinator
16/04/15 11:12:29 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/04/15 11:12:29 INFO SparkUI: Started SparkUI at http://10.24.74.95:4040
16/04/15 11:12:30 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/04/15 11:12:30 INFO Executor: Starting executor ID driver on host localhost
16/04/15 11:12:30 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52670.
16/04/15 11:12:30 INFO NettyBlockTransferService: Server created on 52670
16/04/15 11:12:30 INFO BlockManagerMaster: Trying to register BlockManager
16/04/15 11:12:30 INFO BlockManagerMasterEndpoint: Registering block manager localhost:52670 with 950.4 MB RAM, BlockManagerId(driver, localhost, 52670)
16/04/15 11:12:30 INFO BlockManagerMaster: Registered BlockManager
16/04/15 11:12:30 INFO SparkContext: Starting job: count at JobProcesser.scala:25
16/04/15 11:12:30 INFO DAGScheduler: Got job 0 (count at JobProcesser.scala:25) with 2 output partitions
16/04/15 11:12:30 INFO DAGScheduler: Final stage: ResultStage 0(count at JobProcesser.scala:25)
16/04/15 11:12:30 INFO DAGScheduler: Parents of final stage: List()
16/04/15 11:12:30 INFO DAGScheduler: Missing parents: List()
16/04/15 11:12:30 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at map at JobProcesser.scala:20), which has no missing parents
16/04/15 11:12:31 INFO MemoryStore: ensureFreeSpace(2400) called with curMem=0, maxMem=996566630
16/04/15 11:12:31 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.3 KB, free 950.4 MB)
16/04/15 11:12:31 INFO MemoryStore: ensureFreeSpace(1546) called with curMem=2400, maxMem=996566630
16/04/15 11:12:31 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1546.0 B, free 950.4 MB)
16/04/15 11:12:31 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:52670 (size: 1546.0 B, free: 950.4 MB)
16/04/15 11:12:31 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:861
16/04/15 11:12:31 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at map at JobProcesser.scala:20)
16/04/15 11:12:31 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
16/04/15 11:12:31 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, PROCESS_LOCAL, 2408 bytes)
16/04/15 11:12:31 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/04/15 11:12:37 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 953 bytes result sent to driver
16/04/15 11:12:37 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, PROCESS_LOCAL, 2413 bytes)
16/04/15 11:12:37 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
16/04/15 11:12:37 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 6046 ms on localhost (1/2)
16/04/15 11:12:43 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 953 bytes result sent to driver
16/04/15 11:12:43 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 6012 ms on localhost (2/2)
16/04/15 11:12:43 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
16/04/15 11:12:43 INFO DAGScheduler: ResultStage 0 (count at JobProcesser.scala:25) finished in 12.066 s
16/04/15 11:12:43 INFO DAGScheduler: Job 0 finished: count at JobProcesser.scala:25, took 12.731024 s
*************************************************
job:end
job:end:JobSucceeded
*************************************************
16/04/15 11:12:43 INFO SparkContext: Starting job: reduce at JobProcesser.scala:33
16/04/15 11:12:43 INFO DAGScheduler: Got job 1 (reduce at JobProcesser.scala:33) with 1 output partitions
16/04/15 11:12:43 INFO DAGScheduler: Final stage: ResultStage 1(reduce at JobProcesser.scala:33)
16/04/15 11:12:43 INFO DAGScheduler: Parents of final stage: List()
16/04/15 11:12:43 INFO DAGScheduler: Missing parents: List()
16/04/15 11:12:43 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[6] at map at JobProcesser.scala:26), which has no missing parents
16/04/15 11:12:43 INFO MemoryStore: ensureFreeSpace(2336) called with curMem=3946, maxMem=996566630
16/04/15 11:12:43 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.3 KB, free 950.4 MB)
16/04/15 11:12:43 INFO MemoryStore: ensureFreeSpace(1410) called with curMem=6282, maxMem=996566630
16/04/15 11:12:43 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1410.0 B, free 950.4 MB)
16/04/15 11:12:43 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:52670 (size: 1410.0 B, free: 950.4 MB)
16/04/15 11:12:43 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:861
16/04/15 11:12:43 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[6] at map at JobProcesser.scala:26)
16/04/15 11:12:43 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
16/04/15 11:12:43 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, localhost, PROCESS_LOCAL, 2299 bytes)
16/04/15 11:12:43 INFO Executor: Running task 0.0 in stage 1.0 (TID 2)
16/04/15 11:12:43 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 2)
java.lang.Exception: my exeception
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply$mcII$sp(JobProcesser.scala:29)
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply(JobProcesser.scala:27)
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply(JobProcesser.scala:27)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
	at scala.collection.TraversableOnce$class.reduceLeft(TraversableOnce.scala:172)
	at scala.collection.AbstractIterator.reduceLeft(Iterator.scala:1157)
	at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:993)
	at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:991)
	at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1960)
	at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1960)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:88)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
16/04/15 11:12:43 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2, localhost): java.lang.Exception: my exeception
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply$mcII$sp(JobProcesser.scala:29)
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply(JobProcesser.scala:27)
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply(JobProcesser.scala:27)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
	at scala.collection.TraversableOnce$class.reduceLeft(TraversableOnce.scala:172)
	at scala.collection.AbstractIterator.reduceLeft(Iterator.scala:1157)
	at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:993)
	at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:991)
	at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1960)
	at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1960)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:88)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

16/04/15 11:12:43 ERROR TaskSetManager: Task 0 in stage 1.0 failed 1 times; aborting job
16/04/15 11:12:43 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
16/04/15 11:12:43 INFO TaskSchedulerImpl: Cancelling stage 1
16/04/15 11:12:43 INFO DAGScheduler: ResultStage 1 (reduce at JobProcesser.scala:33) failed in 0.068 s
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 2, localhost): java.lang.Exception: my exeception
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply$mcII$sp(JobProcesser.scala:29)
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply(JobProcesser.scala:27)
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply(JobProcesser.scala:27)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
	at scala.collection.TraversableOnce$class.reduceLeft(TraversableOnce.scala:172)
	at scala.collection.AbstractIterator.reduceLeft(Iterator.scala:1157)
	at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:993)
	at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:991)
	at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1960)
	at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1960)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:88)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
Caused by: java.lang.Exception: my exeception
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply$mcII$sp(JobProcesser.scala:29)
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply(JobProcesser.scala:27)
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply(JobProcesser.scala:27)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
	at scala.collection.TraversableOnce$class.reduceLeft(TraversableOnce.scala:172)
	at scala.collection.AbstractIterator.reduceLeft(Iterator.scala:1157)
	at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:993)
	at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:991)
	at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1960)
	at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1960)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:88)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
16/04/15 11:12:43 INFO DAGScheduler: Job 1 failed: reduce at JobProcesser.scala:33, took 0.103760 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 2, localhost): java.lang.Exception: my exeception
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply$mcII$sp(JobProcesser.scala:29)
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply(JobProcesser.scala:27)
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply(JobProcesser.scala:27)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
	at scala.collection.TraversableOnce$class.reduceLeft(TraversableOnce.scala:172)
	at scala.collection.AbstractIterator.reduceLeft(Iterator.scala:1157)
	at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:993)
	at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:991)
	at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1960)
	at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1960)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:88)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1841)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1961)
	at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:1007)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
	at org.apache.spark.rdd.RDD.reduce(RDD.scala:989)
	at com.suning.spark.JobProcesser$.main(JobProcesser.scala:33)
	at com.suning.spark.JobProcesser.main(JobProcesser.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
*************************************************
	at java.lang.reflect.Method.invoke(Method.java:606)
job:end
job:end:file
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.lang.Exception: my exeception
*************************************************
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply$mcII$sp(JobProcesser.scala:29)
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply(JobProcesser.scala:27)
	at com.suning.spark.JobProcesser$$anonfun$main$2.apply(JobProcesser.scala:27)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
	at scala.collection.TraversableOnce$class.reduceLeft(TraversableOnce.scala:172)
	at scala.collection.AbstractIterator.reduceLeft(Iterator.scala:1157)
	at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:993)
	at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:991)
	at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1960)
	at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1960)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:88)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
*************************************************
16/04/15 11:12:43 INFO SparkContext: Invoking stop() from shutdown hook
app:end
*************************************************
16/04/15 11:12:43 INFO SparkUI: Stopped Spark web UI at http://10.24.74.95:4040
16/04/15 11:12:43 INFO DAGScheduler: Stopping DAGScheduler
16/04/15 11:12:43 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/04/15 11:12:43 INFO MemoryStore: MemoryStore cleared
16/04/15 11:12:43 INFO BlockManager: BlockManager stopped
16/04/15 11:12:43 INFO BlockManagerMaster: BlockManagerMaster stopped
16/04/15 11:12:43 INFO SparkContext: Successfully stopped SparkContext
16/04/15 11:12:43 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/04/15 11:12:43 INFO ShutdownHookManager: Shutdown hook called
16/04/15 11:12:43 INFO ShutdownHookManager: Deleting directory C:\Users\14070345\AppData\Local\Temp\spark-6c67028e-c0e3-412d-8eab-31962a1b0cbe

Process finished with exit code 1






笛香幽谷（欧锐）
关注
  • 0
    点赞
  • 踩
  • 3
    收藏
    觉得还不错? 一键收藏
  • 4
    评论
专栏目录
spark自定义监听器
u013939918的博客
05-24 867
Spark application 支持自定义listener，用户可以实时获取任务状态给自己的监控系统，可以获取以下几个状态： trait SparkListener { /** * 当一个state执行成功或者失败的时候调用，包含了已完成stage的信息 * Called when a stage completes successfully or fails, with information on the completed stage. */ def onStageC...
Spark简介
热门推荐
qinlingheshang的博客
03-03 2万+
Spark简介Spark是什么新的改变Spark集群架构合理的创建标题，有助于目录的生成如何改变文本的样式插入链接与图片如何插入一段漂亮的代码片生成一个适合你的列表创建一个表格设定内容居中、居左、居右SmartyPants创建一个自定义列表如何创建一个注脚注释也是必不可少的KaTeX数学公式新的甘特图功能，丰富你的文章UML 图表FLowchart流程图导出与导入导出导入 Spark是什么 新的改变 我们对Markdown编辑器进行了一些功能拓展与语法支持，除了标准的Markdown编辑器功能，我们增加了如
自定义 Spark application 监听器进行task异常处理 JAVA版
CODE男孩的博客
02-15 1396
最近要截取sparkHistory里面application的运行日志，发现task级别的某些日志拿不到，后来想了个办法搞监听器，然后一点点学习，将经验记录下来。 在spark程序中，task有失败重试机制（根据 spark.task.maxFailures 配置，默认是4次），当task执行失败时，并不会直接导致整个应用程序down掉，只有在重试了 spark.task.maxFailures ...
spark源码 -- spark ListenerBus 监听器
qq_824203453_的博客
07-09 329
Spark 源码中对livelistenerBus进行了这样的注释：即所有spark消息SparkListenerEvents 被异步的发送给已经注册过的SparkListeners. 在SparkContext中, 首先会创建LiveListenerBus实例,这个类主要功能如下:保存有消息队列,负责消息的缓存保存有注册过的listener,负责消息的分发listener链表保存在Listene...
spark 自定义累加器与自定义事件
12-30 223
1 可以实现简单的累加器，功能是只保存最后一个字符串。 import org.apache.spark.util.AccumulatorV2; public class MyAccumulator extends AccumulatorV2<String, String> { private String str = ""; public MyAccumulat...
【spark】metrics信息的获取（监听器等）
lsr40的博客
07-20 2617
一、背景 最近看到了一篇文章，说的是spark小文件合并的问题 Spark 小文件合并优化实践：https://mp.weixin.qq.com/s/195nFBH0kpZEXekHiQAfrA（作者：偷闲小苑） 其实关于小文件合并我之前也写过类似的文章，大体的方案也就是 1、存之前做一个数据量的预估，然后repartition/coalesce 2、存完之后，触发一个merge合并小文件（可以是hive也可以是spark任务） 3、使用一些数据湖（delta lake，hudi，iceber
一文搞定Spark的DAG调度器（DAGScheduler）
LMR的博客
05-10 8607
Spark的DAG调度器1. DAG定义2. DAG实例化3. DAGScheduler划分Stage的原理4 DAGScheduer划分Stage的源代码 1. DAG定义 RDD DAG还 构建了基于数据流之上的操作算子流， 即RDD的各个分区的数据总共会经过哪些 Transformation和 Action这两种类型的一系列操作的调度运行， 从而RDD先被Trans­formation操作转换为新的RDD, 然后被Action操作 将结果反馈到Driver Program或存储到外部存储系统上 。 上
spark streaming流式计算---监听器
u010990043的博客
10-30 3266
       随着对spark的了解，有时会觉得spark就像一个宝盒一样时不时会出现一些难以置信的新功能。每一个新功能被挖掘，就可以使开发过程变得更加便利一点。甚至使很多不可能完成或者完成起来比较复杂的操作，变成简单起来。有些功能是框架专门开放给用户使用，有些则是框架内部使用但是又对外暴露了接口，用户也可以使用的功能。      今天和大家分享的是两个监听器SparkListener和stre...
Spark端口
tangfatter的博客
12-23 5378
一、4040端口 spark任务运行后，会将Driver所在机器绑定到4040端口，提供当前任务的监控页面。 此端口号默认为4040，展示信息如下： 调度器阶段和任务列表 RDD大小和内存使用情况 环境信息 正在运行的executors的信息 演示如下： 通过spark的java接口启动spark SparkConf sc = new SparkConf(); // 创建一个SpakrConf对象用于配置Spark sc.setAppName("name") // 设定程序名字
Flex中itemRenderer的使用简介
07-29
如果需要在ItemRenderer中添加交互行为，可以添加事件监听器。例如，当用户点击一个列表项时触发某个动作： ```mxml (event)" ... /> ``` 在对应的AS3类中实现`handleItemClick`方法。 5. **优化性能**： ...
Flex应用AdvancedDataGrid表头皮肤
05-24
在AdvancedDataGrid中，我们可以创建一个继承自spark.skins.spark.ApplicationSkin的自定义表头皮肤类，然后在XML中定义各个元素的布局和样式。 2. **表头元素**：AdvancedDataGrid的表头由多个元素组成，包括...
使用netty实现TCP长链接消息写入kafka以及kafka批量消费数据
07-20
2. **消息编码与解码**：在Netty中，我们通常会自定义编解码器来将原始字节流转换为业务对象，以便处理接收到的数据。例如，可以创建一个MessageDecoder和MessageEncoder来解析TCP连接中的消息。 3. **Kafka生产者*...
Flex的组件
05-25
1. **组件自定义**：Flex允许开发者通过继承已有的UIComponent或使用MX或Spark组件来创建自定义组件。在这个例子中，我们需要继承Tree组件，并添加复选框和输入框。 2. **事件处理**：添加输入框后，需要监听用户的...
Spark 1.5.2 on yarn升级问题总结
我想静静
04-15 5699
Spark 1.4.0 standlone升级到Spark 1.5.2 on yarn问题总结
spark2.1.0 on yarn with CDH5.8.0 安装实战
我想静静
03-30 5606
Spark 版本发布很快，CDH集成最新Spark版本需要一定时间，并且CDH 集成的Spark版本不支持Spark-sql。本文档的目的在目前cdh平台集成最新spark，方便测试和使用最新功能。
CDH Gateway docker实战
我想静静
01-24 4027
将CDH gateway 安装到docker里面。
Spark standlone driver on cluster 用户权限问题
我想静静
11-18 2932
Spark standlone 模式下面driver 提交到cluster，application 用户变成了spark 部署环境用户，这样会带来两个问题： a) 无法知道那个用户提交的任务； b) spark权限问题无法管理，比如读取Hive、HBase、HDFS数据的权限问题； 本文提出一种简单的修改方法，让Spark standlone driver on cluster 模型下面读取数据权限问题可以解决。
评论 4
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值