Spark application 支持自定义listener,用户可以实时获取任务状态给自己的监控系统,可以获取以下几个状态:
trait SparkListener {
/**
* 当一个state执行成功或者失败的时候调用,包含了已完成stage的信息
* Called when a stage completes successfully or fails, with information on the completed stage.
*/
def onStageCompleted(stageCompleted: SparkListenerStageCompleted) { }
/**
* 当一个state提交的时候的时候调用
* Called when a stage is submitted
*/
def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted) { }
/**
* 当一个task任务开始时候调用
* Called when a task starts
*/
def onTaskStart(taskStart: SparkListenerTaskStart) { }
/**
* 当一个task执行成功或者失败的时候调用,包含了已完成task的信息
* Called when a task begins remotely fetching its result (will not be called for tasks that do
* not need to fetch the result remotely).
*/
def onTaskGettin