DolphinScheduler作业流程

最新推荐文章于 2024-07-25 19:06:24 发布

不甚了然

最新推荐文章于 2024-07-25 19:06:24 发布

阅读量540

点赞数

分类专栏：其他知识文章标签：大数据

本文链接：https://blog.csdn.net/blackjjcat/article/details/130748810

版权

其他知识专栏收录该内容

3 篇文章 0 订阅

订阅专栏

0.核心模块功能

dolphinscheduler-alert：告警模块，提供 AlertServer 服务。
dolphinscheduler-api：Web应用模块，提供 ApiServer 服务。
dolphinscheduler-common：通用的常量枚举、工具类、数据结构或者基类
dolphinscheduler-dao：提供数据库访问等操作。
dolphinscheduler-remote：基于 netty 的客户端、服务端
dolphinscheduler-service：Service模块，包含Quartz、Zookeeper、日志客户端访问服务，便于server模块和api模块调用
dolphinscheduler-ui：前端模块
dolphinscheduler-scheduler-plugin：插件化的调度模块
dolphinscheduler-master：Master模块
dolphinscheduler-worker：Worker模块

1 创建

1.1 工作流创建

UI界面进行工作流定义，拖拽左侧任务时，触发的请求接口为TaskDefinitionController的genTaskCodeList
期间还后置触发了很多查询类的接口，比如worker-groups

@ApiOperation(value = "genTaskCodeList", notes = "GEN_TASK_CODE_LIST_NOTES")
@ApiImplicitParams({
        @ApiImplicitParam(name = "genNum", value = "GEN_NUM", required = true, dataTypeClass = int.class, example = "1")
})
@GetMapping(value = "/gen-task-codes")
@ResponseStatus(HttpStatus.OK)
@ApiException(LOGIN_USER_QUERY_PROJECT_LIST_PAGING_ERROR)
@AccessLogAnnotation(ignoreRequestArgs = "loginUser")
public Result genTaskCodeList(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
                              @RequestParam("genNum") Integer genNum) {
    Map<String, Object> result = taskDefinitionService.genTaskCodeList(genNum);
    return returnDataList(result);
}

点击保存以后，发送的请求是process-definition，也就是ProcessDefinitionController的根定义，对应的是createProcessDefinition接口
参数中有taskDefinitionJson，里面定义了任务类型

@PostMapping()
@ResponseStatus(HttpStatus.CREATED)
@ApiException(CREATE_PROCESS_DEFINITION_ERROR)
@AccessLogAnnotation(ignoreRequestArgs = "loginUser")
public Result createProcessDefinition(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser,

1.2 任务创建

作业类型的定义在t_ds_task_definition表和t_ds_task_definition_log表（t_ds_task_definition_log应该是记录了所有的作业历史，t_ds_task_definition不包含删除的）
UI工作流的定义下方有作业定义的地方，可以直接在作业定义创建作业，在工作流创建以后作业UI上会同步产生作业，也就是上述的createProcessDefinition接口完成后会内部调用Task的接口
Task创建走的是task-definition/save-single接口

@PostMapping("/save-single")
@ResponseStatus(HttpStatus.CREATED)
@ApiException(CREATE_TASK_DEFINITION_ERROR)
@AccessLogAnnotation(ignoreRequestArgs = "loginUser")
public Result createTaskBindsWorkFlow(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
                                      @ApiParam(name = "projectCode", value = "PROJECT_CODE", required = true) @PathVariable long projectCode,
                                      @RequestParam(value = "processDefinitionCode", required = true) long processDefinitionCode,
                                      @RequestParam(value = "taskDefinitionJsonObj", required = true) String taskDefinitionJsonObj,
                                      @RequestParam(value = "upstreamCodes", required = false) String upstreamCodes) {

入参里面有taskDefinitionJsonObj，这里反序列化出来以后就是TaskDefinitionLog，里面包含作业类型
最终调用到ProcessServiceImpl的saveTaskDefine，在此处入库

if (CollectionUtils.isNotEmpty(newTaskDefinitionLogs)) {
    insertResult += taskDefinitionLogMapper.batchInsert(newTaskDefinitionLogs);
}
if (CollectionUtils.isNotEmpty(updateTaskDefinitionLogs)) {
    insertResult += taskDefinitionLogMapper.batchInsert(updateTaskDefinitionLogs);
}

if (CollectionUtils.isNotEmpty(newTaskDefinitionLogs) && Boolean.TRUE.equals(syncDefine)) {
    updateResult += taskDefinitionMapper.batchInsert(newTaskDefinitionLogs);
}
if (CollectionUtils.isNotEmpty(updateTaskDefinitionLogs) && Boolean.TRUE.equals(syncDefine)) {
    for (TaskDefinitionLog taskDefinitionLog : updateTaskDefinitionLogs) {
        updateResult += taskDefinitionMapper.updateById(taskDefinitionLog);
    }
}

1.2.1 工作流和任务

根据工作流DAG图，一个工作流可能拆分成多个任务，任务按依赖关系先后执行。在创建工作流之后，会同步创建对应的任务
当保存工作流，发送的请求是process-definition，其在最后会发送task-definition请求，由TaskDefinitionController处理

@PostMapping()
@ResponseStatus(HttpStatus.CREATED)
@ApiException(CREATE_TASK_DEFINITION_ERROR)
@AccessLogAnnotation(ignoreRequestArgs = "loginUser")
public Result createTaskDefinition(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
                                   @ApiParam(name = "projectCode", value = "PROJECT_CODE", required = true) @PathVariable long projectCode,
                                   @RequestParam(value = "taskDefinitionJson", required = true) String taskDefinitionJson) {
    Map<String, Object> result =
            taskDefinitionService.createTaskDefinition(loginUser, projectCode, taskDefinitionJson);
    return returnDataList(result);
}

接口之后调用到ProcessServiceImpl的saveTaskDefine，由TaskDefinitionMapper完成入库

if (CollectionUtils.isNotEmpty(newTaskDefinitionLogs) && Boolean.TRUE.equals(syncDefine)) {
    updateResult += taskDefinitionMapper.batchInsert(newTaskDefinitionLogs);
}

1.3 上线

点击上线触发的是ProcessDefinitionController的release接口

@PostMapping(value = "/{code}/release")
@ResponseStatus(HttpStatus.OK)
@ApiException(RELEASE_PROCESS_DEFINITION_ERROR)
@AccessLogAnnotation(ignoreRequestArgs = "loginUser")
public Result releaseProcessDefinition(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser,

2.立即触发流程

此处对应的就是点击工作流上的运行命令后的处理流程

2.1 API层操作

2.1.1 入口

ExecutorController的startProcessInstance，是RestAPI形式的接口

@PostMapping(value = "start-process-instance")
@ResponseStatus(HttpStatus.OK)
@ApiException(START_PROCESS_INSTANCE_ERROR)
@AccessLogAnnotation(ignoreRequestArgs = "loginUser")
public Result startProcessInstance(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser,

方法的核心就是调用ExecutorService的execProcessInstance接口

Map<String, Object> result = execService.execProcessInstance(loginUser, projectCode, processDefinitionCode,
        scheduleTime, execType, failureStrategy,
        startNodeList, taskDependType, warningType, warningGroupId, runMode, processInstancePriority,
        workerGroup, environmentCode, timeout, startParamMap, expectedParallelismNumber, dryRun,
        complementDependentMode);

ExecutorService目前有一个实现类ExecutorServiceImpl

2.1.2 入库

execProcessInstance中核心的步骤就是创建一个Command，并将其入元数据表

int create =
        this.createCommand(commandType, processDefinition.getCode(), taskDependType, failureStrategy,
                startNodeList,
                cronTime, warningType, loginUser.getId(), warningGroupId, runMode, processInstancePriority,
                workerGroup,
                environmentCode, startParams, expectedParallelismNumber, dryRun, complementDependentMode);

最终调用到ProcessServiceImpl的createCommand

// add command timezone
Schedule schedule = scheduleMapper.queryByProcessDefinitionCode(command.getProcessDefinitionCode());
if (schedule != null) {
    Map<String, String> commandParams = StringUtils.isNotBlank(command.getCommandParam()) ? JSONUtils.toMap(command.getCommandParam()) : new HashMap<>();
    commandParams.put(Constants.SCHEDULE_TIMEZONE, schedule.getTimezoneId());
    command.setCommandParam(JSONUtils.toJsonString(commandParams));
}
command.setId(null);
result = commandMapper.insert(command);
return result;

scheduleMapper、commandMapper都是dao层的操作，直接对数据库查询和插入

2.2 Master层操作

2.2.1 任务调度分配

API层将Command入库以后，接下来就是Master的操作，Master有一个MasterSchedulerBootstrap做任务调度，其run方法中是一个死循环，其中有一步就是获取前面入库的Commands

List<Command> commands = findCommands();
if (CollectionUtils.isEmpty(commands)) {
    // indicate that no command ,sleep for 1s
    Thread.sleep(Constants.SLEEP_TIME_MILLIS);
    continue;
}

最终走到CommandMapper，也就是SQL的Dao接口，查询出数据库中前面入库的command

public List<Command> findCommandPageBySlot(int pageSize, int pageNumber, int masterCount, int thisMasterSlot) {
    if (masterCount <= 0) {
        return Lists.newArrayList();
    }
    return commandMapper.queryCommandPageBySlot(pageSize, pageNumber * pageSize, masterCount, thisMasterSlot);
}

这边查询时有Slot参数，但并不是入库的Command已经分配了slot，而是查询时基于ID取模，对应的SQL语句如下。这里是dolphinscheduler对任务分配的一个均衡机制，dolphinscheduler虽然有Master、Work的分别，但不是一个中心化的组件，而是非中心化的，可以有多个Master同时工作

<select id="queryCommandPageBySlot" resultType="org.apache.dolphinscheduler.dao.entity.Command">
    select *
    from t_ds_command
    where id % #{masterCount} = #{thisMasterSlot}
    order by process_instance_priority, id asc
        limit #{limit} offset #{offset}
</select>

2.2.2 转换封装

这一步就是把从数据库获得的Command转换成后续执行需要的ProcessInstance，也是在MasterSchedulerBootstrap的run方法中，在取到command之后立刻进行

List<ProcessInstance> processInstances = command2ProcessInstance(commands);
if (CollectionUtils.isEmpty(processInstances)) {
    // indicate that the command transform to processInstance error, sleep for 1s
    Thread.sleep(Constants.SLEEP_TIME_MILLIS);
    continue;
}

转化操作由service模块的ProcessServiceImpl完成，具体就是一个提取参数构建对象的过程

ProcessInstance processInstance = processService.handleCommand(masterAddress, command);
if (processInstance != null) {
    processInstances.add(processInstance);
    logger.info("Master handle command {} end, create process instance {}", command.getId(),
            processInstance.getId());
}

2.2.3 发送执行请求

MasterSchedulerBootstrap的run方法中最后一步就是发送了一个启动工作流的请求

processInstances.forEach(processInstance -> {
    try {
        LoggerUtils.setWorkflowInstanceIdMDC(processInstance.getId());
        if (processInstanceExecCacheManager.contains(processInstance.getId())) {
            logger.error(
                    "The workflow instance is already been cached, this case shouldn't be happened");
        }
        WorkflowExecuteRunnable workflowRunnable = new WorkflowExecuteRunnable(processInstance,
                processService,
                processInstanceDao,
                nettyExecutorManager,
                processAlertManager,
                masterConfig,
                stateWheelExecuteThread,
                curingGlobalParamsService);
        processInstanceExecCacheManager.cache(processInstance.getId(), workflowRunnable);
        workflowEventQueue.addEvent(new WorkflowEvent(WorkflowEventType.START_WORKFLOW,
                processInstance.getId()));
    } finally {
        LoggerUtils.removeWorkflowInstanceIdMDC();
    }
});

2.2.4 处理执行请求

上一步在workflowEventQueue当中加入了一个START_WORKFLOW事件，Master有一个WorkflowEventLooper，同样死循环执行，其执行逻辑就是从workflowEventQueue当中取出事件处理

while (!ServerLifeCycleManager.isStopped()) {
    try {
        workflowEvent = workflowEventQueue.poolEvent();
        LoggerUtils.setWorkflowInstanceIdMDC(workflowEvent.getWorkflowInstanceId());
        logger.info("Workflow event looper receive a workflow event: {}, will handle this", workflowEvent);
        WorkflowEventHandler workflowEventHandler =
                workflowEventHandlerMap.get(workflowEvent.getWorkflowEventType());
        workflowEventHandler.handleWorkflowEvent(workflowEvent);

这里START_WORKFLOW对应的处理类为WorkflowStartEventHandler，这边处理了工作流的状态，变更为submit

public void handleWorkflowEvent(final WorkflowEvent workflowEvent) throws WorkflowEventHandleError {
    logger.info("Handle workflow start event, begin to start a workflow, event: {}", workflowEvent);
    WorkflowExecuteRunnable workflowExecuteRunnable = processInstanceExecCacheManager.getByProcessInstanceId(
        workflowEvent.getWorkflowInstanceId());
    if (workflowExecuteRunnable == null) {
        throw new WorkflowEventHandleError(
            "The workflow start event is invalid, cannot find the workflow instance from cache");
    }
    ProcessInstanceMetrics.incProcessInstanceByState("submit");
    ProcessInstance processInstance = workflowExecuteRunnable.getProcessInstance();
    CompletableFuture.supplyAsync(workflowExecuteRunnable::call, workflowExecuteThreadPool)
        .thenAccept(workflowSubmitStatue -> {
            if (WorkflowSubmitStatue.SUCCESS == workflowSubmitStatue) {

2.2.5 WorkflowExecuteRunnable状态轮转

2.2.4中处理的核心就是workflowExecuteRunnable::call，来到WorkflowExecuteRunnable的call方法

try {
    LoggerUtils.setWorkflowInstanceIdMDC(processInstance.getId());
    if (workflowRunnableStatus == WorkflowRunnableStatus.CREATED) {
        buildFlowDag();
        workflowRunnableStatus = WorkflowRunnableStatus.INITIALIZE_DAG;
        logger.info("workflowStatue changed to :{}", workflowRunnableStatus);
    }
    if (workflowRunnableStatus == WorkflowRunnableStatus.INITIALIZE_DAG) {
        initTaskQueue();
        workflowRunnableStatus = WorkflowRunnableStatus.INITIALIZE_QUEUE;
        logger.info("workflowStatue changed to :{}", workflowRunnableStatus);
    }
    if (workflowRunnableStatus == WorkflowRunnableStatus.INITIALIZE_QUEUE) {
        submitPostNode(null);
        workflowRunnableStatus = WorkflowRunnableStatus.STARTED;
        logger.info("workflowStatue changed to :{}", workflowRunnableStatus);
    }
    return WorkflowSubmitStatue.SUCCESS;

这里是一个分支处理，但实际启动一个工作流的过程就是分支的顺序处理过程，对应三个主要方法buildFlowDag、initTaskQueue、submitPostNode
buildFlowDag就是构建DAG图，主要是获取一些参数然后构建对象，这里很多参数信息都是通过从数据库获取的
initTaskQueue初始化任务队列，这个主要是做一些前置依赖等的处理
submitPostNode将任务提交到队列当中，这里首先把任务封装成TaskInstance，这里会基于dag的上下游关系，先提交前置节点，完成再提交下游节点
submitPostNode当中有工作流任务拆分的过程，在submitPostNode当中，解析DAG图，由于第一次进来parentNodeCode，所以这里只获取到第一个任务

Set<String> submitTaskNodeList =
        DagHelper.parsePostNodes(parentNodeCode, skipTaskNodeMap, dag, getCompleteTaskInstanceMap());

之后第一个任务按正常流程走完，由WorkflowExecuteRunnable的handleEvents处理结束事件，到TaskStateEventHandler当中，走回WorkflowExecuteRunnable的taskFinished，成功和失败都有submitPostNode的调用，这次的输入是结束的任务

if (taskInstance.getState().isSuccess()) {
    completeTaskMap.put(taskInstance.getTaskCode(), taskInstance.getId());
    // todo: merge the last taskInstance
    processInstance.setVarPool(taskInstance.getVarPool());
    processInstanceDao.upsertProcessInstance(processInstance);
    if (!processInstance.isBlocked()) {
        submitPostNode(Long.toString(taskInstance.getTaskCode()));
    }

submitPostNode当中按工作流设置的任务先后关系，顺序执行任务，有三个重要的方法：addTaskToStandByList、submitStandByTask、updateProcessInstanceState，提交的核心自然是中间的submitStandByTask。
addTaskToStandByList负责添加任务，这里处理了任务状态为submit

TaskMetrics.incTaskInstanceByState("submit");
readyToSubmitTaskQueue.put(taskInstance);

updateProcessInstanceState最终处理工作流的状态，发送一个状态变更事件，具体的状态依赖于submitStandByTask中的任务执行情况

WorkflowStateEvent stateEvent = WorkflowStateEvent.builder()
        .processInstanceId(processInstance.getId())
        .status(processInstance.getState())
        .type(StateEventType.PROCESS_STATE_CHANGE)
        .build();
// replace with `stateEvents`, make sure `WorkflowExecuteThread` can be deleted to avoid memory leaks
this.stateEvents.add(stateEvent);

2.2.6 Action-Submit

submitStandByTask的核心处理是submitTaskExec接口，里面有三个Action：SUBMIT、DISPATCH、RUN
submit的核心还是将任务入库

boolean submit = taskProcessor.action(TaskAction.SUBMIT);

实现类走到ProcessServiceImpl.submitTask，最终还是写入数据库

// submit to db
TaskInstance task = submitTaskInstanceToDB(taskInstance, processInstance);

这里分新建任务和更新任务

public boolean saveTaskInstance(TaskInstance taskInstance) {
    if (taskInstance.getId() != null) {
        return updateTaskInstance(taskInstance);
    } else {
        return createTaskInstance(taskInstance);
    }
}

实现还是基于TaskInstanceMapper，是Dao层，处理数据库

public boolean createTaskInstance(TaskInstance taskInstance) {
    int count = taskInstanceMapper.insert(taskInstance);
    return count > 0;
}

2.2.7 Action-DISPATCH

dispatch就是把任务下发到Work进行处理

dispatchTask接口核心就是把任务封装成TaskPriority然后放入TaskPriorityQueueImpl中的队列里，这个队列用的是jdk的PriorityBlockingQueue，是一个支持优先级的无界阻塞队列

TaskPriority taskPriority = new TaskPriority(processInstance.getProcessInstancePriority().getCode(),
        processInstance.getId(), taskInstance.getProcessInstancePriority().getCode(),
        taskInstance.getId(), taskInstance.getTaskGroupPriority(),
        Constants.DEFAULT_WORKER_GROUP);

TaskExecutionContext taskExecutionContext = getTaskExecutionContext(taskInstance);
if (taskExecutionContext == null) {
    logger.error("Get taskExecutionContext fail, task: {}", taskInstance);
    return false;
}

taskPriority.setTaskExecutionContext(taskExecutionContext);

taskUpdateQueue.put(taskPriority);
logger.info("Task {} is submitted to priority queue success by master", taskInstance.getName());

TaskPriorityQueueConsumer是Thread子类，由Spring管理，所以没有看到dolphin代码有对它的创建启动步骤，其运行就是thread的run接口，是一个死循环调用

TaskPriority taskPriority = taskPriorityQueue.poll(Constants.SLEEP_TIME_MILLIS, TimeUnit.MILLISECONDS);
if (Objects.isNull(taskPriority)) {
    latch.countDown();
    continue;
}

consumerThreadPoolExecutor.submit(() -> {
    try {
        boolean dispatchResult = this.dispatchTask(taskPriority);
        if (!dispatchResult) {
            failedDispatchTasks.add(taskPriority);
        }
    } finally {
        // make sure the latch countDown
        latch.countDown();
    }
});

之后调用dispatchTask，然后调用到ExecutorDispatcher的dispatch方法

executorManager.beforeExecute(context);
try {
    // task execute
    return executorManager.execute(context);
} finally {
    executorManager.afterExecute(context);
}

这里的executorManager是基于netty的实现NettyExecutorManager，在这里，执行的任务又还原到Command，但这个Command并不是最原始的那个Command

Command command = context.getCommand();
// execute task host
Host host = context.getHost();
boolean success = false;
while (!success) {
    try {
        doExecute(host, command);

doExecute这里就通过netty客户端将任务下发下去

do {
    try {
        nettyRemotingClient.send(host, command);

2.2.8 状态处理

这个是在submitTaskExec接口当中发送dispatch之后，加了两个事件监听，由StateWheelExecuteThread处理，其实就是定时的去获取作业状态

在添加事件监听之后，如果任务处理结束状态，会添加一个变更事件

除却这个之外，还有一个TaskExecuteResponseProcessor处理Worker完成任务以后发送的TASK_EXECUTE_RESULT事件

if (taskProcessor.taskInstance().getState().isFinished()) {
    
    if (processInstance.isBlocked()) {
        TaskStateEvent processBlockEvent = TaskStateEvent.builder()
                .processInstanceId(processInstance.getId())
                .taskInstanceId(taskInstance.getId())
                .status(taskProcessor.taskInstance().getState())
                .type(StateEventType.PROCESS_BLOCKED)
                .build();
        this.stateEvents.add(processBlockEvent);
    }
    TaskStateEvent taskStateChangeEvent = TaskStateEvent.builder()
            .processInstanceId(processInstance.getId())
            .taskInstanceId(taskInstance.getId())
            .status(taskProcessor.taskInstance().getState())
            .type(StateEventType.TASK_STATE_CHANGE)
            .build();
    this.stateEvents.add(taskStateChangeEvent);
}

StateWheelExecuteThread对监听事件的处理如下，就是定时去获取状态

public void run() {
    final long checkInterval = masterConfig.getStateWheelInterval().toMillis();
    while (!ServerLifeCycleManager.isStopped()) {
        try {
            checkTask4Timeout();
            checkTask4Retry();
            checkTask4State();
            checkProcess4Timeout();
        } catch (Exception e) {
            logger.error("state wheel thread check error:", e);
        }
        try {
            Thread.sleep(checkInterval);
        } catch (InterruptedException e) {
            logger.error("state wheel thread sleep error, will close the loop", e);
            Thread.currentThread().interrupt();
            break;
        }
    }
}

checkTask4State当中，如果不是Finished状态，就会发送一个状态变更事件

private void addTaskStateChangeEvent(TaskInstance taskInstance) {
    TaskStateEvent stateEvent = TaskStateEvent.builder()
            .processInstanceId(taskInstance.getProcessInstanceId())
            .taskInstanceId(taskInstance.getId())
            .taskCode(taskInstance.getTaskCode())
            .type(StateEventType.TASK_STATE_CHANGE)
            .status(TaskExecutionStatus.RUNNING_EXECUTION)
            .build();
    workflowExecuteThreadPool.submitStateEvent(stateEvent);
}

2.3 Worker处理

Worker处理是在接受到Dispatch请求之后的

2.3.1 处理类

Worker启动的时候注册了各种netty请求的处理类，DISPATCH对应的是TaskDispatchProcessor

NettyServerConfig serverConfig = new NettyServerConfig();
serverConfig.setListenPort(workerConfig.getListenPort());
this.nettyRemotingServer = new NettyRemotingServer(serverConfig);
this.nettyRemotingServer.registerProcessor(CommandType.TASK_DISPATCH_REQUEST, taskDispatchProcessor);
this.nettyRemotingServer.registerProcessor(CommandType.TASK_KILL_REQUEST, taskKillProcessor);

2.3.2 入workerManager

TaskDispatchProcessor的处理接口process当中，核心就是把任务封装成一个WorkerDelayTaskExecuteRunnable，然后提交到workerManager

WorkerDelayTaskExecuteRunnable workerTaskExecuteRunnable = WorkerTaskExecuteRunnableFactoryBuilder
        .createWorkerDelayTaskExecuteRunnableFactory(
                taskExecutionContext,
                workerConfig,
                workflowMasterAddress,
                workerMessageSender,
                alertClientService,
                taskPluginManager,
                storageOperate)
        .createWorkerTaskExecuteRunnable();
// submit task to manager
boolean offer = workerManager.offer(workerTaskExecuteRunnable);

2.3.3 任务提交

workerManager，也就是WorkerManagerThread，本身就是一个线程，其run方法也是死循环地从上述的队列中去取任务提交

public void run() {
    Thread.currentThread().setName("Worker-Execute-Manager-Thread");
    while (!ServerLifeCycleManager.isStopped()) {
        try {
            if (!ServerLifeCycleManager.isRunning()) {
                Thread.sleep(Constants.SLEEP_TIME_MILLIS);
            }
            if (this.getThreadPoolQueueSize() <= workerExecThreads) {
                final WorkerDelayTaskExecuteRunnable workerDelayTaskExecuteRunnable = waitSubmitQueue.take();
                workerExecService.submit(workerDelayTaskExecuteRunnable);

2.3.4 任务执行

WorkerTaskExecuteRunnable本身就是一个Runnable，所以上述任务提交以后，其实就是入了线程池，等待分配线程执行
任务执行实际就是WorkerTaskExecuteRunnable的run方法，最终执行其实就是调用的AbstractTask的实现

public void executeTask(TaskCallBack taskCallBack) throws TaskException {
    if (task == null) {
        throw new TaskException("The task plugin instance is not initialized");
    }
    task.handle(taskCallBack);
}

这里注意，flink、spark的实现都是AbstractYarnTask的子类(但是构建命令的时候还是有分支构建了非Yarn的任务命令模式)

public class FlinkTask extends AbstractYarnTask {

public class SparkTask extends AbstractYarnTask {

2.3.5 发送请求

作业执行完成，会发送一个TASK_EXECUTE_RESULT事件，这个事件最终由Master的TaskExecuteResponseProcessor进行后续相应处理

protected void sendTaskResult() {
    taskExecutionContext.setCurrentExecutionStatus(task.getExitStatus());
    taskExecutionContext.setEndTime(new Date());
    taskExecutionContext.setProcessId(task.getProcessId());
    taskExecutionContext.setAppIds(task.getAppIds());
    taskExecutionContext.setVarPool(JSONUtils.toJsonString(task.getParameters().getVarPool()));
    workerMessageSender.sendMessageWithRetry(taskExecutionContext, masterAddress, CommandType.TASK_EXECUTE_RESULT);

    logger.info("Send task execute result to master, the current task status: {}",
            taskExecutionContext.getCurrentExecutionStatus());
}

不甚了然

关注

0
点赞
踩
2

收藏

觉得还不错? 一键收藏
0
评论
DolphinScheduler作业流程

之后第一个任务按正常流程走完，由WorkflowExecuteRunnable的handleEvents处理结束事件，到TaskStateEventHandler当中，走回WorkflowExecuteRunnable的taskFinished，成功和失败都有submitPostNode的调用，这次的输入是结束的任务。submitPostNode当中有工作流任务拆分的过程，在submitPostNode当中，解析DAG图，由于第一次进来parentNodeCode，所以这里只获取到第一个任务。
复制链接

扫一扫

专栏目录