Apache DolphinScheduler3.0源码剖析

已于 2022-09-12 17:43:41 修改
阅读量960 收藏 5
点赞数 3
分类专栏： 代码阅读 文章标签： 源码剖析
于 2022-09-11 21:51:43 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/IT_Fly/article/details/126809808
版权

Apache DolphinScheduler3.0源码剖析

1.工作流操作

1.1概述

用户通过编辑器编辑项目，工作流等信息，保存并运行的工作流的执行过程

1.2 创建工作流

1.2.1 第一步 执行ProcessDefinitionController的createProcessDefinition函数

 @PostMapping()
    @ResponseStatus(HttpStatus.CREATED)
    @ApiException(CREATE_PROCESS_DEFINITION_ERROR)
    @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
    public Result createProcessDefinition(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
                                          @ApiParam(name = "projectCode", value = "PROJECT_CODE", required = true) @PathVariable long projectCode,
                                          @RequestParam(value = "name", required = true) String name,
                                          @RequestParam(value = "description", required = false) String description,
                                          @RequestParam(value = "globalParams", required = false, defaultValue = "[]") String globalParams,
                                          @RequestParam(value = "locations", required = false) String locations,
                                          @RequestParam(value = "timeout", required = false, defaultValue = "0") int timeout,
                                          @RequestParam(value = "tenantCode", required = true) String tenantCode,
                                          @RequestParam(value = "taskRelationJson", required = true) String taskRelationJson,
                                          @RequestParam(value = "taskDefinitionJson", required = true) String taskDefinitionJson,
                                          @RequestParam(value = "otherParamsJson", required = false) String otherParamsJson,
                                          @RequestParam(value = "executionType", defaultValue = "PARALLEL") ProcessExecutionTypeEnum executionType) {
        Map<String, Object> result = processDefinitionService.createProcessDefinition(loginUser, projectCode, name,
                description, globalParams,
                locations, timeout, tenantCode, taskRelationJson, taskDefinitionJson, otherParamsJson, executionType);
        return returnDataList(result);
    }

1.2.2 第二步 执行ProcessDefinitionServiceImpl的createProcessDefinition函数

/**
     * create process definition
     *
     * @param loginUser login user
     * @param projectCode project code
     * @param name process definition name
     * @param description description
     * @param globalParams global params
     * @param locations locations for nodes
     * @param timeout timeout
     * @param tenantCode tenantCode
     * @param taskRelationJson relation json for nodes
     * @param taskDefinitionJson taskDefinitionJson
     * @return create result code
     */
    @Override
    @Transactional
    public Map<String, Object> createProcessDefinition(User loginUser,
                                                       long projectCode,
                                                       String name,
                                                       String description,
                                                       String globalParams,
                                                       String locations,
                                                       int timeout,
                                                       String tenantCode,
                                                       String taskRelationJson,
                                                       String taskDefinitionJson,
                                                       String otherParamsJson,
                                                       ProcessExecutionTypeEnum executionType) {
        Project project = projectMapper.queryByCode(projectCode);
        // 判断用户是否有操作此项目权限
        Map<String, Object> result =
                projectService.checkProjectAndAuth(loginUser, project, projectCode, WORKFLOW_CREATE);
        if (result.get(Constants.STATUS) != Status.SUCCESS) {
            return result;
        }
        if(checkDescriptionLength(description)){
            putMsg(result, Status.DESCRIPTION_TOO_LONG_ERROR);
            return result;
        }
        // 判断新的项目名称是否已存在
        ProcessDefinition definition = processDefinitionMapper.verifyByDefineName(project.getCode(), name);
        if (definition != null) {
            putMsg(result, Status.PROCESS_DEFINITION_NAME_EXIST, name);
            return result;
        }
        List<TaskDefinitionLog> taskDefinitionLogs = JSONUtils.toList(taskDefinitionJson, TaskDefinitionLog.class);
        Map<String, Object> checkTaskDefinitions = checkTaskDefinitionList(taskDefinitionLogs, taskDefinitionJson);
        if (checkTaskDefinitions.get(Constants.STATUS) != Status.SUCCESS) {
            return checkTaskDefinitions;
        }
        List<ProcessTaskRelationLog> taskRelationList =
                JSONUtils.toList(taskRelationJson, ProcessTaskRelationLog.class);
        Map<String, Object> checkRelationJson =
                checkTaskRelationList(taskRelationList, taskRelationJson, taskDefinitionLogs);
        if (checkRelationJson.get(Constants.STATUS) != Status.SUCCESS) {
            return checkRelationJson;
        }
        // 判断用户是否绑定了租户，未绑定则无法执行
        int tenantId = -1;
        if (!Constants.DEFAULT.equals(tenantCode)) {
            Tenant tenant = tenantMapper.queryByTenantCode(tenantCode);
            if (tenant == null) {
                putMsg(result, Status.TENANT_NOT_EXIST);
                return result;
            }
            tenantId = tenant.getId();
        }
        // 未项目定义生成一个编码
        long processDefinitionCode;
        try {
            processDefinitionCode = CodeGenerateUtils.getInstance().genCode();
        } catch (CodeGenerateException e) {
            putMsg(result, Status.INTERNAL_SERVER_ERROR_ARGS);
            return result;
        }
        ProcessDefinition processDefinition =
                new ProcessDefinition(projectCode, name, processDefinitionCode, description,
                        globalParams, locations, timeout, loginUser.getId(), tenantId);
        processDefinition.setExecutionType(executionType);
		// 创建项目的有向无环图
        return createDagDefine(loginUser, taskRelationList, processDefinition, taskDefinitionLogs, otherParamsJson);
    }

1.2.3 第三步 执行ProcessDefinitionServiceImpl的createDagDefine函数

protected Map<String, Object> createDagDefine(User loginUser,
                                                  List<ProcessTaskRelationLog> taskRelationList,
                                                  ProcessDefinition processDefinition,
                                                  List<TaskDefinitionLog> taskDefinitionLogs, String otherParamsJson) {
        Map<String, Object> result = new HashMap<>();
        int saveTaskResult = processService.saveTaskDefine(loginUser, processDefinition.getProjectCode(),
                taskDefinitionLogs, Boolean.TRUE);
        if (saveTaskResult == Constants.EXIT_CODE_SUCCESS) {
            logger.info("The task has not changed, so skip");
        }
        if (saveTaskResult == Constants.DEFINITION_FAILURE) {
            putMsg(result, Status.CREATE_TASK_DEFINITION_ERROR);
            throw new ServiceException(Status.CREATE_TASK_DEFINITION_ERROR);
        }
        // 保存项目定义到数据库中
        int insertVersion = processService.saveProcessDefine(loginUser, processDefinition, Boolean.TRUE, Boolean.TRUE);
        if (insertVersion == 0) {
            putMsg(result, Status.CREATE_PROCESS_DEFINITION_ERROR);
            throw new ServiceException(Status.CREATE_PROCESS_DEFINITION_ERROR);
        }
        int insertResult = processService.saveTaskRelation(loginUser, processDefinition.getProjectCode(),
                processDefinition.getCode(),
                insertVersion, taskRelationList, taskDefinitionLogs, Boolean.TRUE);
        if (insertResult == Constants.EXIT_CODE_SUCCESS) {
            putMsg(result, Status.SUCCESS);
            result.put(Constants.DATA_LIST, processDefinition);
        } else {
            putMsg(result, Status.CREATE_PROCESS_TASK_RELATION_ERROR);
            throw new ServiceException(Status.CREATE_PROCESS_TASK_RELATION_ERROR);
        }
        saveOtherRelation(loginUser, processDefinition, result, otherParamsJson);
        return result;
    }

1.2.4整体执行流程时序图

1.3运行工作流

在这里插入图片描述

2.Master相关流程

2.1概述

Master运行线程，检索出需要执行的命令，经过一系列处理后通过网络发送给Worker执行

2.2 Master类的run函数

    /**
     * run master server
     */
    @PostConstruct
    public void run() throws SchedulerException {
        // 初始化masterRPCServer线程，启动一个netty server进行监听
        this.masterRPCServer.start();

        // 通过SPI机制加载功能模块插件
        this.taskPluginManager.loadPlugin();

        // self tolerant
        this.masterRegistryClient.start();
        this.masterRegistryClient.setRegistryStoppable(this);

		// 初始化并运行masterSchedulerBootstrap线程，从数据库中读取要执行的Command并转换成ProcessInstance进行触发
        this.masterSchedulerBootstrap.init();
        this.masterSchedulerBootstrap.start();

        this.eventExecuteService.start();
        this.failoverExecuteThread.start();
		
		// Quartz定时器启动，实现类QuartzScheduler
        this.schedulerApi.start();

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            if (!ServerLifeCycleManager.isStopped()) {
                close("MasterServer shutdownHook");
            }
        }));
    }

在这里插入图片描述

2.3 MasterRPCServer类的start函数

public void start() {
        logger.info("Starting Master RPC Server...");
        // init remoting server
        NettyServerConfig serverConfig = new NettyServerConfig();
        serverConfig.setListenPort(masterConfig.getListenPort());
        this.nettyRemotingServer = new NettyRemotingServer(serverConfig);
        // 
        this.nettyRemotingServer.registerProcessor(CommandType.TASK_EXECUTE_RUNNING, taskExecuteRunningProcessor);
        this.nettyRemotingServer.registerProcessor(CommandType.TASK_EXECUTE_RESULT, taskExecuteResponseProcessor);
        this.nettyRemotingServer.registerProcessor(CommandType.TASK_KILL_RESPONSE, taskKillResponseProcessor);
        this.nettyRemotingServer.registerProcessor(CommandType.STATE_EVENT_REQUEST, stateEventProcessor);
        this.nettyRemotingServer.registerProcessor(CommandType.TASK_FORCE_STATE_EVENT_REQUEST, taskEventProcessor);
        this.nettyRemotingServer.registerProcessor(CommandType.TASK_WAKEUP_EVENT_REQUEST, taskEventProcessor);
        this.nettyRemotingServer.registerProcessor(CommandType.CACHE_EXPIRE, cacheProcessor);
        this.nettyRemotingServer.registerProcessor(CommandType.TASK_REJECT, taskRecallProcessor);
        this.nettyRemotingServer.registerProcessor(CommandType.WORKFLOW_EXECUTING_DATA_REQUEST, workflowExecutingDataRequestProcessor);
        this.nettyRemotingServer.registerProcessor(CommandType.TASK_EXECUTE_START, taskExecuteStartProcessor);

        // logger server
        this.nettyRemotingServer.registerProcessor(CommandType.GET_LOG_BYTES_REQUEST, loggerRequestProcessor);
        this.nettyRemotingServer.registerProcessor(CommandType.ROLL_VIEW_LOG_REQUEST, loggerRequestProcessor);
        this.nettyRemotingServer.registerProcessor(CommandType.VIEW_WHOLE_LOG_REQUEST, loggerRequestProcessor);
        this.nettyRemotingServer.registerProcessor(CommandType.REMOVE_TAK_LOG_REQUEST, loggerRequestProcessor);

        this.nettyRemotingServer.start();
        logger.info("Started Master RPC Server...");
    }

在这里插入图片描述

2.4 TaskPluginManager类的loadPlugin函数

public void loadPlugin() {
        if (!loadedFlag.compareAndSet(false, true)) {
            logger.warn("The task plugin has already been loaded");
            return;
        }
        PrioritySPIFactory<TaskChannelFactory> prioritySPIFactory = new PrioritySPIFactory<>(TaskChannelFactory.class);
        for (Map.Entry<String, TaskChannelFactory> entry : prioritySPIFactory.getSPIMap().entrySet()) {
            String factoryName = entry.getKey();
            TaskChannelFactory factory = entry.getValue();

            logger.info("Registering task plugin: {} - {}", factoryName, factory.getClass());

            taskChannelFactoryMap.put(factoryName, factory);
            taskChannelMap.put(factoryName, factory.create());

            logger.info("Registered task plugin: {} - {}", factoryName, factory.getClass());
        }

    }

2.5 MasterSchedulerBootstrap执行过程分析

/**
     * run of MasterSchedulerService
     */
    @Override
    public void run() {
        while (!ServerLifeCycleManager.isStopped()) {
            try {
                if (!ServerLifeCycleManager.isRunning()) {
                    // the current server is not at running status, cannot consume command.
                    Thread.sleep(Constants.SLEEP_TIME_MILLIS);
                }
                // todo: if the workflow event queue is much, we need to handle the back pressure
                boolean isOverload =
                    OSUtils.isOverload(masterConfig.getMaxCpuLoadAvg(), masterConfig.getReservedMemory());
                if (isOverload) {
                    MasterServerMetrics.incMasterOverload();
                    Thread.sleep(Constants.SLEEP_TIME_MILLIS);
                    continue;
                }
                // 从数据库中查询当前Master所要运行的Command
                List<Command> commands = findCommands();
                if (CollectionUtils.isEmpty(commands)) {
                    // indicate that no command ,sleep for 1s
                    Thread.sleep(Constants.SLEEP_TIME_MILLIS);
                    continue;
                }
				 // 将Command转换成进程实例
                List<ProcessInstance> processInstances = command2ProcessInstance(commands);
                if (CollectionUtils.isEmpty(processInstances)) {
                    // indicate that the command transform to processInstance error, sleep for 1s
                    Thread.sleep(Constants.SLEEP_TIME_MILLIS);
                    continue;
                }
                MasterServerMetrics.incMasterConsumeCommand(commands.size());
				// 遍历处理所有的进程实例
                processInstances.forEach(processInstance -> {
                    try {
                        LoggerUtils.setWorkflowInstanceIdMDC(processInstance.getId());
                        if (processInstanceExecCacheManager.contains(processInstance.getId())) {
                            logger.error(
                                "The workflow instance is already been cached, this case shouldn't be happened");
                        }
                        WorkflowExecuteRunnable workflowRunnable = new WorkflowExecuteRunnable(processInstance,
                            processService,
                            processInstanceDao,
                            nettyExecutorManager,
                            processAlertManager,
                            masterConfig,
                            stateWheelExecuteThread,
                            curingGlobalParamsService);
                        // 根据processInstance 创建workflowRunnable对象，并缓存到processInstanceExecCacheManager中
                        processInstanceExecCacheManager.cache(processInstance.getId(), workflowRunnable);
                        workflowEventQueue.addEvent(new WorkflowEvent(WorkflowEventType.START_WORKFLOW,
                            processInstance.getId()));
                    } finally {
                        LoggerUtils.removeWorkflowInstanceIdMDC();
                    }
                });
            } catch (InterruptedException interruptedException) {
                logger.warn("Master schedule bootstrap interrupted, close the loop", interruptedException);
                Thread.currentThread().interrupt();
                break;
            } catch (Exception e) {
                logger.error("Master schedule workflow error", e);
                // sleep for 1s here to avoid the database down cause the exception boom
                ThreadUtils.sleep(Constants.SLEEP_TIME_MILLIS);
            }
        }
    }

在这里插入图片描述

2.6 TaskPriorityQueueConsumer执行

TaskPriorityQueueConsumer线程处理上面流程中存储到processInstanceExecCacheManager中的workflowExecuteRunnable ，通过netty发送到worker执行

@Override
    public void run() {
        int fetchTaskNum = masterConfig.getDispatchTaskNumber();
        while (!ServerLifeCycleManager.isStopped()) {
            try {
                List<TaskPriority> failedDispatchTasks = this.batchDispatch(fetchTaskNum);

                if (CollectionUtils.isNotEmpty(failedDispatchTasks)) {
                    TaskMetrics.incTaskDispatchFailed(failedDispatchTasks.size());
                    for (TaskPriority dispatchFailedTask : failedDispatchTasks) {
                        taskPriorityQueue.put(dispatchFailedTask);
                    }
                    // If the all task dispatch failed, will sleep for 1s to avoid the master cpu higher.
                    if (fetchTaskNum == failedDispatchTasks.size()) {
                        TimeUnit.MILLISECONDS.sleep(Constants.SLEEP_TIME_MILLIS);
                    }
                }
            } catch (Exception e) {
                TaskMetrics.incTaskDispatchError();
                logger.error("dispatcher task error", e);
            }
        }
    }

/**
     * batch dispatch with thread pool
     */
    public List<TaskPriority> batchDispatch(int fetchTaskNum) throws TaskPriorityQueueException, InterruptedException {
        List<TaskPriority> failedDispatchTasks = Collections.synchronizedList(new ArrayList<>());
        CountDownLatch latch = new CountDownLatch(fetchTaskNum);

        for (int i = 0; i < fetchTaskNum; i++) {
            TaskPriority taskPriority = taskPriorityQueue.poll(Constants.SLEEP_TIME_MILLIS, TimeUnit.MILLISECONDS);
            if (Objects.isNull(taskPriority)) {
                latch.countDown();
                continue;
            }

            consumerThreadPoolExecutor.submit(() -> {
                try {
                    boolean dispatchResult = this.dispatchTask(taskPriority);
                    if (!dispatchResult) {
                        failedDispatchTasks.add(taskPriority);
                    }
                } finally {
                    // make sure the latch countDown
                    latch.countDown();
                }
            });
        }

        latch.await();

        return failedDispatchTasks;
    }

/**
     * Dispatch task to worker.
     *
     * @param taskPriority taskPriority
     * @return dispatch result, return true if dispatch success, return false if dispatch failed.
     */
    protected boolean dispatchTask(TaskPriority taskPriority) {
        TaskMetrics.incTaskDispatch();
        boolean result = false;
        try {
            WorkflowExecuteRunnable workflowExecuteRunnable =
                    processInstanceExecCacheManager.getByProcessInstanceId(taskPriority.getProcessInstanceId());
            if (workflowExecuteRunnable == null) {
                logger.error("Cannot find the related processInstance of the task, taskPriority: {}", taskPriority);
                return true;
            }
            Optional<TaskInstance> taskInstanceOptional =
                    workflowExecuteRunnable.getTaskInstance(taskPriority.getTaskId());
            if (!taskInstanceOptional.isPresent()) {
                logger.error("Cannot find the task instance from related processInstance, taskPriority: {}",
                        taskPriority);
                // we return true, so that we will drop this task.
                return true;
            }
            TaskInstance taskInstance = taskInstanceOptional.get();
            TaskExecutionContext context = taskPriority.getTaskExecutionContext();
            ExecutionContext executionContext =
                    new ExecutionContext(toCommand(context), ExecutorType.WORKER, context.getWorkerGroup(),
                            taskInstance);

            if (isTaskNeedToCheck(taskPriority)) {
                if (taskInstanceIsFinalState(taskPriority.getTaskId())) {
                    // when task finish, ignore this task, there is no need to dispatch anymore
                    return true;
                }
            }

            result = dispatcher.dispatch(executionContext);

            if (result) {
                logger.info("Master success dispatch task to worker, taskInstanceId: {}, worker: {}",
                        taskPriority.getTaskId(),
                        executionContext.getHost());
                addDispatchEvent(context, executionContext);
            } else {
                logger.info("Master failed to dispatch task to worker, taskInstanceId: {}, worker: {}",
                        taskPriority.getTaskId(),
                        executionContext.getHost());
            }
        } catch (RuntimeException | ExecuteException e) {
            logger.error("Master dispatch task to worker error, taskPriority: {}", taskPriority, e);
        }
        return result;
    }

/**
     * task dispatch
     *
     * @param context context
     * @return result
     * @throws ExecuteException if error throws ExecuteException
     */
    public Boolean dispatch(final ExecutionContext context) throws ExecuteException {
        // get executor manager
        ExecutorManager<Boolean> executorManager = this.executorManagers.get(context.getExecutorType());
        if (executorManager == null) {
            throw new ExecuteException("no ExecutorManager for type : " + context.getExecutorType());
        }

        // host select
        Host host = hostManager.select(context);
        if (StringUtils.isEmpty(host.getAddress())) {
            logger.warn("fail to execute : {} due to no suitable worker, current task needs worker group {} to execute",
                context.getCommand(), context.getWorkerGroup());
            return false;
        }
        context.setHost(host);
        executorManager.beforeExecute(context);
        try {
            // task execute
            return executorManager.execute(context);
        } finally {
            executorManager.afterExecute(context);
        }
    }

在这里插入图片描述

2.7 NettyExecutorManager运行时序图

/**
     * execute logic
     *
     * @param context context
     * @return result
     * @throws ExecuteException if error throws ExecuteException
     */
    @Override
    public Boolean execute(ExecutionContext context) throws ExecuteException {
        // all nodes
        Set<String> allNodes = getAllNodes(context);
        // fail nodes
        Set<String> failNodeSet = new HashSet<>();
        // build command accord executeContext
        Command command = context.getCommand();
        // execute task host
        Host host = context.getHost();
        boolean success = false;
        while (!success) {
            try {
                doExecute(host, command);
                success = true;
                context.setHost(host);
                // We set the host to taskInstance to avoid when the worker down, this taskInstance may not be
                // failovered, due to the taskInstance's host
                // is not belongs to the down worker ISSUE-10842.
                context.getTaskInstance().setHost(host.getAddress());
            } catch (ExecuteException ex) {
                logger.error(String.format("execute command : %s error", command), ex);
                try {
                    failNodeSet.add(host.getAddress());
                    Set<String> tmpAllIps = new HashSet<>(allNodes);
                    Collection<String> remained = CollectionUtils.subtract(tmpAllIps, failNodeSet);
                    if (remained != null && remained.size() > 0) {
                        host = Host.of(remained.iterator().next());
                        logger.error("retry execute command : {} host : {}", command, host);
                    } else {
                        throw new ExecuteException("fail after try all nodes");
                    }
                } catch (Throwable t) {
                    throw new ExecuteException("fail after try all nodes");
                }
            }
        }

        return success;
    }

Netty执行发送任务时序图：
在这里插入图片描述

3.Worker相关流程

3.1 WorkerServer运行

在这里插入图片描述

3.2 WorkerRpcServer运行

在这里插入图片描述

3.3 TaskDispatchProcessor运行

在这里插入图片描述

3.4 WorkerManagerThread运行

在这里插入图片描述

3.4 WorkerTaskExecuteRunnable运行

在这里插入图片描述

4.Task执行代码剖析

4.1 HttpTask运行

在这里插入图片描述

4.2 ShellTask运行

在这里插入图片描述

梧桐168
关注
  • 3
    点赞
  • 踩
  • 5
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
DolphinScheduler 3.1.9 启动、任务执行过程 源码 解析
TNT_D的博客
07-02 980
Apache DolphinScheduler 是一个分布式易扩展的可视化DAG工作流任务调度开源系统。适用于企业级场景，提供了一个可视化操作任务、工作流和全生命周期数据处理过程的解决方案。旨在解决复杂的大数据任务依赖关系，并为应用程序提供数据和各种 OPS 编排中的关系。解决数据研发ETL依赖错综复杂，无法监控任务健康状态的问题。
DolphinScheduler（apache-dolphinscheduler-3.0.0-beta-1-src.tar.gz
06-23
Apache DolphinScheduler（apache-dolphinscheduler-3.0.0-beta-1-src.tar.gz 源码）是一个分布式易扩展的可视化DAG工作流任务调度开源系统。解决数据研发ETL 错综复杂的依赖关系，不能直观监控任务健康状态等问题。DolphinScheduler以DAG流式的方式将Task组装起来，可实时监控任务的运行状态，同时支持重试、从指定节点恢复失败、暂停及Kill任务等操作。
Dolphinscheduler3.0源码分析
qq_26777585的博客
08-05 1793
研究Dolphinscheduler也是机缘巧合，平时负责基于xxl-job二次开发出来的调度平台，因为遇到了并发性能瓶颈，到了不得不优化重构的地步，所以搜索市面上应用较广的调度平台以借鉴优化思路。在阅读完DolphinScheduler代码之后，便生出了将其设计与思考记录下来的念头，这边是此篇文章的来源。因为没有正式生产使用，业务理解不一定透彻，理解可能有偏差，欢迎大家交流讨论。......
DolphinScheduler源码分析
Apache DolphinScheduler开源社区
09-10 983
本文是基于1.2.0版本进行分析，与最新版本的实现有一些出入，还请读者辩证的看待本源码分析。具体细节可能描述的不是很准确，仅供参考。1.源码版本DolphinScheduler-...
一文读懂，硬核 Apache DolphinScheduler3.0 源码解析
微信搜：import_bigdata，大数据领域硬核原创作者
10-26 1157
全网最全大数据面试提升手册！本文目录DolphinScheduler的设计与策略1.1 分布式设计1.1.1 中心化1.1.2 去中心化1.2 DophinScheduler架构设计1.3 容错问题1.3.1 宕机容错1.3.2 失败重试1.4 远程日志访问DolphinScheduler源码分析2.1 工程模块介绍与配置文件2.1.1 工程模块介绍2.1.2 配置文件2.2 Api主要任务操作接...
海豚调度Dolphinscheduler源码分析（三）
Mrerlou的博客
09-27 1992
今天继续分析海豚调度的源码 上回分析的是dolphinscheduler-service模块zookeeper相关的代码 这回分析是dolphinscheduler-server模块zookeeper相关的代码 ZkMasterClient master服务zk客户端类 类继承的关系如下： 这个类的方法如下： 方法介绍： start()根据路径dolphinscheduler/...
DolphinScheduler（apache-dolphinscheduler-3.0.0-beta-1-bin.tar.gz
06-23
Apache DolphinScheduler（apache-dolphinscheduler-3.0.0-beta-1-bin.tar.gz）是一个分布式易扩展的可视化DAG工作流任务调度开源系统。解决数据研发ETL 错综复杂的依赖关系，不能直观监控任务健康状态等问题。...
DolphinScheduler（apache-dolphinscheduler-2.0.5-src.tar.gz）源码
06-23
Apache DolphinScheduler（apache-dolphinscheduler-2.0.5-src.tar.gz 源码）是一个分布式易扩展的可视化DAG工作流任务调度开源系统。解决数据研发ETL 错综复杂的依赖关系，不能直观监控任务健康状态等问题。Dolphin...
apache-dolphinscheduler-3.1.4-bin
最新发布
02-20
Apache DolphinScheduler是一款强大的分布式工作流管理系统，用于大数据任务的工作流编排和调度。它提供了图形化的工作流设计界面，使得用户可以方便地构建复杂的任务流程。在这个“apache-dolphinscheduler-3.1.4-...
apache-dolphinscheduler-3.1.3-bin.tar.gz
01-10
Apache DolphinScheduler 是一个分布式易扩展的可视化DAG工作流任务调度开源系统。适用于企业级场景，提供了一个可视化操作任务、工作流和全生命周期数据处理过程的解决方案。 Apache DolphinScheduler 旨在解决...
Dolphinscheduler3.0源码分析之XxlJob优化之路
qq_26777585的博客
08-23 1356
研究Dolphinscheduler也是机缘巧合，平时负责基于xxl-job二次开发出来的调度平台，因为遇到了并发性能瓶颈，到了不得不优化重构的地步，所以搜索市面上应用较广的调度平台以借鉴优化思路。在阅读完DolphinScheduler代码之后，便生出了将其设计与思考记录下来的念头，这边是此篇文章的来源。因为没有正式生产使用，业务理解不一定透彻，理解可能有偏差，欢迎大家交流讨论。
海豚调度DolphinScheduler源码分析（一）
Mrerlou的博客
09-25 5925
系统架构设计 在对调度系统架构说明之前，我们先来认识一下调度系统常用的名词 1.名词解释 DAG：全称Directed Acyclic Graph，简称DAG。工作流中的Task任务以有向无环图的形式组装起来，从入度为零的节点进行拓扑遍历，直到无后继节点为止。举例如下图： dag示例 流程定义：通过拖拽任务节点并建立任务节点的关联所形成的可视化DAG 流程实例：流程实例是流程定义的实...
Dolphinscheduler海豚调度源码解析
MyAzhe0ci3的博客
03-24 945
分三块：数据源controller层 org/apache/dolphinscheduler/api/controller/DataSourceController.java 数据源impl层 org.apache.dolphinscheduler.api.service.impl.DataSourceServiceImpl 数据源dao层 org.apache.dolphinscheduler.dao.mapper
DolphinScheduler教程（03）- 源码分析
阿甘兄
07-05 2791
Apache DolphinScheduler是一个分布式去中心化，易扩展的可视化DAG工作流任务调度平台。致力于解决数据处理流程中错综复杂的依赖关系，使调度系统在数据处理流程中开箱即用。其原理图如下： 接下来，本文一步一步详细地讲解其源码。首先使用从项目到本地，项目地址：https://github.com/apache/dolphinscheduler 导入项目后，可以看到其主要核心模块如下：在项目目录下，有数据库初始化脚本dolphinscheduler_ddl.sql及dolphinschedule
Apache DolphinScheduler 源码解析-任务生命周期（1）
qq_32613771的博客
03-18 1494
ApacheDolphinScheduler 是一个分布式去中心化，易扩展的可视化 DAG 工作流任务调度平台。本篇对 Apache DolphinScheduler 调度平台从源码上解析任务从创建，构建，分发，执行，结束五个阶段的处理，涉及 Api-Server，Master-Server，Worker-Server三个服务。 ApacheDolphinScheduler 官网: https://dolphinscheduler.apache.org/ github: https://g...
收藏假期干货：Apache DolphinScheduler源码分析系列（超详细）
Apache DolphinScheduler开源社区
09-29 4089
01引言Apache DolphinScheduler官方文档地址：https://dolphinscheduler.apache.org/zh-cn/index.htmlApache DolphinScheduler是一个分布式去中心化，易扩展的可视化DAG工作流任务调度平台。致力于解决数据处理流程中错综复杂的依赖关系，使调度系统在数据处理流程中开箱即用。其原理图如下：接下来，本文一步一步详细地...
apache dolphinscheduler worker核心源码解析2.0.1版本
m0_37404492的博客
02-19 167
入口类：WorkerServer，关键类和设计模式。 TaskPluginManager的函数installPlugin使用Java spi的serviceLoader注入成员taskChannelMap。 TaskChannel使用抽像工厂模式TaskChannelFactory进行创建生成。
Apache DolphinScheduler 任务调度3.1.0版本源码剖析
Apache DolphinScheduler开源社区
12-21 2063
目录（本文章基于3.1.0版本研究）1、数据库表介绍2、 整体流程运行3、 源码剖析3.1 apiserver任务执行入口3.2 master调度任务3.2.1 master启动3.2.2 command扫描3.2.3 workerFlowEvent消费3.2.4 workerflow事件处理逻辑3.2.5 workerflowRunnable运行逻辑3.2.6 任务消费3.2.7 任务分派3.3...
apache dolphinscheduler 源码
05-20
Apache DolphinScheduler 的源代码可以在它的官方 GitHub 仓库中获取：https://github.com/apache/dolphinscheduler 在这个仓库中，你可以找到所有关于 DolphinScheduler 的源代码，包括后端、前端和调度引擎等部分。如果你想要了解 DolphinScheduler 的架构和实现细节，这些源代码将会是非常有用的资源。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值