Flink任务如何实时获取控制台日志

开源数据开发平台：GitHub - 642933588/jiron-cloud: 该项目整合了多款优秀的开源产品，构建了一个功能全面的数据开发平台。平台提供了强大的数据集成、数据开发、数据查询、数据服务、数据质量管理、工作流调度和元数据管理功能。#dinky #dolphinscheduler #datavines #flinkcdc #openmetadata #flink #数据开发 #数据平台 # 数据开发平台 #大数据

利用jiron数据开发平台提交flink任务到flink 集群，实现IDE控制台实时展示日志信息。

实现效果：

定义注解：

（开始和结束流程的切面）

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Inherited
public @interface ExecuteProcess {

    ProcessType type();
}

（执行过程记录）

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Inherited
public @interface ProcessStep {
    ProcessStepType type();
}

配置切面拦截注解。把执行的流程注册到上下文中，然后执行具体的操作，根据执行结果更新流程结果。

@Around(value = "@annotation(executeProcess)")
    public Object processAround(ProceedingJoinPoint joinPoint, ExecuteProcess executeProcess) throws Throwable {

        Object result;
        Object processId = getProcessId(joinPoint);
        String name = StrFormatter.format("{}/{}", executeProcess.type().getValue(), String.valueOf(processId));
        ProcessType type = executeProcess.type();
//注册流程
        contextHolder.registerProcess(type, name);
        MDC.put(PROCESS_NAME, name);

        try {
//处理
            result = joinPoint.proceed();
//更新流程结果
            contextHolder.finishedProcess(name, ProcessStatus.FINISHED, null);
        } catch (Throwable e) {
//更新流程结果
            contextHolder.finishedProcess(name, ProcessStatus.FAILED, e);
            throw e;
        } finally {
            MDC.clear();
        }
        return result;
    }

配置执行步骤切面进行日志更新

@Around(value = "@annotation(processStep)")
    public Object processStepAround(ProceedingJoinPoint joinPoint, ProcessStep processStep) throws Throwable {

        String processName = MDC.get(PROCESS_NAME);
        if (TextUtils.isEmpty(processName)) {
            log.warn(
                    "Process {} does not exist, This registration step {} was abandoned",
                    processName,
                    processStep.type());
            return joinPoint.proceed();
        }

        Object result;
        // Record the current step and restore it after the execution is completed
        String parentStep = MDC.get(PROCESS_STEP);
        ProcessStepType processStepType = processStep.type();
        ProcessStepEntity step = contextHolder.registerProcessStep(processStepType, MDC.get(PROCESS_NAME), parentStep);
        MDC.put(PROCESS_STEP, step.getKey());
        contextHolder.appendLog(processName, step.getKey(), "Start Process Step:" + step.getType(), true);

        try {
            result = joinPoint.proceed();
            contextHolder.finishedStep(MDC.get(PROCESS_NAME), step, ProcessStatus.FINISHED, null);
        } catch (Exception e) {
            contextHolder.finishedStep(MDC.get(PROCESS_NAME), step, ProcessStatus.FAILED, e);
            throw e;
        } finally {
            // restored after the execution is complete
            MDC.put(PROCESS_STEP, parentStep);
        }
        return result;
    }

首先，提交任务，注册流程(日志：Start Process:FlinkSubmit/4)

@GetMapping("/submitTask")
    @ApiOperation("Submit Task")
    @Log(title = "Submit Task", businessType = BusinessType.SUBMIT)
//注册流程
    @ExecuteProcess(type = ProcessType.FLINK_SUBMIT)
    public Result<JobResult> submitTask(@ProcessId @RequestParam Integer id) throws Exception {
        JobResult jobResult =
                taskService.submitTask(TaskSubmitDto.builder().id(id).build());
        if (jobResult.isSuccess()) {
            return Result.succeed(jobResult, Status.EXECUTE_SUCCESS);
        } else {
            return Result.failed(jobResult, jobResult.getError());
        }
    }

（Start Process Step:SUBMIT_PRECHECK）

@ProcessStep(type = ProcessStepType.SUBMIT_PRECHECK)
    public TaskDTO prepareTask(TaskSubmitDto submitDto) {
        TaskDTO task = this.getTaskInfoById(submitDto.getId());

        log.info("Start check and config task, task:{}", task.getName());

        DinkyAssert.check(task);

        if (StringUtils.isNotBlank(submitDto.getSavePointPath())) {
            task.setSavePointStrategy(SavePointStrategy.CUSTOM.getValue());
            task.setSavePointPath(submitDto.getSavePointPath());
        }
        task.setVariables(Optional.ofNullable(submitDto.getVariables()).orElse(new HashMap<>()));
        return task;
    }

........中间步骤省略

（Start Process Step:SUBMIT_EXECUTE）

@ProcessStep(type = ProcessStepType.SUBMIT_EXECUTE)
    public JobResult executeJob(TaskDTO task) throws Exception {
        JobResult jobResult = BaseTask.getTask(task).execute();
        log.info("execute job finished,status is {}", jobResult.getStatus());
        return jobResult;
    }

前端通过api请求获取执行结果，在控制台展示

@GetMapping("/getProcess")
    @ApiOperation("get process")
    @ApiImplicitParam(name = "processName", value = "process name", dataType = "ProcessEntity")
    public Result<ProcessEntity> listAllProcess(@RequestParam String processName) {
        return Result.succeed(ConsoleContextHolder.getInstances().getProcess(processName));
    }