flink runtime 三个组件 dispatcher,jobmanager,jobmaster

模式
在yarn-session模式下,共享一个dispatcher
org.apache.flink.runtime.dispatcher.Dispatcher 这个类,这里接收到客户端传上来的作业图

代码分析
 //这里就是分发器接收到作业图
    @Override
    public CompletableFuture<Acknowledge> submitJob(JobGraph jobGraph, Time timeout) {
        log.info("Received JobGraph submission {} ({}).", jobGraph.getJobID(), jobGraph.getName());

        try {
            if (isDuplicateJob(jobGraph.getJobID())) {
                return FutureUtils.completedExceptionally(
                    new DuplicateJobSubmissionException(jobGraph.getJobID()));
            } else if (isPartialResourceConfigured(jobGraph)) {
                return FutureUtils.completedExceptionally(
                    new JobSubmissionException(jobGraph.getJobID(), "Currently jobs is not supported if parts of the vertices have " +
                            "resources configured. The limitation will be removed in future versions."));
            } else {
                return internalSubmitJob(jobGraph);
            }
        } catch (FlinkException e) {
            return FutureUtils.completedExceptionally(e);
        }
    }

从这个方法跳出这个类,创建jobmanager ,实际上也就是这个类
JobManagerRunnerImpl

private CompletableFuture<JobManagerRunner> createJobManagerRunner(JobGraph jobGraph) {
        final RpcService rpcService = getRpcService();

        return CompletableFuture.supplyAsync(
            CheckedSupplier.unchecked(() ->
                jobManagerRunnerFactory.createJobManagerRunner(
                    jobGraph,
                    configuration,
                    rpcService,
                    highAvailabilityServices,
                    heartbeatServices,
                    jobManagerSharedServices,
                    new DefaultJobManagerJobMetricGroupFactory(jobManagerMetricGroup),
                    fatalErrorHandler)),
            rpcService.getExecutor());
    }

org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl 看这个的构造函数
创建jobmaster

public JobManagerRunnerImpl(
            final JobGraph jobGraph,
            final JobMasterServiceFactory jobMasterFactory,
            final HighAvailabilityServices haServices,
            final LibraryCacheManager libraryCacheManager,
            final Executor executor,
            final FatalErrorHandler fatalErrorHandler) throws Exception {

        this.resultFuture = new CompletableFuture<>();
        this.terminationFuture = new CompletableFuture<>();
        this.leadershipOperation = CompletableFuture.completedFuture(null);

        // make sure we cleanly shut down out JobManager services if initialization fails
        try {
            this.jobGraph = checkNotNull(jobGraph);
            this.libraryCacheManager = checkNotNull(libraryCacheManager);
            this.executor = checkNotNull(executor);
            this.fatalErrorHandler = checkNotNull(fatalErrorHandler);

            checkArgument(jobGraph.getNumberOfVertices() > 0, "The given job is empty");

            // libraries and class loader first
            try {
                libraryCacheManager.registerJob(
                        jobGraph.getJobID(), jobGraph.getUserJarBlobKeys(), jobGraph.getClasspaths());
            } catch (IOException e) {
                throw new Exception("Cannot set up the user code libraries: " + e.getMessage(), e);
            }

            final ClassLoader userCodeLoader = libraryCacheManager.getClassLoader(jobGraph.getJobID());
            if (userCodeLoader == null) {
                throw new Exception("The user code class loader could not be initialized.");
            }

            // high availability services next
            this.runningJobsRegistry = haServices.getRunningJobsRegistry();
            this.leaderElectionService = haServices.getJobManagerLeaderElectionService(jobGraph.getJobID());

            this.leaderGatewayFuture = new CompletableFuture<>();

            // now start the JobManager
            //这里创建 jobmaster
            
            this.jobMasterService = jobMasterFactory.createJobMasterService(jobGraph, this, userCodeLoader);
        }
        catch (Throwable t) {
            terminationFuture.completeExceptionally(t);
            resultFuture.completeExceptionally(t);

            throw new JobExecutionException(jobGraph.getJobID(), "Could not set up JobManager", t);
        }
    }

实际创建jobmaster的代码
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory

@Override
    public JobMaster createJobMasterService(
            JobGraph jobGraph,
            OnCompletionActions jobCompletionActions,
            ClassLoader userCodeClassloader) throws Exception {

        return new JobMaster(
            rpcService,
            jobMasterConfiguration,
            ResourceID.generate(),
            jobGraph,
            haServices,
            slotPoolFactory,
            schedulerFactory,
            jobManagerSharedServices,
            heartbeatServices,
            jobManagerJobMetricGroupFactory,
            jobCompletionActions,
            fatalErrorHandler,
            userCodeClassloader,
            schedulerNGFactory,
            shuffleMaster,
            lookup -> new JobMasterPartitionTrackerImpl(
                jobGraph.getJobID(),
                shuffleMaster,
                lookup
            ));
    }

总结
从这里分析得到yarn-session 集群中只有一个dispatcher,每一个作业对应一个jobmanager,jobmaster
 

  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值