hadoop源码解析-Yarn源码解析

LD is tigger forever，CG are not brothers forever， throw the pot and shine.
Efficient work is better than attitude。All right, hide it。Advantages should be hidden.
talk is selected, show others the code,Keep progress，make a better result.

目录

概述

hadoop源码解析-Yarn源码解析

正文：

0.mr程序提交到客户端所在的节点
1.由YarnRunner运行。
2.申请一个Applicaction,交给ResourceManager:
3.提交job运行所需资源
4.资源提交完毕，
申请运行mrAppMaster
5.将用户的请求初始化成一个Task，然后领取到Task任务。
对于NodeManager来说，进行2个操作，创建容器container,下载job资源到本地。
9.申请运行MapTask容器，由NodeManager创建容器，
10.NodeManager发送程序启动脚本到NodeManager.
11.向RM申请2个容器，运行ReduceTask任务，执行NodeManager的reduceTask任务。
12.Reduce向map获取相应的分区的数据。
13.程序运行完成后，MR会向RM注销自己。

源码如下：
4.1 Yarn 客户端向 RM 提交作业
1）在 wordcount 程序的驱动类中点击
Job.java
boolean result = job.waitForCompletion(true);
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
public void submit()
throws IOException, InterruptedException, ClassNotFoundException {
ensureState(JobState.DEFINE);
setUseNewAPI();
connect();
final JobSubmitter submitter =
getJobSubmitter(cluster.getFileSystem(), cluster.getClient());
status = ugi.doAs(new PrivilegedExceptionAction() {
public JobStatus run() throws IOException, InterruptedException,
ClassNotFoundException {
return submitter.submitJobInternal(Job.this, cluster);
}
});
state = JobState.RUNNING;
LOG.info("The url to track the job: " + getTrackingURL());
}
JobStatus submitJobInternal(Job job, Cluster cluster)
throws ClassNotFoundException, InterruptedException, IOException {
… …
status = submitClient.submitJob(
jobId, submitJobDir.toString(), job.getCredentials());
… …
}
public JobStatus submitJob(JobID jobId, String jobSubmitDir, Credentials
ts) throws IOException, InterruptedException;

2）创建提交环境
public JobStatus submitJob(JobID jobId, String jobSubmitDir, Credentials
ts)
throws IOException, InterruptedException {
addHistoryToken(ts);
// 创建提交环境：
ApplicationSubmissionContext appContext =
createApplicationSubmissionContext(conf, jobSubmitDir, ts);
// 向 RM 提交一个应用程序，appContext 里面封装了启动 mrappMaster 和运行 container
的命令
ApplicationId applicationId =
resMgrDelegate.submitApplication(appContext);
// 获取提交响应
ApplicationReport appMaster = resMgrDelegate
.getApplicationReport(applicationId);
String diagnostics =
(appMaster == null ?
“application report is null” : appMaster.getDiagnostics());
if (appMaster == null
|| appMaster.getYarnApplicationState() ==
YarnApplicationState.FAILED
|| appMaster.getYarnApplicationState() ==
YarnApplicationState.KILLED) {
throw new IOException("Failed to run job : " +
diagnostics);
}
return clientCache.getClient(jobId).getJobStatus(jobId);
} catch (YarnException e) {
throw new IOException(e);
}
}
public ApplicationSubmissionContext createApplicationSubmissionContext(
Configuration jobConf, String jobSubmitDir, Credentials ts)
throws IOException {
ApplicationId applicationId = resMgrDelegate.getApplicationId();
// Setup LocalResources
// 封装了本地资源相关路径
Map<String, LocalResource> localResources =
setupLocalResources(jobConf, jobSubmitDir);
// Setup security tokens
DataOutputBuffer dob = new DataOutputBuffer();
ts.writeTokenStorageToStream(dob);
ByteBuffer securityTokens =
ByteBuffer.wrap(dob.getData(), 0, dob.getLength());
// Setup ContainerLaunchContext for AM container
List vargs = setupAMCommand(jobConf);
ContainerLaunchContext amContainer = setupContainerLaunchContextForAM(
jobConf, localResources, securityTokens, vargs);
… …
return appContext;
}

设计思路

相关代码如下：

完整版下载： hadoop源码解析-Yarn源码解析

小结：

主要讲述了自己的一些体会，里面有许多不足，请大家指正~

参考资料和推荐阅读

深度开源: link

欢迎阅读，各位老铁，如果对你有帮助，点个赞加个关注呗！~