Yarn源码分析之MRAppMaster:作业运行方式Local、Uber、Non-Uber

本文介绍了MRAppMaster提供的三种作业运行方式:本地Local模式、Uber模式与Non-Uber模式。详细阐述了每种模式的特点及其在资源分配和服务启动过程中的具体实现。

        基于作业大小因素,MRAppMaster提供了三种作业运行方式:本地Local模式、Uber模式、Non-Uber模式。其中,

        1、本地Local模式:通常用于调试;

        2、Uber模式:为降低小作业延迟而设计的一种模式,所有任务,不管是Map Task,还是Reduce Task,均在同一个Container中顺序执行,这个Container其实也是MRAppMaster所在Container;

        3、Non-Uber模式:对于运行时间较长的大作业,先为Map Task申请资源,当Map Task运行完成数目达到一定比例后再为Reduce Task申请资源。


         在Yarn中,作业运行的资源,统一被抽象为容器Container,在MRAppMaster中关于作业运行时需要的资源的分配与加载代码中,容器分配申请服务、容器分配完成后加载服务中,都有关于Uber模式和Non-Uber模式的处理,如下:

        1、容器分配申请路由服务

         容器分配申请路由服务ContainerAllocatorRouter继承自AbstractService,是Hadoop中一个典型的服务,其正常提供服务需要经历初始化init、启动start等过程,而在服务启动的serviceStart()方法中,存在以下关于Uber模式和Non-Uber模式的处理:

      // 如果Job在Uber模式下运行,启动构造容器分配器LocalContainerAllocator实例
      if (job.isUber()) {
        MRApps.setupDistributedCacheLo
sqoop import \ --connect jdbc:mysql://localhost:3306/userdb \ --username root \ --password 123456 \ --target-dir /sqoopresult \ --table emp \ --num-mappers 1 Warning: /path/to/hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /path/to/accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 2025-12-05 14:10:46,101 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 2025-12-05 14:10:46,207 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 2025-12-05 14:10:46,516 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 2025-12-05 14:10:46,517 INFO tool.CodeGenTool: Beginning code generation Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 2025-12-05 14:10:47,805 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `emp` AS t LIMIT 1 2025-12-05 14:10:47,953 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `emp` AS t LIMIT 1 2025-12-05 14:10:47,979 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop 注: /tmp/sqoop-hadoop/compile/af9bbaac7600f834291d5e54960cd5e6/emp.java使用或覆盖了已过时的 API。 注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。 2025-12-05 14:10:52,152 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/af9bbaac7600f834291d5e54960cd5e6/emp.jar 2025-12-05 14:10:52,207 WARN manager.MySQLManager: It looks like you are importing from mysql. 2025-12-05 14:10:52,208 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 2025-12-05 14:10:52,208 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 2025-12-05 14:10:52,208 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 2025-12-05 14:10:52,222 INFO mapreduce.ImportJobBase: Beginning import of emp 2025-12-05 14:10:52,226 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2025-12-05 14:10:52,540 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 2025-12-05 14:10:54,496 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 2025-12-05 14:10:54,674 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at localhost/127.0.0.1:8032 2025-12-05 14:10:56,522 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1764912232622_0003 2025-12-05 14:11:03,239 INFO db.DBInputFormat: Using read commited transaction isolation 2025-12-05 14:11:03,858 INFO mapreduce.JobSubmitter: number of splits:1 2025-12-05 14:11:04,290 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1764912232622_0003 2025-12-05 14:11:04,290 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2025-12-05 14:11:04,884 INFO conf.Configuration: resource-types.xml not found 2025-12-05 14:11:04,884 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2025-12-05 14:11:05,076 INFO impl.YarnClientImpl: Submitted application application_1764912232622_0003 2025-12-05 14:11:05,196 INFO mapreduce.Job: The url to track the job: http://yu-virtual-machine:8088/proxy/application_1764912232622_0003/ 2025-12-05 14:11:05,199 INFO mapreduce.Job: Running job: job_1764912232622_0003 2025-12-05 14:11:12,358 INFO mapreduce.Job: Job job_1764912232622_0003 running in uber mode : false 2025-12-05 14:11:12,369 INFO mapreduce.Job: map 0% reduce 0% 2025-12-05 14:11:12,410 INFO mapreduce.Job: Job job_1764912232622_0003 failed with state FAILED due to: Application application_1764912232622_0003 failed 2 times due to AM Container for appattempt_1764912232622_0003_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2025-12-05 14:11:11.833]Exception from container-launch. Container id: container_1764912232622_0003_02_000001 Exit code: 1 [2025-12-05 14:11:11.837]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : 错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster [2025-12-05 14:11:11.839]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : 错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster For more detailed output, check the application tracking page: http://yu-virtual-machine:8088/cluster/app/application_1764912232622_0003 Then click on links to logs of each attempt. . Failing the application. 2025-12-05 14:11:12,499 INFO mapreduce.Job: Counters: 0 2025-12-05 14:11:12,525 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 2025-12-05 14:11:12,531 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 17.9976 seconds (0 bytes/sec) 2025-12-05 14:11:12,548 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 2025-12-05 14:11:12,550 INFO mapreduce.ImportJobBase: Retrieved 0 records. 2025-12-05 14:11:12,551 ERROR tool.ImportTool: Import failed: Import job failed!
最新发布
12-06
[root@hadoop1 jar]# hadoop jar DataCleaning.jar cn.itcast.jobinfo.JobDriver /job/origin/2023-12-08 /job/output/2023-12-08 2025-10-13 10:58:48,402 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1760323847563_0002 0 [main] INFO org.apache.hadoop.mapreduce.JobResourceUploader - Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1760323847563_0002 2025-10-13 10:58:49,040 INFO input.FileInputFormat: Total input files to process : 34 638 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input files to process : 34 2025-10-13 10:58:50,050 INFO mapreduce.JobSubmitter: number of splits:34 1648 [main] INFO org.apache.hadoop.mapreduce.JobSubmitter - number of splits:34 2025-10-13 10:58:50,667 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1760323847563_0002 2265 [main] INFO org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1760323847563_0002 2025-10-13 10:58:50,669 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2267 [main] INFO org.apache.hadoop.mapreduce.JobSubmitter - Executing with tokens: [] 2025-10-13 10:58:50,867 INFO conf.Configuration: resource-types.xml not found 2465 [main] INFO org.apache.hadoop.conf.Configuration - resource-types.xml not found 2025-10-13 10:58:50,868 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2466 [main] INFO org.apache.hadoop.yarn.util.resource.ResourceUtils - Unable to find 'resource-types.xml'. 2025-10-13 10:58:50,915 INFO impl.YarnClientImpl: Submitted application application_1760323847563_0002 2513 [main] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1760323847563_0002 2025-10-13 10:58:50,952 INFO mapreduce.Job: The url to track the job: http://hadoop1.chensp.vip:8088/proxy/application_1760323847563_0002/ 2550 [main] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://hadoop1.chensp.vip:8088/proxy/application_1760323847563_0002/ 2025-10-13 10:58:50,953 INFO mapreduce.Job: Running job: job_1760323847563_0002 2551 [main] INFO org.apache.hadoop.mapreduce.Job - Running job: job_1760323847563_0002 2025-10-13 10:59:04,133 INFO mapreduce.Job: Job job_1760323847563_0002 running in uber mode : false 15731 [main] INFO org.apache.hadoop.mapreduce.Job - Job job_1760323847563_0002 running in uber mode : false 2025-10-13 10:59:04,135 INFO mapreduce.Job: map 0% reduce 0% 15733 [main] INFO org.apache.hadoop.mapreduce.Job - map 0% reduce 0% 2025-10-13 10:59:04,156 INFO mapreduce.Job: Job job_1760323847563_0002 failed with state FAILED due to: Application application_1760323847563_0002 failed 2 times due to AM Container for appattempt_1760323847563_0002_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2025-10-13 10:59:03.957]Exception from container-launch. Container id: container_1760323847563_0002_02_000001 Exit code: 1 [2025-10-13 10:59:03.958]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. [2025-10-13 10:59:03.959]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. For more detailed output, check the application tracking page: http://hadoop1.chensp.vip:8088/cluster/app/application_1760323847563_0002 Then click on links to logs of each attempt. . Failing the application. 15754 [main] INFO org.apache.hadoop.mapreduce.Job - Job job_1760323847563_0002 failed with state FAILED due to: Application application_1760323847563_0002 failed 2 times due to AM Container for appattempt_1760323847563_0002_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2025-10-13 10:59:03.957]Exception from container-launch. Container id: container_1760323847563_0002_02_000001 Exit code: 1 [2025-10-13 10:59:03.958]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. [2025-10-13 10:59:03.959]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. For more detailed output, check the application tracking page: http://hadoop1.chensp.vip:8088/cluster/app/application_1760323847563_0002 Then click on links to logs of each attempt. . Failing the application. 2025-10-13 10:59:04,179 INFO mapreduce.Job: Counters: 0 15777 [main] INFO org.apache.hadoop.mapreduce.Job - Counters: 0
10-14
C:\hadoop-3.0.0\bin>hadoop jar C:\hadoop-3.0.0\share\hadoop\mapreduce\hadoop-mapreduce-examples-3.0.0.jar wordcount /user /output_dir/output.txt 2025-11-02 00:11:08,487 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 2025-11-02 00:11:10,136 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/???ú/.staging/job_1762012997911_0001 2025-11-02 00:11:10,542 INFO input.FileInputFormat: Total input files to process : 1 2025-11-02 00:11:10,689 INFO mapreduce.JobSubmitter: number of splits:1 2025-11-02 00:11:10,744 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 2025-11-02 00:11:10,896 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1762012997911_0001 2025-11-02 00:11:10,900 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2025-11-02 00:11:11,214 INFO conf.Configuration: resource-types.xml not found 2025-11-02 00:11:11,215 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2025-11-02 00:11:11,863 INFO impl.YarnClientImpl: Submitted application application_1762012997911_0001 2025-11-02 00:11:11,924 INFO mapreduce.Job: The url to track the job: http://LAPTOP-NU3UFCJE:8088/proxy/application_1762012997911_0001/ 2025-11-02 00:11:11,926 INFO mapreduce.Job: Running job: job_1762012997911_0001 2025-11-02 00:11:37,304 INFO mapreduce.Job: Job job_1762012997911_0001 running in uber mode : false 2025-11-02 00:11:37,306 INFO mapreduce.Job: map 0% reduce 0% 2025-11-02 00:11:37,335 INFO mapreduce.Job: Job job_1762012997911_0001 failed with state FAILED due to: Application application_1762012997911_0001 failed 2 times due to AM Container for appattempt_1762012997911_0001_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2025-11-02 00:11:30.654]Exception from container-launch. Container id: container_1762012997911_0001_02_000001 Exit code: 1 Exception message: 系统找不到指定的路径。 系统找不到指定的路径。 系统找不到指定的路径。 [2025-11-02 00:11:30.659]Container exited with a non-zero exit code 1. [2025-11-02 00:11:30.659]Container exited with a non-zero exit code 1. For more detailed output, check the application tracking page: http://LAPTOP-NU3UFCJE:8088/cluster/app/application_1762012997911_0001 Then click on links to logs of each attempt. . Failing the application. 2025-11-02 00:11:37,385 INFO mapreduce.Job: Counters: 0为什么不能输出
11-03
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值