Hive，MapReduce执行流程分析

Hive 任务提交流程

MapReduce

Map 执行流程

OldMap(Hive MapExec)执行流程

RM中事件转发器：AsyncDispatcher

通过rmcontext启动AM
this.rmContext.getDispatcher().getEventHandler()
.handle(new RMAppEvent(applicationId, RMAppEventType.START));
这里看到getDispatcher取rmDispatcher。

rmDispatcher初始化在ResourceManager类中：
rmDispatcher.register(SchedulerEventType.class, schedulerDispatcher);

  // Register event handler for RmAppEvents
  rmDispatcher.register(RMAppEventType.class,
      new ApplicationEventDispatcher(rmContext));

  // Register event handler for RmAppAttemptEvents
  rmDispatcher.register(RMAppAttemptEventType.class,
      new ApplicationAttemptEventDispatcher(rmContext));

  // Register event handler for RmNodes
  rmDispatcher.register(
      RMNodeEventType.class, new NodeEventDispatcher(rmContext));

ApplicationEventDispatcher.handle –> RMAppImpl.handle 这里的start事件仅仅做了一个状态转换处理

YarnChild 启动

• 查看进程：ps -ef|grep java|grep YarnChild
• default_container_executor.sh 位置：/data/yarn/local/usercache/hadoop/appcache/application_1486524626038_0307/container_e33_1486524626038_0307_01_000002/default_container_executor.sh
• 启动程序Java命令

$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -agentlib:jdwp=transport=dt_socket,server=y,address=8002,suspend=y -Xmx1639m -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/data/yarn/logs/application_1486524626038_0307/container_e33_1486524626038_0307_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.20.9.42 41382 attempt_1486524626038_0307_m_000000_1 36283883716611 1>/data/yarn/logs/application_1486524626038_0307/container_e33_1486524626038_0307_01_000003/stdout 2>/data/yarn/logs/application_1486524626038_0307/container_e33_1486524626038_0307_01_000003/stderr

• 示例Map执行流程
  Operator List:
  MapOperator -> TableScanOperator -> FilterOperator -> SelectOperator -> LimitOperator -> ReduceSinkOperator -> OutputCollector out.collect(key,value)

MR 流程图