Hive 任务提交流程
MapReduce
Map 执行流程
OldMap(Hive MapExec)执行流程
RM中事件转发器:AsyncDispatcher
通过rmcontext启动AM
this.rmContext.getDispatcher().getEventHandler()
.handle(new RMAppEvent(applicationId, RMAppEventType.START));
这里看到getDispatcher取rmDispatcher。
rmDispatcher初始化在ResourceManager类中:
rmDispatcher.register(SchedulerEventType.class, schedulerDispatcher);
// Register event handler for RmAppEvents
rmDispatcher.register(RMAppEventType.class,
new ApplicationEventDispatcher(rmContext));
// Register event handler for RmAppAttemptEvents
rmDispatcher.register(RMAppAttemptEventType.class,
new ApplicationAttemptEventDispatcher(rmContext));
// Register event handler for RmNodes
rmDispatcher.register(
RMNodeEventType.class, new NodeEventDispatcher(rmContext));
ApplicationEventDispatcher.handle –> RMAppImpl.handle 这里的start事件仅仅做了一个状态转换处理
YarnChild 启动
- 查看进程:
ps -ef|grep java|grep YarnChild
- default_container_executor.sh 位置:
/data/yarn/local/usercache/hadoop/appcache/application_1486524626038_0307/container_e33_1486524626038_0307_01_000002/default_container_executor.sh
- 启动程序Java命令
$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -agentlib:jdwp=transport=dt_socket,server=y,address=8002,suspend=y -Xmx1639m -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/data/yarn/logs/application_1486524626038_0307/container_e33_1486524626038_0307_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.20.9.42 41382 attempt_1486524626038_0307_m_000000_1 36283883716611 1>/data/yarn/logs/application_1486524626038_0307/container_e33_1486524626038_0307_01_000003/stdout 2>/data/yarn/logs/application_1486524626038_0307/container_e33_1486524626038_0307_01_000003/stderr
- 示例Map执行流程
Operator List:
MapOperator -> TableScanOperator -> FilterOperator -> SelectOperator -> LimitOperator -> ReduceSinkOperator -> OutputCollector out.collect(key,value)