2.7.3-YARN-获取debug命令:resourceManager+nodeManager

最新推荐文章于 2023-07-25 09:37:33 发布

weixin_34204057

最新推荐文章于 2023-07-25 09:37:33 发布

阅读量547

点赞数

文章标签：大数据 python java

原文链接：https://my.oschina.net/qiangzigege/blog/1933559

版权

2019独角兽企业重金招聘Python工程师标准>>>

1 源码和二进制下载

http://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/

源码

http://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3-src.tar.gz

二进制

http://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3.tar.gz

2yarn启动在machine0上

3启动脚本分析./sbin/start-yarn.sh

3.1正常打印

略

3.2libexec/yarn-config.sh【完毕】

3.2.1libexec/hadoop-config.sh【ing】

关于配置文件的位置说明

直接看代码

# Allow alternate conf dir location.

if [ -e "${HADOOP_PREFIX}/conf/hadoop-env.sh" ]; then

DEFAULT_CONF_DIR="conf"

else

DEFAULT_CONF_DIR="etc/hadoop"

3.2.1.1 hadoop-env.sh【完毕】

层次还挺深...

3.3sbin/yarn-daemon.sh启动resourcemanager【完毕】

真实的命令

# start resourceManager

"$bin"/yarn-daemon.sh --config $YARN_CONF_DIR start resourcemanager

让我们看看具体做了什么

规范：

yarn-daemon.sh [--config <conf-dir>] [--hosts hostlistfile] (start|stop) <yarn-command>

3.3.1 libexec/yarn-config.sh【同上】

3.3.2/etc/hadoop/yarn-env.sh【完毕】

3.3.3执行start函数获得resourceManager启动命令

/root/jdk1.8.0_181/bin/java -Dproc_resourcemanager -Xmx100m -Dhadoop.log.dir=/root/yarn_study/hadoop-2.7.3/logs -Dyarn.log.dir=/root/yarn_study/hadoop-2.7.3/logs -Dhadoop.log.file=yarn-yarn-resourcemanager-machine0.log -Dyarn.log.file=yarn-yarn-resourcemanager-machine0.log -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/root/yarn_study/hadoop-2.7.3/lib/native -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/root/yarn_study/hadoop-2.7.3/logs -Dyarn.log.dir=/root/yarn_study/hadoop-2.7.3/logs -Dhadoop.log.file=yarn-yarn-resourcemanager-machine0.log -Dyarn.log.file=yarn-yarn-resourcemanager-machine0.log -Dyarn.home.dir=/root/yarn_study/hadoop-2.7.3 -Dhadoop.home.dir=/root/yarn_study/hadoop-2.7.3 -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/root/yarn_study/hadoop-2.7.3/lib/native -classpath /root/yarn_study/hadoop-2.7.3/etc/hadoop:/root/yarn_study/hadoop-2.7.3/etc/hadoop:/root/yarn_study/hadoop-2.7.3/etc/hadoop:/root/yarn_study/hadoop-2.7.3/share/hadoop/common/lib/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/common/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/hdfs:/root/yarn_study/hadoop-2.7.3/share/hadoop/hdfs/lib/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/hdfs/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/yarn/lib/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/yarn/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/mapreduce/lib/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/mapreduce/*:/root/yarn_study/hadoop-2.7.3/contrib/capacity-scheduler/*.jar:/root/yarn_study/hadoop-2.7.3/contrib/capacity-scheduler/*.jar:/root/yarn_study/hadoop-2.7.3/contrib/capacity-scheduler/*.jar:/root/yarn_study/hadoop-2.7.3/share/hadoop/yarn/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/yarn/lib/*:/root/yarn_study/hadoop-2.7.3/etc/hadoop/rm-config/log4j.properties org.apache.hadoop.yarn.server.resourcemanager.ResourceManager

3.4sbin/yarn-daemons.sh启动nodemanager

3.4.1 libexec/yarn-config.sh【结束】

3.4.2sbin/slaves.sh

3.4.2.1libexec/hadoop-config.sh【结束】

3.4.2.2/etc/hadoop/hadoop-env.sh

3.4.2.3循环连接slave执行远程命令

cd /root/yarn_study/hadoop-2.7.3 ;

/root/yarn_study/hadoop-2.7.3/sbin/yarn-daemon.sh --config /root/yarn_study/hadoop-2.7.3/etc/hadoop start nodemanager

这个就类似于3.3的命令了！

回去处理3.3就可以！

4 本质调用的/bin/yarn --config $YARN_CONF_DIR $command "$@"

细节不说了！

5总结2个的jdb命令

5.1 resourcemananger

/root/jdk1.8.0_181/bin/java -Dproc_resourcemanager -Xmx100m -Dhadoop.log.dir=/root/yarn_study/hadoop-2.7.3/logs -Dyarn.log.dir=/root/yarn_study/hadoop-2.7.3/logs -Dhadoop.log.file=yarn-root-resourcemanager-machine0.log -Dyarn.log.file=yarn-root-resourcemanager-machine0.log -Dyarn.home.dir= -Dyarn.id.str=root -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/root/yarn_study/hadoop-2.7.3/lib/native -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/root/yarn_study/hadoop-2.7.3/logs -Dyarn.log.dir=/root/yarn_study/hadoop-2.7.3/logs -Dhadoop.log.file=yarn-root-resourcemanager-machine0.log -Dyarn.log.file=yarn-root-resourcemanager-machine0.log -Dyarn.home.dir=/root/yarn_study/hadoop-2.7.3 -Dhadoop.home.dir=/root/yarn_study/hadoop-2.7.3 -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/root/yarn_study/hadoop-2.7.3/lib/native -classpath /root/yarn_study/hadoop-2.7.3/etc/hadoop:/root/yarn_study/hadoop-2.7.3/etc/hadoop:/root/yarn_study/hadoop-2.7.3/etc/hadoop:/root/yarn_study/hadoop-2.7.3/share/hadoop/common/lib/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/common/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/hdfs:/root/yarn_study/hadoop-2.7.3/share/hadoop/hdfs/lib/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/hdfs/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/yarn/lib/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/yarn/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/mapreduce/lib/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/mapreduce/*:/root/yarn_study/hadoop-2.7.3/contrib/capacity-scheduler/*.jar:/root/yarn_study/hadoop-2.7.3/contrib/capacity-scheduler/*.jar:/root/yarn_study/hadoop-2.7.3/contrib/capacity-scheduler/*.jar:/root/yarn_study/hadoop-2.7.3/share/hadoop/yarn/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/yarn/lib/*:/root/yarn_study/hadoop-2.7.3/etc/hadoop/rm-config/log4j.properties org.apache.hadoop.yarn.server.resourcemanager.ResourceManager

5.2nodemanange

/root/jdk1.8.0_181/bin/java -Dproc_nodemanager -Xmx100m -Dhadoop.log.dir=/root/yarn_study/hadoop-2.7.3/logs -Dyarn.log.dir=/root/yarn_study/hadoop-2.7.3/logs -Dhadoop.log.file=yarn-root-nodemanager-machine1.log -Dyarn.log.file=yarn-root-nodemanager-machine1.log -Dyarn.home.dir= -Dyarn.id.str=root -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/root/yarn_study/hadoop-2.7.3/lib/native -Dyarn.policy.file=hadoop-policy.xml -server -Dhadoop.log.dir=/root/yarn_study/hadoop-2.7.3/logs -Dyarn.log.dir=/root/yarn_study/hadoop-2.7.3/logs -Dhadoop.log.file=yarn-root-nodemanager-machine1.log -Dyarn.log.file=yarn-root-nodemanager-machine1.log -Dyarn.home.dir=/root/yarn_study/hadoop-2.7.3 -Dhadoop.home.dir=/root/yarn_study/hadoop-2.7.3 -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/root/yarn_study/hadoop-2.7.3/lib/native -classpath /root/yarn_study/hadoop-2.7.3/etc/hadoop:/root/yarn_study/hadoop-2.7.3/etc/hadoop:/root/yarn_study/hadoop-2.7.3/etc/hadoop:/root/yarn_study/hadoop-2.7.3/share/hadoop/common/lib/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/common/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/hdfs:/root/yarn_study/hadoop-2.7.3/share/hadoop/hdfs/lib/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/hdfs/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/yarn/lib/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/yarn/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/mapreduce/lib/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/mapreduce/*:/root/yarn_study/hadoop-2.7.3/contrib/capacity-scheduler/*.jar:/root/yarn_study/hadoop-2.7.3/contrib/capacity-scheduler/*.jar:/root/yarn_study/hadoop-2.7.3/share/hadoop/yarn/*:/root/yarn_study/hadoop-2.7.3/share/hadoop/yarn/lib/*:/root/yarn_study/hadoop-2.7.3/etc/hadoop/nm-config/log4j.properties org.apache.hadoop.yarn.server.nodemanager.NodeManager