YARN-2.7.3-源码分析系列2：启动脚本原理的分析

于 2018-09-18 22:14:00 发布
阅读量168 收藏
点赞数
文章标签： 大数据 python java
原文链接：https://my.oschina.net/qiangzigege/blog/2054888
版权

2019独角兽企业重金招聘Python工程师标准>>> hot3.png

上一节，我们把二进制和源码都下载好了，这一把，我们来启动脚本的研究，看看究竟是如何把RM和NM启动起来的！

-------------启动脚本是 ./sbin/start-yarn.sh 那就让我们来分析一下

不重要的不说，我们主要看yarn这个文件里核心的


# figure out which class to run
if [ "$COMMAND" = "classpath" ] ; then
  if [ "$#" -gt 0 ]; then
    CLASS=org.apache.hadoop.util.Classpath
  else
    if $cygwin; then
      CLASSPATH=$(cygpath -p -w "$CLASSPATH" 2>/dev/null)
    fi
    echo $CLASSPATH
    exit 0
  fi
elif [ "$COMMAND" = "rmadmin" ] ; then
  CLASS='org.apache.hadoop.yarn.client.cli.RMAdminCLI'
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "scmadmin" ] ; then
  CLASS='org.apache.hadoop.yarn.client.SCMAdmin'
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "application" ] || 
     [ "$COMMAND" = "applicationattempt" ] || 
     [ "$COMMAND" = "container" ]; then
  CLASS=org.apache.hadoop.yarn.client.cli.ApplicationCLI
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
  set -- $COMMAND $@
elif [ "$COMMAND" = "node" ] ; then
  CLASS=org.apache.hadoop.yarn.client.cli.NodeCLI
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "queue" ] ; then
  CLASS=org.apache.hadoop.yarn.client.cli.QueueCLI
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "resourcemanager" ] ; then
  CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/rm-config/log4j.properties
  CLASS='org.apache.hadoop.yarn.server.resourcemanager.ResourceManager'
  YARN_OPTS="$YARN_OPTS $YARN_RESOURCEMANAGER_OPTS"
  if [ "$YARN_RESOURCEMANAGER_HEAPSIZE" != "" ]; then
    JAVA_HEAP_MAX="-Xmx""$YARN_RESOURCEMANAGER_HEAPSIZE""m"
  fi
elif [ "$COMMAND" = "historyserver" ] ; then
  echo "DEPRECATED: Use of this command to start the timeline server is deprecated." 1>&2
  echo "Instead use the timelineserver command for it." 1>&2
  CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/ahs-config/log4j.properties
  CLASS='org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer'
  YARN_OPTS="$YARN_OPTS $YARN_HISTORYSERVER_OPTS"
  if [ "$YARN_HISTORYSERVER_HEAPSIZE" != "" ]; then
    JAVA_HEAP_MAX="-Xmx""$YARN_HISTORYSERVER_HEAPSIZE""m"
  fi
elif [ "$COMMAND" = "timelineserver" ] ; then
  CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/timelineserver-config/log4j.properties
  CLASS='org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer'
  YARN_OPTS="$YARN_OPTS $YARN_TIMELINESERVER_OPTS"
  if [ "$YARN_TIMELINESERVER_HEAPSIZE" != "" ]; then
    JAVA_HEAP_MAX="-Xmx""$YARN_TIMELINESERVER_HEAPSIZE""m"
  fi
elif [ "$COMMAND" = "sharedcachemanager" ] ; then
  CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/scm-config/log4j.properties
  CLASS='org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager'
  YARN_OPTS="$YARN_OPTS $YARN_SHAREDCACHEMANAGER_OPTS"
  if [ "$YARN_SHAREDCACHEMANAGER_HEAPSIZE" != "" ]; then
    JAVA_HEAP_MAX="-Xmx""$YARN_SHAREDCACHEMANAGER_HEAPSIZE""m"
  fi
elif [ "$COMMAND" = "nodemanager" ] ; then
  CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties
  CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager'
  YARN_OPTS="$YARN_OPTS -server $YARN_NODEMANAGER_OPTS"
  if [ "$YARN_NODEMANAGER_HEAPSIZE" != "" ]; then
    JAVA_HEAP_MAX="-Xmx""$YARN_NODEMANAGER_HEAPSIZE""m"
  fi
elif [ "$COMMAND" = "proxyserver" ] ; then
  CLASS='org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer'
  YARN_OPTS="$YARN_OPTS $YARN_PROXYSERVER_OPTS"
  if [ "$YARN_PROXYSERVER_HEAPSIZE" != "" ]; then
    JAVA_HEAP_MAX="-Xmx""$YARN_PROXYSERVER_HEAPSIZE""m"
  fi
elif [ "$COMMAND" = "version" ] ; then
  CLASS=org.apache.hadoop.util.VersionInfo
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "jar" ] ; then
  CLASS=org.apache.hadoop.util.RunJar
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "logs" ] ; then
  CLASS=org.apache.hadoop.yarn.client.cli.LogsCLI
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "daemonlog" ] ; then
  CLASS=org.apache.hadoop.log.LogLevel
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "cluster" ] ; then
  CLASS=org.apache.hadoop.yarn.client.cli.ClusterCLI
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
else
  CLASS=$COMMAND
fi

这个就是主类: org.apache.hadoop.yarn.server.resourcemanager.ResourceManager

对应的NodeManager你懂的

 

启动的时候，如果报错：

machine2: Error: JAVA_HOME is not set and could not be found.
machine1: Error: JAVA_HOME is not set and could not be found.

直接修改/root/hadoop-2.7.3/etc/hadoop/hadoop-env.sh里面，添加

#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/root/jdk1.8.0_181

就可以正常启动了

下面是resourceManager的启动命令

/root/jdk1.8.0_181/bin/java -Dproc_resourcemanager -Xmx90m -Dhadoop.log.dir=/root/hadoop-2.7.3/logs -Dyarn.log.dir=/root/hadoop-2.7.3/logs -Dhadoop.log.file=yarn-root-resourcemanager-machine0.log -Dyarn.log.file=yarn-root-resourcemanager-machine0.log -Dyarn.home.dir= -Dyarn.id.str=root -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/root/hadoop-2.7.3/lib/native -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/root/hadoop-2.7.3/logs -Dyarn.log.dir=/root/hadoop-2.7.3/logs -Dhadoop.log.file=yarn-root-resourcemanager-machine0.log -Dyarn.log.file=yarn-root-resourcemanager-machine0.log -Dyarn.home.dir=/root/hadoop-2.7.3 -Dhadoop.home.dir=/root/hadoop-2.7.3 -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/root/hadoop-2.7.3/lib/native -classpath /root/hadoop-2.7.3/etc/hadoop:/root/hadoop-2.7.3/etc/hadoop:/root/hadoop-2.7.3/etc/hadoop:/root/hadoop-2.7.3/share/hadoop/common/lib/*:/root/hadoop-2.7.3/share/hadoop/common/*:/root/hadoop-2.7.3/share/hadoop/hdfs:/root/hadoop-2.7.3/share/hadoop/hdfs/lib/*:/root/hadoop-2.7.3/share/hadoop/hdfs/*:/root/hadoop-2.7.3/share/hadoop/yarn/lib/*:/root/hadoop-2.7.3/share/hadoop/yarn/*:/root/hadoop-2.7.3/share/hadoop/mapreduce/lib/*:/root/hadoop-2.7.3/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/contrib/capacity-scheduler/*.jar:/contrib/capacity-scheduler/*.jar:/root/hadoop-2.7.3/share/hadoop/yarn/*:/root/hadoop-2.7.3/share/hadoop/yarn/lib/*:/root/hadoop-2.7.3/etc/hadoop/rm-config/log4j.properties org.apache.hadoop.yarn.server.resourcemanager.ResourceManager

下面是nodemanager的启动命令

/root/jdk1.8.0_181/bin/java -Dproc_nodemanager -Xmx1000m -Dhadoop.log.dir=/root/hadoop-2.7.3/logs -Dyarn.log.dir=/root/hadoop-2.7.3/logs -Dhadoop.log.file=yarn-root-nodemanager-machine1.log -Dyarn.log.file=yarn-root-nodemanager-machine1.log -Dyarn.home.dir= -Dyarn.id.str=root -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/root/hadoop-2.7.3/lib/native -Dyarn.policy.file=hadoop-policy.xml -server -Dhadoop.log.dir=/root/hadoop-2.7.3/logs -Dyarn.log.dir=/root/hadoop-2.7.3/logs -Dhadoop.log.file=yarn-root-nodemanager-machine1.log -Dyarn.log.file=yarn-root-nodemanager-machine1.log -Dyarn.home.dir=/root/hadoop-2.7.3 -Dhadoop.home.dir=/root/hadoop-2.7.3 -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/root/hadoop-2.7.3/lib/native -classpath /root/hadoop-2.7.3/etc/hadoop:/root/hadoop-2.7.3/etc/hadoop:/root/hadoop-2.7.3/etc/hadoop:/root/hadoop-2.7.3/share/hadoop/common/lib/*:/root/hadoop-2.7.3/share/hadoop/common/*:/root/hadoop-2.7.3/share/hadoop/hdfs:/root/hadoop-2.7.3/share/hadoop/hdfs/lib/*:/root/hadoop-2.7.3/share/hadoop/hdfs/*:/root/hadoop-2.7.3/share/hadoop/yarn/lib/*:/root/hadoop-2.7.3/share/hadoop/yarn/*:/root/hadoop-2.7.3/share/hadoop/mapreduce/lib/*:/root/hadoop-2.7.3/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/contrib/capacity-scheduler/*.jar:/root/hadoop-2.7.3/share/hadoop/yarn/*:/root/hadoop-2.7.3/share/hadoop/yarn/lib/*:/root/hadoop-2.7.3/etc/hadoop/nm-config/log4j.properties org.apache.hadoop.yarn.server.nodemanager.NodeManager

然后根据这个命令修改成debug命令就行了

转载于:https://my.oschina.net/qiangzigege/blog/2054888

weixin_34319640
关注
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值