背景:
最近新扩容一批nodemanager服务器,由于没有申请额外的磁盘,只有默认的40GB空间,当执行较大的mapreduce任务时app_cache/file_cache容易将磁盘写满,因此需要让ResourceManager 将mapreduce任务调度到有额外磁盘的nodemanager
基本步骤:
//增加下列配置到yarn-site.xml
<!-- node labels -->
<property>
<name>yarn.node-labels.fs-store.root-dir</name>
<value>hdfs://bj-e-hadoop-test01/yarn/store/node-labels/</value>
</property>
<property>
<name>yarn.node-labels.enabled</name>
<value>true</value>
</property>
//增加node labels
yarn rmadmin -addToClusterNodeLabels "withoutExtraDisks,withExtraDisks"
//关联node 和node labels
yarn rmadmin -replaceLabelsOnNode "bj-e-hadoop-test01-1-216=withExtraDisks bj-e-hadoop-test01-1-217=withoutExtraDisks bj-e-hadoop-test01-1-218=withoutExtraDisks"
参考资料
https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/NodeLabel.html