前情提要:
搭建好Hadoop HA集群后,试着跑一个wordcount,
结果居然出了这个问题:
Container exited with a non-zero exit code 1. Error file: prelaunch.err.
捏🐴,怎么回事?之前伪分布式或者非高可用的完全分布式好像没有这个问题啊?
于是百度,噢,原来是mapred-site.xml和yarn-site.xml里要加上classpath
输入hadoop classpath,出来一大串
然后复制这一大串,到mapred-site.xml和yarn-site.xml上
mapred-site.xml:
<property>
<name>mapreduce.application.classpath</name>
<value>/home/hadoop/app/hadoop/etc/hadoop:/home/hadoop/app/hadoop/share/hadoop/common/lib/*:/home/hadoop/app/hadoop/share/hadoop/common/*:/home/hadoop/app/hadoop/share/hadoop/hdfs:/home/hadoop/app/hadoop/share/hadoop/hdfs/lib/*:/home/hadoop/app/hadoop/share/hadoop/hdfs/*:/home/hadoop/app/hadoop/share/hadoop/mapreduce/*:/home/hadoop/app/hadoop/share/hadoop/yarn:/home/hadoop/app/hadoop/share/hadoop/yarn/lib/*:/home/hadoop/app/hadoop/share/hadoop/yarn/*
</value>
</property>
yarn-site.xml
<property>
<name>yarn.application.classpath</name>
<value>/home/hadoop/app/hadoop/etc/hadoop:/home/hadoop/app/hadoop/share/hadoop/common/lib/*:/home/hadoop/app/hadoop/share/hadoop/common/*:/home/hadoop/app/hadoop/share/hadoop/hdfs:/home/hadoop/app/hadoop/share/hadoop/hdfs/lib/*:/home/hadoop/app/hadoop/share/hadoop/hdfs/*:/home/hadoop/app/hadoop/share/hadoop/mapreduce/*:/home/hadoop/app/hadoop/share/hadoop/yarn:/home/hadoop/app/hadoop/share/hadoop/yarn/lib/*:/home/hadoop/app/hadoop/share/hadoop/yarn/*
</value>
</property>
(value里改成自己出来的那个classpath)
’
‘
那么改完之后就完事了吗?
当然是并没有。
那么继续找,噢,原来是hadoop高版本需要配置yarn对外暴露的端口:
<!--配置rm2-->
<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>worker1:8132</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm2</name>
<value>worker1:8130</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>worker1:8188</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
<value>worker1:8131</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm2</name>
<value>worker1:8133</value>
</property>
<property>
<name>yarn.resourcemanager.ha.admin.address.rm2</name>
<value>worker1:23142</value>
</property>
<!--配置rm3-->
<property>
<name>yarn.resourcemanager.address.rm3</name>
<value>worker2:8132</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm3</name>
<value>worker2:8130</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm3</name>
<value>worker2:8188</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm3</name>
<value>worker2:8131</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm3</name>
<value>worker2:8133</value>
</property>
<property>
<name>yarn.resourcemanager.ha.admin.address.rm3</name>
<value>worker2:23142</value>
</property>
那么我个人的完整yarn-site.xml如下:
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- 开启RM高可用 -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<!-- 指定RM的cluster id -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yrc</value>
</property>
<!-- 指定RM的名字 -->
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm2,rm3</value>
</property>
<!-- 分别指定RM的地址 -->
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>worker1</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm3</name>
<value>worker2</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://master:19888/jobhistory/logs</value>
</property>
<!--配置rm2-->
<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>worker1:8132</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm2</name>
<value>worker1:8130</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>worker1:8188</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
<value>worker1:8131</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm2</name>
<value>worker1:8133</value>
</property>
<property>
<name>yarn.resourcemanager.ha.admin.address.rm2</name>
<value>worker1:23142</value>
</property>
<!--配置rm3-->
<property>
<name>yarn.resourcemanager.address.rm3</name>
<value>worker2:8132</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm3</name>
<value>worker2:8130</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm3</name>
<value>worker2:8188</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm3</name>
<value>worker2:8131</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm3</name>
<value>worker2:8133</value>
</property>
<property>
<name>yarn.resourcemanager.ha.admin.address.rm3</name>
<value>worker2:23142</value>
</property>
<!-- 指定zk集群地址 -->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>master:2181,worker1:2181,worker2:2181</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>86400</value>
</property>
<!-- 启用自动恢复 -->
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<!-- 制定resourcemanager的状态信息存储在zookeeper集群上 -->
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
<property>
<name>yarn.application.classpath</name>
<value>/home/hadoop/app/hadoop/etc/hadoop:/home/hadoop/app/hadoop/share/hadoop/common/lib/*:/home/hadoop/app/hadoop/share/hadoop/common/*:/home/hadoop/app/hadoop/share/hadoop/hdfs:/home/hadoop/app/hadoop/share/hadoop/hdfs/lib/*:/home/hadoop/app/hadoop/share/hadoop/hdfs/*:/home/hadoop/app/hadoop/share/hadoop/mapreduce/*:/home/hadoop/app/hadoop/share/hadoop/yarn:/home/hadoop/app/hadoop/share/hadoop/yarn/lib/*:/home/hadoop/app/hadoop/share/hadoop/yarn/*
</value>
</property>
</configuration>
修改完毕,问题总算解决了。
网上那些让改classpath的,还是改环境变量的,还是都改的,真的让我改到怀疑人生~~