这里不讲步骤,只讲遇到问题
首先配置几个-site.xml
hadoop3端口号默认的改了
Namenode 端口:
50470 --> 9871
50070 --> 9870
8020 --> 9820
Secondary NN 端口:
50091 --> 9869
50090 --> 9868
Datanode 端口:
50020 --> 9867
50010 --> 9866
50475 --> 9865
50075 --> 9864
namenode可以配置成hadoop2的50070
hdfs-site.xml
<!-- 指定上述命名空间下管理的主备namenode节点 -->
<property>
<name>dfs.ha.namenodes.iknow</name>
<value>nn1,nn2</value>
</property>
<!-- 指定该命名空间下的namenode节点RPC服务地址 -->
<property>
<name>dfs.namenode.rpc-address.iknow.nn1</name>
<value>hadoop-master-1:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.iknow.nn2</name>
<value>hadoop-master-2:8020</value>
</property>
<!-- 指定该命名空间下的namenode节点http服务地址 -->
<property>
<name>dfs.namenode.http-address.iknow.nn1</name>
<value>hadoop-master-1:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.iknow.nn2</name>
<value>hadoop-master-2:50070</value>
</property>
hadoop3可以n个namenode
配置resourcemanage高可用
第一次这样配置。进入两个8088页面都会自动跳到启动的那个。
yarn-site.xml
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>hadoop-master-1</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>hadoop-master-2</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>hadoop-slave-61:3181,hadoop-slave-60:3181,hadoop-slave-59:3181</value>
<description>For multiple zk services, separate them with comma</description>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yarn-ha</value>
</property>
格式化之前要先格式化zk zkfc -formatZK,格式完后要将namenode元数据信息复制到其他namenod机器的namenode中
slave文件变成worker,这个就升高到了人权问题上了
测试跑任务
hadoop jar /home/hadoop/softwares/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /input/output
报错
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
mapred-site.xml中添加上面的配置文件${full path of your hadoop distribution directory}改成hadoophome的目录
网上说还要配置
<property>
<name>mapreduce.application.classpath</name>
<value>
/home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/etc/*,
/home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/etc/hadoop/*,
/home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/lib/*,
/home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/common/*,
/home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/common/lib/*,
/home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/mapreduce/*,
/home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/mapreduce/lib-examples/*,
/home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/hdfs/*,
/home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/hdfs/lib/*,
/home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/yarn/*,
/home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/yarn/lib/*,
</value>
</property>
我试了,其实不需要
然后又报一个错,搞了半天
2018-11-23 10:02:14,145 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:178)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:979)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1761)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort(MRClientService.java:177)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:159)
... 14 more
2018-11-23 10:02:14,148 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
研究很久,通过看源码发现MRAppMaster的端口没有从配置文件中找到,网上找的配置resourcemanage高可用的xml少了一个配置,
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>hadoop-master-50:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>hadoop-master-51:8088</value>
</property>
这个我就奇怪了,这个是默认的,而且8088也能访问,为什么还要配置文件找它。
这个加上去 MR程序也能跑过了。