hadoop3.1.1集群安装问题

这里不讲步骤，只讲遇到问题
首先配置几个-site.xml
hadoop3端口号默认的改了
Namenode 端口:
50470 --> 9871
50070 --> 9870
8020 --> 9820

Secondary NN 端口:
50091 --> 9869
50090 --> 9868

Datanode 端口:
50020 --> 9867
50010 --> 9866
50475 --> 9865
50075 --> 9864

namenode可以配置成hadoop2的50070
hdfs-site.xml

   <!-- 指定上述命名空间下管理的主备namenode节点 -->
   <property>
      <name>dfs.ha.namenodes.iknow</name>
      <value>nn1,nn2</value>
   </property>
   <!-- 指定该命名空间下的namenode节点RPC服务地址 -->
   <property>
      <name>dfs.namenode.rpc-address.iknow.nn1</name>
      <value>hadoop-master-1:8020</value>
   </property>
   <property>
      <name>dfs.namenode.rpc-address.iknow.nn2</name>
      <value>hadoop-master-2:8020</value>
   </property>
   <!-- 指定该命名空间下的namenode节点http服务地址 -->
   <property>
      <name>dfs.namenode.http-address.iknow.nn1</name>
      <value>hadoop-master-1:50070</value>
   </property>
   <property>
      <name>dfs.namenode.http-address.iknow.nn2</name>
      <value>hadoop-master-2:50070</value>
   </property>

hadoop3可以n个namenode

配置resourcemanage高可用
第一次这样配置。进入两个8088页面都会自动跳到启动的那个。

yarn-site.xml

    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
    </property>

    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>hadoop-master-1</value>
    </property>

    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>hadoop-master-2</value>
    </property>

    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
    </property>

    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>
    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>hadoop-slave-61:3181,hadoop-slave-60:3181,hadoop-slave-59:3181</value>
        <description>For multiple zk services, separate them with comma</description>
    </property>

    <property>
          <name>yarn.resourcemanager.cluster-id</name>
          <value>yarn-ha</value>
    </property>

格式化之前要先格式化zk zkfc -formatZK，格式完后要将namenode元数据信息复制到其他namenod机器的namenode中

slave文件变成worker，这个就升高到了人权问题上了

测试跑任务

hadoop jar /home/hadoop/softwares/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /input/output  

报错

Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>

mapred-site.xml中添加上面的配置文件${full path of your hadoop distribution directory}改成hadoophome的目录
网上说还要配置

<property>
  <name>mapreduce.application.classpath</name>
      <value>
       /home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/etc/*,
           /home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/etc/hadoop/*,
           /home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/lib/*,
           /home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/common/*,
           /home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/common/lib/*,
           /home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/mapreduce/*,
           /home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/mapreduce/lib-examples/*,
           /home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/hdfs/*,
           /home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/hdfs/lib/*,
           /home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/yarn/*,
           /home/tx/opt/Hadoop/hadoop-3.0.0-alpha2/share/hadoop/yarn/lib/*,
      </value>
</property>

我试了，其实不需要

然后又报一个错，搞了半天

2018-11-23 10:02:14,145 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:178)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:979)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
	at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1761)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort(MRClientService.java:177)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:159)
	... 14 more
2018-11-23 10:02:14,148 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException

研究很久，通过看源码发现MRAppMaster的端口没有从配置文件中找到，网上找的配置resourcemanage高可用的xml少了一个配置，

 <property>
    <name>yarn.resourcemanager.webapp.address.rm1</name>
    <value>hadoop-master-50:8088</value>
</property>

<property>
    <name>yarn.resourcemanager.webapp.address.rm2</name>
    <value>hadoop-master-51:8088</value>
</property>

这个我就奇怪了，这个是默认的，而且8088也能访问，为什么还要配置文件找它。
这个加上去 MR程序也能跑过了。