黑猴子的家：手动故障转移HDFS-HA

1、规划集群

hadoop102	hadoop103	hadoop104
NameNode	NameNode	""
JournalNode	JournalNode	JournalNode
DataNode	DataNode	DataNode

2、修改配置文件

core-site.xml

<configuration>

     <!-- 把两个NameNode的地址组装成一个集群mycluster -->
     <property>
          <name>fs.defaultFS</name>
          <value>hdfs://mycluster</value>
     </property>
     
     <!-- 指定hadoop运行时产生文件的存储目录 -->
     <property>
          <name>hadoop.tmp.dir</name>
          <value>/opt/module/HA/hadoop-2.7.2/data</value>
     </property>
     
     <!-- 声明journalnode服务器存储目录-->
     <property>
          <name>dfs.journalnode.edits.dir</name>
          <value>/opt/module/HA/hadoop-2.7.2/data/jn/mycluster</value>
     </property>
     
</configuration>

hdfs-site.xml

<configuration>

    <!-- 完全分布式集群名称 -->
    <property>
        <name>dfs.nameservices</name>
        <value>mycluster</value>
    </property>
    
    <!-- 集群中NameNode节点都有哪些 -->
    <property>
        <name>dfs.ha.namenodes.mycluster</name>
        <value>nn1,nn2</value>
    </property>
    
    <!-- nn1的RPC通信地址 -->
    <property>
        <name>dfs.namenode.rpc-address.mycluster.nn1</name>
        <value>hadoop102:9000</value>
    </property>
    
    <!-- nn2的RPC通信地址 -->
    <property>
        <name>dfs.namenode.rpc-address.mycluster.nn2</name>
        <value>hadoop103:9000</value>
    </property>
    
    <!-- nn1的http通信地址 -->
    <property>
        <name>dfs.namenode.http-address.mycluster.nn1</name>
        <value>hadoop102:50070</value>
    </property>
    
    <!-- nn2的http通信地址 -->
    <property>
        <name>dfs.namenode.http-address.mycluster.nn2</name>
        <value>hadoop103:50070</value>
    </property>
    
    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://hadoop102:8485;hadoop103:8485;hadoop104:8485/mycluster</value>
    </property>
    
    <!-- 配置隔离机制，即同一时刻只能有一台服务器对外响应 -->
    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence</value>
    </property>
    
    <!-- 使用隔离机制时需要ssh无秘钥登录-->
    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/home/yinggu/.ssh/id_rsa</value>
    </property>
    
    <!-- 关闭权限检查-->
    <property>
        <name>dfs.permissions.enable</name>
        <value>false</value>
    </property>
    
    <!-- 访问代理类：client，mycluster，active配置失败自动切换实现方式-->
    <property>
        <name>dfs.client.failover.proxy.provider.mycluster</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    
    <!-- 延长等待时间 -->
    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence</value>
    </property>
    
    <property>
        <name>dfs.ha.fencing.ssh.connect-timeout</name>
        <value>60000</value>
    </property>

</configuration>

slaves

hadoop102
hadoop103
hadoop104

hadoop-env.sh

export JAVA_HOME=/opt/module/HA/jdk1.8.0_144

yarn-env.sh

export JAVA_HOME=/opt/module/HA/jdk1.8.0_144

mapred-env.sh

export JAVA_HOME=/opt/module/HA/jdk1.8.0_144

3、分发hadoop

[yinggu@hadoop102 HA]$ scp -r hadoop-2.7.2/  yinggu@hadoop103:/opt/module/HA/
[yinggu@hadoop102 HA]$ scp -r hadoop-2.7.2/  yinggu@hadoop104:/opt/module/HA/

4、启动各个机器的JN节点，检查一下是否启动成功

[yinggu@hadoop102 hadoop-2.7.2]$ sbin/hadoop-daemon.sh start journalnode
[yinggu@hadoop103 hadoop-2.7.2]$ sbin/hadoop-daemon.sh start journalnode
[yinggu@hadoop104 hadoop-2.7.2]$ sbin/hadoop-daemon.sh start journalnode

5、在hadoop102节点上，格式化主NameNode

[yinggu@hadoop102 hadoop-2.7.2]$ bin/hdfs namenode -format

6、在hadoop102节点上，启动格式化后的主NameNode

[yinggu@hadoop102 hadoop-2.7.2]$ sbin/hadoop-daemon.sh start namenode

7、在hadoop103节点上，同步namenode元数据信息

备用NameNode同步主NameNode的元数据信息

[yinggu@hadoop103 hadoop-2.7.2]$ bin/hdfs namenode -bootstrapStandby
尖叫提示：在备用NameNode所在机器执行

8、在hadoop103节点上，启动备用NameNode

[yinggu@hadoop103 hadoop-2.7.2]$ sbin/hadoop-daemon.sh start namenode

9、启动所有DataNode

[yinggu@hadoop102 hadoop-2.7.2]$ sbin/hadoop-daemons.sh start datanode
[yinggu@hadoop103 hadoop-2.7.2]$ sbin/hadoop-daemons.sh start datanode
[yinggu@hadoop104 hadoop-2.7.2]$ sbin/hadoop-daemons.sh start datanode

10、此时两个namenode都为Standby模式，切换主NameNode为Active

[yinggu@hadoop102 hadoop-2.7.2]$ bin/hdfs haadmin -transitionToActive nn1
尖叫提示：nn1是我们在hdfs-site.xml配置文件中配置的名称

11、切换备用NameHadoop103为Active

[yinggu@hadoop102 hadoop-2.7.2]$ bin/hdfs haadmin -transitionToStandby nn1
[yinggu@hadoop102 hadoop-2.7.2]$ bin/hdfs haadmin -transitionToActive nn2
注意：先把nn1设置为Standby模式，如果此时主NameNode无法联系，则无法切换备用NameNode为Active

12、关闭HDFS

[yinggu@hadoop102 hadoop-2.7.2]$ sbin/stop-dfs.sh

13、启动HDFS

[yinggu@hadoop102 hadoop-2.7.2]$ sbin/start-dfs.sh