Hadoop---HA安装

配置细节

---------------

       1.s101和s108具有完全一致的配置，尤其是ssh.

          将S108 authorized_keys cat 到S101中，并分发到其它node

      2.配置nameservice

              [hdfs-site.xml]

              <property>

                     <name>dfs.nameservices</name>

                     <value>mycluster</value>

              </property>

         3. dfs.ha.namenodes.[nameservice ID]

              [hdfs-site.xml]

              <!-- myucluster下的名称节点两个id -->

              <property>

                     <name>dfs.ha.namenodes.mycluster</name>

                     <value>nn1,nn2</value>

              </property>

   4. dfs.namenode.rpc-address.[nameservice ID].[name node ID]

              [hdfs-site.xml]

              配置每个nn的rpc地址。

              <property>

                     <name>dfs.namenode.rpc-address.mycluster.nn1</name>

                     <value>s101:8020</value>

              </property>

              <property>

                     <name>dfs.namenode.rpc-address.mycluster.nn2</name>

                     <value>s108:8020</value>

              </property>

      5. dfs.namenode.http-address.[nameservice ID].[name node ID]

              配置webui端口

              [hdfs-site.xml]

              <property>

                     <name>dfs.namenode.http-address.mycluster.nn1</name>

                     <value>s101:50070</value>

              </property>

              <property>

                     <name>dfs.namenode.http-address.mycluster.nn2</name>

                     <value>s108:50070</value>

              </property>

     6. dfs.namenode.shared.edits.dir

              名称节点共享编辑目录.

              [hdfs-site.xml]

              <property>

                     <name>dfs.namenode.shared.edits.dir</name>

                     <value>qjournal://s102:8485;s103:8485;s104:8485/mycluster</value>

              </property>

  7. dfs.client.failover.proxy.provider.[nameservice ID]

              java类，client使用它判断哪个节点是激活态。

              [hdfs-site.xml]

              <property>

                     <name>dfs.client.failover.proxy.provider.mycluster</name>

                     <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

              </property>

        8. dfs.ha.fencing.methods

              脚本列表或者java类，在容灾保护激活态的nn.

              [hdfs-site.xml]

              <property>

                     <name>dfs.ha.fencing.methods</name>

                     <value>

                                   sshfence

                                   shell(/bin/true)

                     </value>

              </property>

              <property>

                     <name>dfs.ha.fencing.ssh.private-key-files</name>

                     <value>/root/.ssh/id_rsa</value>

              </property>

 

       9. s.defaultFS

              配置hdfs文件系统名称服务。

              [core-site.xml]

              <property>

                     <name>fs.defaultFS</name>

                     <value>hdfs://mycluster</value>

              </property>

      

       10. dfs.journalnode.edits.dir

              配置JN存放edit的本地路径。

              [hdfs-site.xml]

              <property>

                     <name>dfs.journalnode.edits.dir</name>

                     <value>/root/hadoop/journal</value>

              </property>

  

11. 配置name的存放路径

      <property>

           <name>dfs.namenode.name.dir</name>

           <value>/root/hadoop/dfs/name</value>

      </property>

 

部署细节

----------------

       1.在jn节点分别启动jn进程

              $>hadoop-daemon.sh start journalnode

 

       2.启动jn之后，在两个NN之间进行disk元数据同步

              a)如果是全新集群，先format文件系统,只需要在一个nn上执行。

                     [s101]

                     $>hadoop namenode -format

             

              b)如果将非HA集群转换成HA集群，复制原NN的metadata到另一个nn.

                     1.步骤一

                            [s101]

                            $>scp -r /root/hadoop/dfs S108:/home/centos/hadoop/

 

                     2.步骤二

                            在新的nn(未格式化的nn)上运行一下命令，实现待命状态引导。

                            [S108]

                            $>hdfs namenode -bootstrapStandby           //需要s201为启动状态,提示是否格式化,选择N.

                           

                     3)在一个NN上执行以下命令，完成edit日志到jn节点的传输。

                            $>hdfs namenode -initializeSharedEdits

                            #查看s102,s103是否有edit数据.

 

                     4)启动所有节点.

                            [s101]

                            #hadoop-daemon.sh start namenode          //启动名称节点

                            #hadoop-daemons.sh start datanode           //启动所有数据节点

 

                            [s108]

                            $>hadoop-daemon.sh start namenode        //启动名称节点

                    

 

HA管理

-----------------

       $>hdfs haadmin -transitionToActive nn1                           //切成激活态

       $>hdfs haadmin -transitionToStandby nn1                        //切成待命态

       $>hdfs haadmin -transitionToActive --forceactive nn2//强行激活

       $>hdfs haadmin -failover nn1 nn2                             //模拟容灾演示,从nn1切换到nn2