linux上部署hadoop集群 HA+Federation篇

介绍

本文继续介绍hadoop部署，来说如何搭建HA+Federation，HA的意思是高可用，Federation是联合的意思，也可以理解为负载均衡的意思。

 

正文

1 环境

10台机器：

四台namenode(一个集群，每两台做一个namespace，每个ns中的两个namenode作HA)，三台datanode，三台zookeeper（也可以省去这三台，把zookeeper daemon部署在其他机器上）。实际上还需要3台journalnode,但因为它比较轻量级，所以这里就把它部署在datanode上了。

为了方便，本文后面就用ns代替namespace。

2 配置文件修改

2.1 core-site.xml

修改ns1中的两台namenode的core-site.xml

    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/hadoop-2.6.0/tmp</value>
    </property>
    <property>
      <name>fs.defaultFS</name>
        <value>hdfs://cluster1</value>
        </property>
     <property>
        <name>ha.zookeeper.quorum</name>
           <value>10.9.214.167:2181,10.9.214.18:2181,10.9.214.211:2181</value>
            </property>

修改ns2中的两台namenode的core-site.xml

    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/hadoop-2.6.0/tmp</value>
    </property>
    <property>
      <name>fs.defaultFS</name>
        <value>hdfs://cluster2</value>
        </property>
     <property>
        <name>ha.zookeeper.quorum</name>
           <value>10.9.214.167:2181,10.9.214.18:2181,10.9.214.211:2181</value>
            </property>

2.2 hdfs-site.xml

修改ns1中的两台namenode的hdfs-site.xml

     <property>  
             <name>dfs.namenode.name.dir</name>  
                     <value>file:/opt/hadoop-2.6.0/hdfs/name</value>  
                         </property>  
    <property>  
            <name>dfs.datanode.data.dir</name>  
                    <value>file:/opt/hadoop-2.6.0/hdfs/data</value>  
                        </property>  
    <property>  
            <name>dfs.replication</name>  
                    <value>3</value>  
                        </property>

    <property>
      <name>dfs.nameservices</name>
        <value>cluster1,cluster2</value>
        </property>
    <property>
      <name>dfs.ha.namenodes.cluster1</name>
        <value>nn1,nn2</value>
        </property>
    <property>
      <name>dfs.namenode.rpc-address.cluster1.nn1</name>
        <value>10.9.214.151:8020</value>
        </property>
        <property>
          <name>dfs.namenode.rpc-address.cluster1.nn2</name>
            <value>10.9.214.15:8020</value>
            </property>
     <property>
       <name>dfs.namenode.http-address.cluster1.nn1</name>
         <value>10.9.214.151:50070</value>
         </property>
         <property>
           <name>dfs.namenode.http-address.cluster1.nn2</name>
             <value>10.9.214.15:50070</value>
             </property>
    <property>
      <name>dfs.ha.namenodes.cluster2</name>
        <value>nn3,nn4</value>
        </property>
    <property>
      <name>dfs.namenode.rpc-address.cluster2.nn3</name>
        <value>10.9.214.105:8020</value>
        </property>
        <property>
          <name>dfs.namenode.rpc-address.cluster2.nn4</name>
            <value>10.9.214.113:8020</value>
            </property>
     <property>
       <name>dfs.namenode.http-address.cluster2.nn3</name>
         <value>10.9.214.105:50070</value>
         </property>
         <property>
           <name>dfs.namenode.http-address.cluster2.nn4</name>
             <value>10.9.214.113:50070</value>
             </property>

     <property>
       <name>dfs.namenode.shared.edits.dir</name>
         <value>qjournal://10.9.214.158:8485;10.9.214.160:8485;10.9.214.149:8485/cluster1</value>
         </property>
     <property>
       <name>dfs.client.failover.proxy.provider.cluster1</name>
         <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
         </property>
     <property>
       <name>dfs.ha.fencing.methods</name>
         <value>sshfence</value>
         </property>
     <property>
       <name>dfs.ha.fencing.methods</name>
         <value>sshfence</value>
         </property>
     <property>
       <name>dfs.ha.fencing.ssh.private-key-files</name>
         <value>/home/hadoop/.ssh/id_rsa</value>
         </property>
     <property>
       <name>dfs.journalnode.edits.dir</name>
         <value>/opt/hadoop-2.6.0/journalnode</value>
         </property>
      <property>
         <name>dfs.ha.automatic-failover.enabled</name>
            <value>true</value>
             </property>

修改ns2中的两台namenode的hdfs-site.xml

     <property>  
             <name>dfs.namenode.name.dir</name>  
                     <value>file:/opt/hadoop-2.6.0/hdfs/name</value>  
                         </property>  
    <property>  
            <name>dfs.datanode.data.dir</name>  
                    <value>file:/opt/hadoop-2.6.0/hdfs/data</value>  
                        </property>  
    <property>  
            <name>dfs.replication</name>  
                    <value>3</value>  
                        </property>

    <property>
      <name>dfs.nameservices</name>
        <value>cluster1,cluster2</value>
        </property>
    <property>
      <name>dfs.ha.namenodes.cluster1</name>
        <value>nn1,nn2</value>
        </property>
    <property>
      <name>dfs.namenode.rpc-address.cluster1.nn1</name>
        <value>10.9.214.151:8020</value>
        </property>
        <property>
          <name>dfs.namenode.rpc-address.cluster1.nn2</name>
            <value>10.9.214.15:8020</value>
            </property>
     <property>
       <name>dfs.namenode.http-address.cluster1.nn1</name>
         <value>10.9.214.151:50070</value>
         </property>
         <property>
           <name>dfs.namenode.http-address.cluster1.nn2</name>
             <value>10.9.214.15:50070</value>
             </property>
    <property>
      <name>dfs.ha.namenodes.cluster2</name>
        <value>nn3,nn4</value>
        </property>
    <property>
      <name>dfs.namenode.rpc-address.cluster2.nn3</name>
        <value>10.9.214.105:8020</value>
        </property>
        <property>
          <name>dfs.namenode.rpc-address.cluster2.nn4</name>
            <value>10.9.214.113:8020</value>
            </property>
     <property>
       <name>dfs.namenode.http-address.cluster2.nn3</name>
         <value>10.9.214.105:50070</value>
         </property>
         <property>
           <name>dfs.namenode.http-address.cluster2.nn4</name>
             <value>10.9.214.113:50070</value>
             </property>

     <property>
       <name>dfs.namenode.shared.edits.dir</name>
         <value>qjournal://10.9.214.158:8485;10.9.214.160:8485;10.9.214.149:8485/cluster2</value>
         </property>
     <property>
       <name>dfs.client.failover.proxy.provider.cluster2</name>
         <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
         </property>
     <property>
       <name>dfs.ha.fencing.methods</name>
         <value>sshfence</value>
         </property>
     <property>
       <name>dfs.ha.fencing.methods</name>
         <value>sshfence</value>
         </property>
     <property>
       <name>dfs.ha.fencing.ssh.private-key-files</name>
         <value>/home/hadoop/.ssh/id_rsa</value>
         </property>
     <property>
       <name>dfs.journalnode.edits.dir</name>
         <value>/opt/hadoop-2.6.0/journalnode</value>
         </property>
      <property>
         <name>dfs.ha.automatic-failover.enabled</name>
            <value>true</value>
             </property>

2.3 mapred-site.xml

修改4台namenode上的mapred-site.xml

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>

 

2.4 yarn-site.xml

修改ns1的yarn-site.xml

<property>
<name>yarn.resourcemanager.address</name>
<value>10.9.214.151:18040</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>10.9.214.151:18030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>10.9.214.151:18088</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>10.9.214.151:18025</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>10.9.214.151:18141</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

修改ns2的yarn-site.xml

<property>
<name>yarn.resourcemanager.address</name>
<value>10.9.214.105:18040</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>10.9.214.105:18030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>10.9.214.105:18088</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>10.9.214.105:18025</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>10.9.214.105:18141</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
 

3 操作细节

3.1 启动zookeeper daemons

在三台zookeeper机器上执行

zkServer.sh start

3.2 格式化zookeeper

在ns1，集群2的各一个namenode上执行

hdfs zkfc -formatZK

3.3 启动journalnode

在三台journalnode机器上执行

hadoop-daemon.sh start journalnode

3.4 格式化namenode

在ns1和ns2中各一台namenode上执行

hdfs namenode -format -clusterId cluster

注意：clusterid 只有一个，因为四个namenode共用三个datanode，所以只有一个集群。

3.5 启动namenode

在ns1和ns2中各一台namenode上执行

hadoop-daemon.sh start namenode

验证

在浏览器中输入10.9.214.151：50070 可以看到hdfs状态

3.6 做namenode 数据同步到namenode standby

在ns1和ns2中另一台namenode上执行

hdfs namenode -bootstrapStandby

3.7 启动namenode standby上的namenode

hadoop-daemon.sh start namenode

3.8 启动datanode

在任意一台namenode上执行

hadoop-daemons.sh start datanode

3.9 启动zkfc

在每台namenode上执行

hadoop-daemon.sh --script hdfs start zkfc

验证

在任意一台namenode上执行以下命令分别查看各个节点状态

hdfs haadmin -ns cluster1 -getServiceState nn1
hdfs haadmin -ns cluster1 -getServiceState nn2
hdfs haadmin -ns cluster2 -getServiceState nn3
hdfs haadmin -ns cluster2 -getServiceState nn4

切换active和standby节点，看是否成功

hdfs haadmin -ns cluster1 -failover nn1 nn2

尝试关掉一个主namenode，看主备能否正常切换

尝试关掉一个datanode，看会发生什么（datanode和namenode之间是通过heartbeat每隔几分钟检测一次，若某个datanode没有反应了，namenode就会把这个datanode变为四节点，namenode每隔10分钟检测一次）

3.10 启动yarn

在ns1，ns2的各一个namenode上启动

start-yarn.sh

验证

在浏览器中输入http://10.9.214.151:18088 可看到application界面

查看hdfs文件系统内容：

hadoop dfs -ls /

拷贝一个文件到hadoop集群：

hadoop dfs -put core-site.xml /