搭建自动切换的HA

----------------------------------------------------------------------------------------------
2.搭建自动切换的ha(比手工切换ha集群，多了zookeeper集群)
----------------------------------------------------------------------------------------------
zookeeper：hadoop2、hadoop3、hadoop4
namenode：haoop0和hadoop1
datanode：hadoop2、hadoop3、hadoop4
journalnode：haoop0、hadoop1、hadoop2
resourcemanager：hadoop0
nodemanager：hadoop2、hadoop3、hadoop4


2.0 搭建zookeeper集群，并启动
 2.0.1  在hadoop2上解压缩，重命名为zookeeper，把conf/zoo_sample.cfg重命名为conf/zoo.cfg
  修改文件conf/zoo.cfg
  (1)dataDir=/usr/local/zookeeper/data
  (2)增加以下内容
     server.2=hadoop2:2888:3888（对外开放两个端口）
server.3=hadoop3:2888:3888
server.4=hadoop4:2888:3888
（2,3,4分别表示hadoop2，hadoop3和hadoop4在zookeeper集群中的id）
  创建目录mkdir zookeeper/data
  写入文件echo 2 zookeeper/data/myid
  
  复制zookeeper文件夹到hadoop3、hadoop4上
  scp -rq zookeeper  hadoop3:/usr/local
  scp -rq zookeeper  hadoop4:/usr/local
  
  在hadoop3上执行命令echo 3 zookeeper/data/myid
  在hadoop4上执行命令echo 4 zookeeper/data/myid
 2.0.2 启动
  在hadoop2、hadoop3、haodop4上，分别执行命令zookeeper/bin/zkServer.sh start
 2.0.3 验证
  执行命令zookeeper/bin/zkCli.sh 
  进入后执行ls / 
2.1 配置文件(hadoop-env.sh、core-site.xml、hdfs-site.xml、yarn-site.xml、mapred-site.xml、slaves)
2.1.1 hadoop-env.sh
  export JAVA_HOME=/usr/local/jdk
2.1.2 core-site.xml


<property>
<name>fs.defaultFS</name>
<value>hdfs://cluster1</value>
</property>


<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>


#指定zookeeper集群
<property>
<name>ha.zookeeper.quorum</name>
<value>hadoop2:2181,hadoop3:2181,hadoop4:2181</value>
</property>


2.1.3 hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>


<property>
<name>dfs.nameservices</name>
<value>cluster1</value>
</property>


<property>
<name>dfs.ha.namenodes.cluster1</name>
<value>hadoop101,hadoop102</value>
</property>


<property>
<name>dfs.namenode.rpc-address.cluster1.hadoop101</name>
<value>hadoop0:9000</value>
</property>


<property>
<name>dfs.namenode.http-address.cluster1.hadoop101</name>
<value>hadoop0:50070</value>
</property>


<property>
<name>dfs.namenode.rpc-address.cluster1.hadoop102</name>
<value>hadoop1:9000</value>
</property>


<property>
<name>dfs.namenode.http-address.cluster1.hadoop102</name>
<value>hadoop1:50070</value>
</property>


<property>
<name>dfs.ha.automatic-failover.enabled.cluster1</name>
<value>true</value>
</property>


<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoop0:8485;hadoop1:8485;hadoop2:8485/cluster1</value>
</property>


<property>
<name>dfs.journalnode.edits.dir</name>
<value>/usr/local/hadoop/tmp/journal</value>
</property>


<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>


<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>


<property>
<name>dfs.client.failover.proxy.provider.cluster1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>


2.1.4 yarn-site.xml


<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop0</value>
</property>


<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>


2.1.5 mapred-site.xml


<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>


2.1.6 slaves
hadoop2
hadoop3
hadoop4


2.1.7 删除其他节点的hadoop文件夹，然后把hadoop0上的hadoop文件夹复制到hadoop1、hadoop2、hadoop3、hadoop4节点
2.2 格式化zk集群
  在hadoop0上执行hadoop/bin/hdfs zkfc -formatZK，应该会在Zookeeper看到输出结果。
2.3 启动journalnode集群
  在hadoop0、hadoop1、hadoop2上分别执行hadoop/sbin/hadoop-daemon.sh start journalnode
2.4 格式化namenode、启动namenode
  在hadoop0上执行hadoop/bin/hdfs namenode -format
  在hadoop0上分别执行hadoop/sbin/hadoop-daemon.sh start namenode
  在hadoop1上执行hadoop/bin/hdfs namenode -bootstrapStandby
  在hadoop1上分别执行hadoop/sbin/hadoop-daemon.sh start namenode
2.5 启动datanode
  在hadoop0上分别执行hadoop/sbin/hadoop-daemons.sh start datanode
2.6 启动ZKFC（哪个节点上有namenode，在哪个节点启动）
  在hadoop0、hadoop1上 启动zkfc，执行命令hadoop/sbin/hadoop-daemon.sh start zkfc
2.7 启动resourcemanager和nodemanager
  在hadoop0上执行 hadoop/sbin/start-yarn.sh start resourcemanager
  hadoop/sbin/start-yarn.sh start nodemanager


通过杀死hadoop0和hadoop1上的namenode进程来测试active和standby的切换。


总结：
  自动切换比手工切换多出来的
  (1)配置上core-site.xml增加了配置项ha.zookeeper.quorum；hdfs-site.xml中把dfs.ha.automatic-failover.enabled.cluster1改为true
  (2)操作上格式化zk，执行命令bin/hdfs zkfc -formatZK；启动zkfc，执行命令sbin/hadoop-daemon.sh start zkfc