/**
* author:it-WScheng
* 注:装载请注明出处
*/
Hadoop2.X HA 搭建
首先准备四台机器:node1,node2,node3,node4
1.编辑core-site.xml文件
vi opt/hadoop/etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://myhadoop</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>node1:2181,node2:2181,node3:2181</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop2</value>
</property>
</configuration>
2.编辑hdfs-site.xml
vi opt/hadoop/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.nameservices</name>
<value>myhadoop</value>
</property>
<property>
<name>dfs.ha.namenodes.myhadoop</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.myhadoop.nn1</name>
<value>node1:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.myhadoop.nn2</name>
<value>node2:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.myhadoop.nn1</name>
<value>node1:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.myhadoop.nn2</name>
<value>node2:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://node2:8485;node3:8485;node4:8485/myhadoop</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.myhadoop
</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.
ConfiguredFailoverProxyProvider
</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/opt/jn/data</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
</configuration>
3.编辑mapred-site.xml
vi opt/hadoop/etc/hadoop/mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
4.编辑yarn-site.xml
vi opt/hadoop/etc/hadoop/yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>node1</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class
</name>
<value>org.apache.hadoop.mapred.ShuffleHandler
</value>
</property>
</configuration>
5.准备zookeeper
a)三台zookeeper:node1,node2,node3
可以将zookeeper-3.4.10解压在/opt/下
b)配置zoo.cfg文件
1. cd /opt/zookeeper-3.4.10/conf
2.cp -d zoo_sample.cfg zoo.cfg
3.vi zoo.cfg
① 修改dataDir=/opt/zookeeper
② server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888
c)在dataDir目录中创建一个myid的文件
vi /opt/zookeeper/myid
每个节点的dataDir都要创建,文件的内容分别是1,2,3
6.配置hadoop中的slaves(写上datenode)
vi opt/hadoop/etc/hadoop/slaves
node2
node3
node4
7.启动三个zookeeper:./zkServer.sh start
8.启动三个journalNode:./hadoop-daemon.sh start journalnode
9.在其中一个namenode上格式化:
./hdfs namenode -format
10.把刚刚格式化之后的元数据拷贝到另外一个 namenode上
a)启动刚刚格式化的namenode
b)在没有格式化的namenode上执行:
./hdfs namenode -bootstrapStandby
c) 启动第二个namenode
./hdfs zkfc -formatZk