【笔记】Hadoop-HA集群搭建

journalnode(3)
namenode(2)
datanode(3)
resourcemanager(2)
nodemanager(3)
富人版13台主机：
hadoop-ha-journalnode-a
hadoop-ha-journalnode-b
hadoop-ha-journalnode-c
hadoop-ha-namenode-a
hadoop-ha-namenode-b
hadoop-ha-datanode-a
hadoop-ha-datanode-b
hadoop-ha-datanode-c
hadoop-ha-resourcemanager-a
hadoop-ha-resourcemanager-b
hadoop-ha-nodemanager-a
hadoop-ha-nodemanager-b
hadoop-ha-nodemanager-c
经济版5台主机：
hadoop-ha-namenode-a
hadoop-ha-namenode-b
hadoop-ha-dnj-a
hadoop-ha-dnj-b
hadoop-ha-dnj-c
1. 定义集群主机
修改所有主机名称：vim /etc/hostname
重启：reboot
所有主机修改hosts映射文件： vim /etc/hosts
192.168.179.133    hadoop-ha-namenode-a
192.168.179.134    hadoop-ha-namenode-b
192.168.179.135    hadoop-ha-dnj-a
192.168.179.136    hadoop-ha-dnj-b
192.168.179.137    hadoop-ha-dnj-c
journalnode主机[hadoop-ha-dnj-*]配置ssh免密登录：
生成SSH_KEY： ssh-keygen -t rsa
本机免密登录配置： cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
测试本机登录： ssh root@hadoop-ha-dnj-*
journalnode主机之间互相配置免密登录：
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-ha-dnj-*
journalnode主机配置发送到namenode,resourcemanager主机上：
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-ha-namenode-*
namenode主机[hadoop-ha-namenode-*]同样进行ssh配置，并发送给datanode主机
resourcemanager主机也进行ssh配置，并发送给nodemanager主机
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-ha-namenode-*
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-ha-dnj-*
这样：
journalnode、namenode、resourcemanager各类主机之间可以互相免密登录
journalnode主机可以免密登录namenode主机、resourcemanager主机
namenode主机可以免密登录datanode主机
resourcemanager主机可以免密登录nodemanger主机
2. journalnode上配置zookeeper
【hadoop-ha-dnj-a】通过ftp将zookeeper开发包上传到主机中，解压：
tar xzvf /srv/ftp/zookeeper-3.4.10.tar.gz -C /usr/local
更名： mv /usr/local/zookeeper-3.4.10/ /usr/local/zookeeper
复制配置文件： cp /usr/local/zookeeper/conf/zoo_sample.cfg /usr/local/zookeeper/conf/zoo.cfg
建立zookeeper的数据文件目录： mkdir -p /usr/data/zookeeper
修改zoo.cfg配置文件： vim /usr/local/zookeeper/conf/zoo.cfg
dataDir=/usr/data/zookeeper
server.1=hadoop-ha-dnj-a:2888:3888 
server.2=hadoop-ha-dnj-b:2888:3888 
server.3=hadoop-ha-dnj-c:2888:3888
生成myid配置文件：echo 1 >> /usr/data/zookeeper/myid
将配置好的zookeeper发送给其他journalnode主机： 
scp -r /usr/local/zookeeper/ hadoop-ha-dnj-b:/usr/local/
scp -r /usr/local/zookeeper/ hadoop-ha-dnj-c:/usr/local/
在b，c主机上建立各自的文件目录： mkdir -p /usr/data/zookeeper
修改各自的myid配置文件：
echo 2 >> /usr/data/zookeeper/myid
echo 3 >> /usr/data/zookeeper/myid
启动所有的zookeeper进程： /usr/local/zookeeper/bin/zkServer.sh start
查看各自的状态： /usr/local/zookeeper/bin/zkServer.sh status


3. 在HA集群主机上配置DFS
namenode主机上【hadoop-ha-namenode-a】上传hadoop开发包，并解压缩：
tar xzvf /srv/ftp/hadoop-2.8.0.tar.gz -C /usr/local/
删除hadoop中的所有doc文档： rm -r /usr/local/hadoop-2.8.0/share/doc/
更名： mv /usr/local/hadoop-2.8.0/ /usr/local/hadoop
修改hadoop使用的JDK配置项： vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/local/jdk
建立数据的保存目录：
mkdir -p /usr/data/hadoop/{dfs,tmp}
mkdir -p /usr/data/hadoop/dfs/{name,data}
修改core-site.xml配置文件： vim /usr/local/hadoop/etc/hadoop/core-site.xml
<configuration>
       <property>
               <name>hadoop.tmp.dir</name>
               <value>/usr/data/hadoop/tmp</value>
       </property>
       <property>
               <name>fs.defaultFS</name>
               <value>hdfs://name-service-cluster</value>
       </property>
       <property>
               <name>ha.zookeeper.quorum</name>
               <value>hadoop-ha-dnj-a:2181,hadoop-ha-dnj-b:2181,hadoop-ha-dnj-c:2181</value>
       </property>
</configuration>
修改hdfs-site.xml文件： vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property> 
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.nameservices</name>
<value>name-service-cluster</value>
</property>
<property>
<name>dfs.ha.namenodes.name-service-cluster</name>
<value>nna,nnb</value>
</property>
<property>
<name>dfs.namenode.rpc-address.name-service-cluster.nna</name>
<value>hadoop-ha-namenode-a:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.name-service-cluster.nna</name>
<value>hadoop-ha-namenode-a:50070</value>
</property>
<property>
<name>dfs.namenode.rpc-address.name-service-cluster.nnb</name>
<value>hadoop-ha-namenode-b:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.name-service-cluster.nnb</name>
<value>hadoop-ha-namenode-b:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoop-ha-dnj-a:8485;hadoop-ha-dnj-b:8485;hadoop-ha-dnj-c:8485/name-service-cluster</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/usr/data/hadoop/journal</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.name-service-cluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>
</configuration>
在所有的journalnode主机上都创建edits目录： mkdir -p /usr/data/hadoop/journal
修改maper-site.xml配置文件：
拷贝配置文件： cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml
编辑配置文件： vim /usr/local/hadoop/etc/hadoop/mapred-site.xml
<configuration> 
<property> 
<name>mapreduce.framework.name</name> 
<value>yarn</value> 
</property> 
</configuration>
修改所有的从节点信息： vim /usr/local/hadoop/etc/hadoop/slaves
hadoop-ha-dnj-a
hadoop-ha-dnj-b
hadoop-ha-dnj-c
将此时的配置发送到其他的namenode主机和所有的datanode节点之中：
scp -r /usr/local/hadoop/ hadoop-ha-namenode-b:/usr/local/
scp -r /usr/local/hadoop/ hadoop-ha-dnj-a:/usr/local/
scp -r /usr/local/hadoop/ hadoop-ha-dnj-b:/usr/local/
scp -r /usr/local/hadoop/ hadoop-ha-dnj-c:/usr/local/
这样：
就完成了dfs的相关配置
4. 在HA集群主机上配置YARN：
（如果是分开的DFS和YARN，先将namenode已配置过的hadoop拷贝到resourcemanager主机上，此处不用）
resourcemanager主机【hadoop-ha-namenode-a】修改yarn-site.xml配置文件： vim /usr/local/hadoop/etc/hadoop/yarn-site.xml
<configuration>
       <property>
               <name>yarn.resourcemanager.ha.enabled</name>
               <value>true</value>
       </property>
       <property>
               <name>yarn.resourcemanager.cluster-id</name>
               <value>yarn-cluster</value>
       </property>
       <property>
               <name>yarn.resourcemanager.ha.rm-ids</name>
               <value>rma,rmb</value>
       </property>
       <property>
               <name>yarn.resourcemanager.hostname.rma</name>
               <value>hadoop-ha-namenode-a</value>
       </property>
       <property>
               <name>yarn.resourcemanager.hostname.rmb</name>
               <value>hadoop-ha-namenode-b</value>
       </property>
       <property>
               <name>yarn.resourcemanager.zk-address</name>
               <value>hadoop-ha-dnj-a:2181,hadoop-ha-dnj-b:2181,hadoop-ha-dnj-c:2181</value>
       </property>
       <property>
               <name>yarn.nodemanager.aux-services</name>
               <value>mapreduce_shuffle</value>
       </property>
</configuration>
修改slaves配置文件： vim /usr/local/hadoop/etc/hadoop/slaves
hadoop-ha-dnj-a
hadoop-ha-dnj-b
hadoop-ha-dnj-c
将配置好的hadoop拷贝到其他resourcemanager主机和所有nodemanager主机上： 
scp -r /usr/local/hadoop/ hadoop-ha-namenode-b:/usr/local/
scp -r /usr/local/hadoop/ hadoop-ha-dnj-a:/usr/local/
scp -r /usr/local/hadoop/ hadoop-ha-dnj-b:/usr/local/
scp -r /usr/local/hadoop/ hadoop-ha-dnj-c:/usr/local/


这样：
dfs和yarn的基本逻辑配置都完成了，所有的逻辑配置都是将节点指向了journalnode之中
5. 初始化HA集群：
将namenode主机的hadoop相关配置拷贝到journalnode主机中： scp -r /usr/local/hadoop/ hadoop-ha-dnj-*:/usr/local/
journalnode主机分别启动各自的journalnode进程： /usr/local/hadoop/sbin/hadoop-daemon.sh start journalnode
namenode-a主机初始化namenode节点信息： /usr/local/hadoop/bin/hdfs namenode -format
把namenode-a主机的配置文件拷贝到其他namenode主机： scp -r /usr/data/ hadoop-ha-namenode-b:/usr/
在任意的一台namenode主机上启动zkfc进程，用于和journalnode通讯： /usr/local/hadoop/bin/hdfs zkfc -formatZK
(如果此时成功了，应该会在ZooKeeper里面保存有相应的节点信息/hadoop-ha)
namenode主机启动dfs相关进程： /usr/local/hadoop/sbin/start-dfs.sh
如果成功，会出现：
Starting namenodes on [hadoop-ha-namenode-a hadoop-ha-namenode-b] 
hadoop-ha-namenode-a: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-hadoop-ha-namenode-a.out 
hadoop-ha-namenode-b: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-hadoop-ha-namenode-b.out 
hadoop-ha-datanode-c: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-hadoop-ha-datanode-c.out 
hadoop-ha-datanode-b: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-hadoop-ha-datanode-b.out 
hadoop-ha-datanode-a: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-hadoop-ha-datanode-a.out 
Starting journal nodes [hadoop-ha-journalnode-a hadoop-ha-journalnode-b hadoop-ha-journalnode-c] 
hadoop-ha-journalnode-c: journalnode running as process 2932. Stop it first. 
hadoop-ha-journalnode-b: journalnode running as process 2966. Stop it first. 
hadoop-ha-journalnode-a: journalnode running as process 3261. Stop it first. 
Starting ZK Failover Controllers on NN hosts [hadoop-ha-namenode-a hadoop-ha-namenode-b] 
hadoop-ha-namenode-a: starting zkfc, logging to /usr/local/hadoop/logs/hadoop-root-zkfc-hadoop-ha-namenode-a.out 
hadoop-ha-namenode-b: starting zkfc, logging to /usr/local/hadoop/logs/hadoop-root-zkfc-hadoop-ha-namenode-b.out
resourcemanager主机启动yarn相关线程:  /usr/local/hadoop/sbin/start-yarn.sh
如果成功，会显示：
starting yarn daemons 
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-hadoop-ha-resourcemanager-a.out 
hadoop-ha-nodemanager-c: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-hadoop-ha-nodemanager-c.out 
hadoop-ha-nodemanager-a: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-hadoop-ha-nodemanager-a.out 
hadoop-ha-nodemanager-b: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-hadoop-ha-nodemanager-b.out
resourcemanager-b启动resourcemanager: /usr/local/hadoop/sbin/yarn-daemon.sh start resourcemanager
查看页面显示信息：hadoop-ha-namenode-a:50070 hadoop-ha-namenode-b:50070
修改本机的hosts配置文件（C:\Windows\System32\drivers\etc\hosts）
192.168.179.133    hadoop-ha-namenode-a
192.168.179.134    hadoop-ha-namenode-b
6. HA测试：
当关掉当前active的主机，或者kill了当前active主机的zkfc进程，会发生主备的切换操作处理，并且不影响此时在进行的上传操作