journalnode(3)
namenode(2)
datanode(3)
resourcemanager(2)
nodemanager(3)
富人版13台主机:
hadoop-ha-journalnode-a
hadoop-ha-journalnode-b
hadoop-ha-journalnode-c
hadoop-ha-namenode-a
hadoop-ha-namenode-b
hadoop-ha-datanode-a
hadoop-ha-datanode-b
hadoop-ha-datanode-c
hadoop-ha-resourcemanager-a
hadoop-ha-resourcemanager-b
hadoop-ha-nodemanager-a
hadoop-ha-nodemanager-b
hadoop-ha-nodemanager-c
经济版5台主机:
hadoop-ha-namenode-a
hadoop-ha-namenode-b
hadoop-ha-dnj-a
hadoop-ha-dnj-b
hadoop-ha-dnj-c
1. 定义集群主机
修改所有主机名称:vim /etc/hostname
重启:reboot
所有主机修改hosts映射文件: vim /etc/hosts
192.168.179.133 hadoop-ha-namenode-a
192.168.179.134 hadoop-ha-namenode-b
192.168.179.135 hadoop-ha-dnj-a
192.168.179.136 hadoop-ha-dnj-b
192.168.179.137 hadoop-ha-dnj-c
journalnode主机[hadoop-ha-dnj-*]配置ssh免密登录:
生成SSH_KEY: ssh-keygen -t rsa
本机免密登录配置: cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
测试本机登录: ssh root@hadoop-ha-dnj-*
journalnode主机之间互相配置免密登录:
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-ha-dnj-*
journalnode主机配置发送到namenode,resourcemanager主机上:
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-ha-namenode-*
namenode主机[hadoop-ha-namenode-*]同样进行ssh配置,并发送给datanode主机
resourcemanager主机也进行ssh配置,并发送给nodemanager主机
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-ha-namenode-*
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-ha-dnj-*
这样:
journalnode、namenode、resourcemanager各类主机之间可以互相免密登录
journalnode主机可以免密登录namenode主机、resourcemanager主机
namenode主机可以免密登录datanode主机
resourcemanager主机可以免密登录nodemanger主机
2. journalnode上配置zookeeper
【hadoop-ha-dnj-a】通过ftp将zookeeper开发包上传到主机中,解压:
tar xzvf /srv/ftp/zookeeper-3.4.10.tar.gz -C /usr/local
更名: mv /usr/local/zookeeper-3.4.10/ /usr/local/zookeeper
复制配置文件: cp /usr/local/zookeeper/conf/zoo_sample.cfg /usr/local/zookeeper/conf/zoo.cfg
建立zookeeper的数据文件目录: mkdir -p /usr/data/zookeeper
修改zoo.cfg配置文件: vim /usr/local/zookeeper/conf/zoo.cfg
dataDir=/usr/data/zookeeper
server.1=hadoop-ha-dnj-a:2888:3888
server.2=hadoop-ha-dnj-b:2888:3888
server.3=hadoop-ha-dnj-c:2888:3888
生成myid配置文件:echo 1 >> /usr/data/zookeeper/myid
将配置好的zookeeper发送给其他journalnode主机:
scp -r /usr/local/zookeeper/ hadoop-ha-dnj-b:/usr/local/
scp -r /usr/local/zookeeper/ hadoop-ha-dnj-c:/usr/local/
在b,c主机上建立各自的文件目录: mkdir -p /usr/data/zookeeper
修改各自的myid配置文件:
echo 2 >> /usr/data/zookeeper/myid
echo 3 >> /usr/data/zookeeper/myid
启动所有的zookeeper进程: /usr/local/zookeeper/bin/zkServer.sh start
查看各自的状态: /usr/local/zookeeper/bin/zkServer.sh status
3. 在HA集群主机上配置DFS
namenode主机上【hadoop-ha-namenode-a】上传hadoop开发包,并解压缩:
tar xzvf /srv/ftp/hadoop-2.8.0.tar.gz -C /usr/local/
删除hadoop中的所有doc文档: rm -r /usr/local/hadoop-2.8.0/share/doc/
更名: mv /usr/local/hadoop-2.8.0/ /usr/local/hadoop
修改hadoop使用的JDK配置项: vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/local/jdk
建立数据的保存目录:
mkdir -p /usr/data/hadoop/{dfs,tmp}
mkdir -p /usr/data/hadoop/dfs/{name,data}
修改core-site.xml配置文件: vim /usr/local/hadoop/etc/hadoop/core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/data/hadoop/tmp</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://name-service-cluster</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>hadoop-ha-dnj-a:2181,hadoop-ha-dnj-b:2181,hadoop-ha-dnj-c:2181</value>
</property>
</configuration>
修改hdfs-site.xml文件: vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.nameservices</name>
<value>name-service-cluster</value>
</property>
<property>
<name>dfs.ha.namenodes.name-service-cluster</name>
<value>nna,nnb</value>
</property>
<property>
<name>dfs.namenode.rpc-address.name-service-cluster.nna</name>
<value>hadoop-ha-namenode-a:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.name-service-cluster.nna</name>
<value>hadoop-ha-namenode-a:50070</value>
</property>
<property>
<name>dfs.namenode.rpc-address.name-service-cluster.nnb</name>
<value>hadoop-ha-namenode-b:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.name-service-cluster.nnb</name>
<value>hadoop-ha-namenode-b:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoop-ha-dnj-a:8485;hadoop-ha-dnj-b:8485;hadoop-ha-dnj-c:8485/name-service-cluster</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/usr/data/hadoop/journal</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.name-service-cluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>
</configuration>
在所有的journalnode主机上都创建edits目录: mkdir -p /usr/data/hadoop/journal
修改maper-site.xml配置文件:
拷贝配置文件: cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml
编辑配置文件: vim /usr/local/hadoop/etc/hadoop/mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
修改所有的从节点信息: vim /usr/local/hadoop/etc/hadoop/slaves
hadoop-ha-dnj-a
hadoop-ha-dnj-b
hadoop-ha-dnj-c
将此时的配置发送到其他的namenode主机和所有的datanode节点之中:
scp -r /usr/local/hadoop/ hadoop-ha-namenode-b:/usr/local/
scp -r /usr/local/hadoop/ hadoop-ha-dnj-a:/usr/local/
scp -r /usr/local/hadoop/ hadoop-ha-dnj-b:/usr/local/
scp -r /usr/local/hadoop/ hadoop-ha-dnj-c:/usr/local/
这样:
就完成了dfs的相关配置
4. 在HA集群主机上配置YARN:
(如果是分开的DFS和YARN,先将namenode已配置过的hadoop拷贝到resourcemanager主机上,此处不用)
resourcemanager主机【hadoop-ha-namenode-a】修改yarn-site.xml配置文件: vim /usr/local/hadoop/etc/hadoop/yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yarn-cluster</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rma,rmb</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rma</name>
<value>hadoop-ha-namenode-a</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rmb</name>
<value>hadoop-ha-namenode-b</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>hadoop-ha-dnj-a:2181,hadoop-ha-dnj-b:2181,hadoop-ha-dnj-c:2181</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
修改slaves配置文件: vim /usr/local/hadoop/etc/hadoop/slaves
hadoop-ha-dnj-a
hadoop-ha-dnj-b
hadoop-ha-dnj-c
将配置好的hadoop拷贝到其他resourcemanager主机和所有nodemanager主机上:
scp -r /usr/local/hadoop/ hadoop-ha-namenode-b:/usr/local/
scp -r /usr/local/hadoop/ hadoop-ha-dnj-a:/usr/local/
scp -r /usr/local/hadoop/ hadoop-ha-dnj-b:/usr/local/
scp -r /usr/local/hadoop/ hadoop-ha-dnj-c:/usr/local/
这样:
dfs和yarn的基本逻辑配置都完成了,所有的逻辑配置都是将节点指向了journalnode之中
5. 初始化HA集群:
将namenode主机的hadoop相关配置拷贝到journalnode主机中: scp -r /usr/local/hadoop/ hadoop-ha-dnj-*:/usr/local/
journalnode主机分别启动各自的journalnode进程: /usr/local/hadoop/sbin/hadoop-daemon.sh start journalnode
namenode-a主机初始化namenode节点信息: /usr/local/hadoop/bin/hdfs namenode -format
把namenode-a主机的配置文件拷贝到其他namenode主机: scp -r /usr/data/ hadoop-ha-namenode-b:/usr/
在任意的一台namenode主机上启动zkfc进程,用于和journalnode通讯: /usr/local/hadoop/bin/hdfs zkfc -formatZK
(如果此时成功了,应该会在ZooKeeper里面保存有相应的节点信息/hadoop-ha)
namenode主机启动dfs相关进程: /usr/local/hadoop/sbin/start-dfs.sh
如果成功,会出现:
Starting namenodes on [hadoop-ha-namenode-a hadoop-ha-namenode-b]
hadoop-ha-namenode-a: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-hadoop-ha-namenode-a.out
hadoop-ha-namenode-b: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-hadoop-ha-namenode-b.out
hadoop-ha-datanode-c: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-hadoop-ha-datanode-c.out
hadoop-ha-datanode-b: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-hadoop-ha-datanode-b.out
hadoop-ha-datanode-a: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-hadoop-ha-datanode-a.out
Starting journal nodes [hadoop-ha-journalnode-a hadoop-ha-journalnode-b hadoop-ha-journalnode-c]
hadoop-ha-journalnode-c: journalnode running as process 2932. Stop it first.
hadoop-ha-journalnode-b: journalnode running as process 2966. Stop it first.
hadoop-ha-journalnode-a: journalnode running as process 3261. Stop it first.
Starting ZK Failover Controllers on NN hosts [hadoop-ha-namenode-a hadoop-ha-namenode-b]
hadoop-ha-namenode-a: starting zkfc, logging to /usr/local/hadoop/logs/hadoop-root-zkfc-hadoop-ha-namenode-a.out
hadoop-ha-namenode-b: starting zkfc, logging to /usr/local/hadoop/logs/hadoop-root-zkfc-hadoop-ha-namenode-b.out
resourcemanager主机启动yarn相关线程: /usr/local/hadoop/sbin/start-yarn.sh
如果成功,会显示:
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-hadoop-ha-resourcemanager-a.out
hadoop-ha-nodemanager-c: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-hadoop-ha-nodemanager-c.out
hadoop-ha-nodemanager-a: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-hadoop-ha-nodemanager-a.out
hadoop-ha-nodemanager-b: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-hadoop-ha-nodemanager-b.out
resourcemanager-b启动resourcemanager: /usr/local/hadoop/sbin/yarn-daemon.sh start resourcemanager
查看页面显示信息:hadoop-ha-namenode-a:50070 hadoop-ha-namenode-b:50070
修改本机的hosts配置文件(C:\Windows\System32\drivers\etc\hosts)
192.168.179.133 hadoop-ha-namenode-a
192.168.179.134 hadoop-ha-namenode-b
6. HA测试:
当关掉当前active的主机,或者kill了当前active主机的zkfc进程,会发生主备的切换操作处理,并且不影响此时在进行的上传操作