Hadoop 2.7集群安装
安装Zookeeper集群
安装ZooKeeper集群需要至少3个以上奇数个节点,具体安装细节,请查阅我的博客。
Hadoop配置
core-site.xml
#修改为:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoopmaster:9000</value>
<description>NameNode URI</description>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/opt/h2/HA/data/jn</value>
<description>journalnode</description>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/h2/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/opt/h2/hdfs/datanode</value>
</property>
<property>
<name>dfs.nameservices</name>
<value>ha-cluster</value>
</property>
<property>
<name>dfs.ha.namenodes.ha-cluster</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ha-cluster.nn1</name>
<value>hadoopmaster:9000</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ha-cluster.nn2</name>
<value>hadoopmaster2:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.ha-cluster.nn1</name>
<value>hadoopmaster:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.ha-cluster.nn2</name>
<value>hadoopmaster2:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoopmaster:8485;node1:8485;node2:8485/ha-cluster<value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.ha-cluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>192.168.2.76:2181,192.168.2.77:2181,192.168.2.88:2181</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/app/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
</configuration>
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>Execution framework.</description>
</property>
</configuration>
yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
复制配置文件到各个node节点中,并且保证hadoopmaster、hadoopmaster2能够ssh到各个节点中,master机器可以彼此通信。具体细节,请参考hadoop集群安装的ssh设置步骤。
启动进程操作
启动journalnode
在3个节点(hadoopmaster、node1、node2)上启动journalnode进程:
hadoop-daemon.sh start journalnode
格式化Zookeeper文件系统
在active的namenode上运行:
hdfs zkfc -formatZK
格式化namenode
在active的namenode格式化namenode,并启动namenode:
hdfs namenode -format
hadoop-daemon.sh start namenode
在standby的namenode格式化namenode:
hdfs namenode -bootstrapStandby
启动hadoop所有进程
先stop已经启动的进程,然后,再启动hadoop所有进程。
stop-all.sh
start-all.sh
查看运行的进程
active的master进程:
$ jps
2551 JournalNode
2839 ResourceManager
2329 NameNode
2731 DFSZKFailoverController
3102 Jps
25934 JobHistoryServer
standby的master进程:
$ jps
11668 DFSZKFailoverController
11550 NameNode
11743 Jps
node1:
$ jps
4035 NodeManager
3924 JournalNode
4164 Jps
3818 DataNode
node2:
$ jps
17459 NodeManager
17588 Jps
17349 JournalNode
17243 DataNode
浏览WEB UI
MapReduce 管理页面:
http://{hadoopmaster}:50070/
http://{hadoopmaster}:50070/
验证HA
kill掉active的namenode,看看standby的namnode是否会变成active的。
http://{hadoopmaster2}:50070/
http://{hadoopmaster2}:50070/dfshealth.html#tab-datanode
验证HA有效,并且是成功的。
欢迎加入微信公众号