Hadoop2.7.4集群搭建
一、集群架构
简介:本hadoop集群采用了2namenode,3datanode的数据分布,各服务器具体运行进程见下表
注释:各服务的说明如下
- NameNode:负责请求的分发,两个namenode做主备
- DataNode:负责数据的存储
- DFSZKFailoverController:负责namenode主节点的选取
- JournalNode:负责namenode主节点的数据同步
- ResourceManager:负责数据节点管理
- NodeManager:被管理节点
二、下载上传解压安装
2.1下载hadoop-2.7.4
地址:(https://archive.apache.org/dist/hadoop/common/)
2.2上传hadoop-2.7.4到linux上
2.2.1 上传hadoop安装包到/opt/software下
2.2.2 解压hadoop-2.7.4到/opt/servers下
命令:tar -xvf hadoop-2.7.4.tar.gz -C ../servers
2.3修改配置文件,配置文件目录 /opt/servers/hadoop-2.7.4/etc/hadoop下
2.3.1修改配置core-site.xml文件
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://oms-hadoop-hbase-node1:9000</value>
</property>
<!-- 临时文件存储目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/home/admin/hadoop/datas/tmp</value>
</property>
<!-- 缓冲区大小,实际工作中根据服务器性能动态调整 -->
<property>
<name>io.file.buffer.size</name>
<value>8192</value>
</property>
<!-- 开启hdfs的垃圾桶机制,删除掉的数据可以从垃圾桶中回收,单位分钟 -->
<property>
<name>fs.trash.interval</name>
<value>10080</value>
</property>
<!-- 指定zookeeper地址 -->
<property>
<name>ha.zookeeper.quorum</name>
<value>oms-zookeeper-kafka-1:2181,oms-zookeeper-kafka-2:2181,oms-zookeeper-kafka-3:2181</value>
<!--<value>oms-hadoop-hbase-node1:2181,oms-hadoop-hbase-node2:2181,oms-hadoop-hbase-node3:2181,oms-hadoop-hbase-node4:2181</value>-->
</property>
</configuration>
2.3.2修改配置hadoop-env.sh文件
配置jdk环境变量: export JAVA_HOME=/opt/edas/jdk/java
2.3.3修改配置hdfs-site.xml文件
<configuration>
<!--指定hdfs的nameservice为hdfs,需要和core-site.xml中的保持一致 -->
<property>
<name>dfs.nameservices</name>
<value>hdfs</value>
</property>
<!-- ns1下面有两个NameNode,分别是nn1,nn2 -->
<property>
<name>dfs.ha.namenodes.hdfs</name>
<value>oms-hadoop-hbase-node1,oms-hadoop-hbase-node2</value>
</property>
<!-- nn1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.hdfs.oms-hadoop-hbase-node1</name>
<value>oms-hadoop-hbase-node1:9000</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>127.0.0.1:50071</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>127.0.0.1:50091</value>
</property>
<!-- nn1的http通信地址 -->
<property>
<name>dfs.namenode.http-address.hdfs.oms-hadoop-hbase-node1</name>
<value>oms-hadoop-hbase-node1:50071</value>
</property>
<!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.hdfs.oms-hadoop-hbase-node2</name>
<value>oms-hadoop-hbase-node2:9000</value>
</property>
<!-- nn2的http通信地址 -->
<property>
<name>dfs.namenode.http-address.hdfs.oms-hadoop-hbase-node2</name>
<value>oms-hadoop-hbase-node2:50070</value>
</property>
<!-- 指定NameNode的元数据在JournalNode上的存放位置 -->
<!-- <property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://oms-hadoop-hbase-node2:8485;oms-hadoop-hbase-node3:8485;oms-hadoop-hbase-node4:8485/hdfs</value>
</property> -->
<!-- 指定JournalNode在本地磁盘存放数据的位置 -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/admin/hadoop/datas/journal</value>
</property>
<!-- 开启NameNode失败自动切换 -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!-- 配置失败自动切换实现方式 -->
<property>
<name>dfs.client.failover.proxy.provider.ns1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- 配置隔离机制 -->
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<!-- 使用隔离机制时需要ssh免登陆 -->
<!-- <property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property> -->
<!-- 文件副本数 -->
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/admin/hadoop/datas/hdfs/name</value>
</property>
<!-- datanode(具体数据)数据存放目录 -->
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/admin/hadoop/datas/datanode/datanodeDatas</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.directoryscan.throttle.limit.ms.per.sec</name>
<value>1000</value>
</property>
<!-- 设置文件分块的大小:Block 128m-->
<property>
<name>dfs.block.size</name>
<value>128m</value>
</property>
<property>
<name>dfs.balance.bandwidthPerSec</name>
<value>52428800</value>
</property>
<!-- 设置HDFS日志文件存放目录 -->
<!-- <property>
<name>dfs.namenode.edits.dir</name>
<value>file:///opt/servers/hadoop-2.7.4/datas/dfs/nn/edits</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file:///opt/servers/hadoop-2.7.4/datas/dfs/snn/name</value>
</property> -->
</configuration>
2.3.4 修改配置mapred-site.xml文件
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>oms-hadoop-hbase-node1:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>oms-hadoop-hbase-node1:19888</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>http://oms-hadoop-hbase-node1:9001</value>
</property>
</configuration>
2.3.5修改配置yarn-site.xml文件
<configuration>
<!-- Site specific YARN configuration properties -->
<!-- 指定resourcemanager地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>oms-hadoop-hbase-node1</value>
</property>
<!-- 指定nodemanager启动时加载server的方式 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>oms-hadoop-hbase-node1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>oms-hadoop-hbase-node1:8031</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>oms-hadoop-hbase-node1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>oms-hadoop-hbase-node1:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>oms-hadoop-hbase-node1:8089</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>6</value>
</property>
<!-- 设置不检查虚拟内存的值,不然内存不够会报错 -->
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>1024</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>1</value>
</property>
</configuration>
2.3.6修改配置slaves文件
配置所有的从机:
oms-hadoop-hbase-node2
oms-hadoop-hbase-node3
oms-hadoop-hbase-node4
2.4创建配置文件所需的文件存放目录
mkdir -p /home/admin/hadoop/datas/tmp
mkdir -p /home/admin/hadoop/datas/journal
mkdir -p /home/admin/hadoop/datas/hdfs/name
mkdir -p /home/admin/hadoop/datas/datanode/datanodeDatas
2.5把node1节点安装包分发到其它从机同目录下
scp -r hadoop-2.7.4/ oms-hadoop-hbase-node2:$PWD
scp -r hadoop-2.7.4/ oms-hadoop-hbase-node3:$PWD
scp -r hadoop-2.7.4/ oms-hadoop-hbase-node4:$PWD
2.6设置每个节点的环境变量
2.6.1 编辑 vi/etc/profile
配置 export HADOOP_HOME=/opt/servers/hadoop-2.7.4/
export PATH=:$HADOOP_HOME/bin/:$HADOOP_HOME/sbin:$PATH
重新加载 source /etc/profile
2.7格式化HDFS
2.7.1在hadoop主目录下执行命令:bin/hdfs namenode -format
2.8启动集群
2.8.1 整体启动顺序(两种启动模式)
1.在node1主机执行:
1)启动journalnode hadoop-daemons.sh start journalnode
2)启动namenode /opt/servers/hadoop-2.7.4/sbin/start-dfs.sh
3)启动nodemanager /opt/servers/hadoop-2.7.4/sbin/start-yarn.sh
2.按步骤启动顺序:
1)在oms-hadoop-hbase-1启动namenode
命令:/opt/hadoop/sbin/hadoop-daemon.sh start namenode
2)在oms-hadoop-hbase-node2,oms-hadoop-hbase-node3,oms-hadoop-hbase-node4启动journalnode
命令:/opt/hadoop/sbin/hadoop-daemon.sh start journalnode
3)在oms-hadoop-hbase-node1,oms-hadoop-hbase-node2上启动DFSZKFailoverController
命令:/opt/hadoop/sbin/hadoop-daemon.sh start zkfc
4)在oms-hadoop-hbase-2启动namenode
命令:/opt/hadoop/sbin/hadoop-daemon.sh start namenode
5)在oms-hadoop-hbase-2,oms-hadoop-hbase-3,oms-hadoop-hbase-4启动datanode
命令:/opt/hadoop/sbin/hadoop-daemon.sh start datanode
6)在oms-hadoop-hbase-1启动ResourceManager
命令:/opt/hadoop/sbin/start-yarn.sh
2.9验证集群是否成功
2.9.1使用jps命令验证进程启动情况(在每台节点上使用jps输出如下)
[18:05:20 root@oms-hadoop-hbase-node1 ~]# jps
1632 ResourceManager
1538 DFSZKFailoverController
5609 Jps
1199 NameNode
[18:06:24 root@oms-hadoop-hbase-node2 ~]# jps
514 DataNode
866 NodeManager
756 DFSZKFailoverController
421 NameNode
615 JournalNode
7981 Jps
[18:06:45 root@oms-hadoop-hbase-node3 ~]# jps
10276 DataNode
10376 JournalNode
16572 Jps
10478 NodeManager
[18:06:59 root@oms-hadoop-hbase-node4 ~]# jps
9748 NodeManager
9546 DataNode
18461 Jps
9646 JournalNode
2.9.2使用hadoop dfsadmin -report命令查询hadoop整个集群运行状态
[18:05:20 root@oms-hadoop-hbase-node1 ~]# /opt/hadoop/bin/hadoop dfsadmin -report
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Configured Capacity: 300497776640 (279.86 GB)
Present Capacity: 300111845822 (279.50 GB)
DFS Remaining: 296452663851 (276.09 GB)
DFS Used: 3659181971 (3.41 GB)
DFS Used%: 1.22%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Live datanodes (3):
Name: 192.168.0.77:50010 (oms-hadoop-hbase-2)
Hostname: oms-hadoop-hbase-2
Decommission Status : Normal
Configured Capacity: 107320709120 (99.95 GB)
2.9.3web访问oms-hadoop-hbase-node1:50070,oms-hadoop-hbase-node2:50070查看namenode节点运行状态。
注释:可以看到oms-hadoop-hbase-node1是活动主节点,oms-hadoop-hbase-node2是备用节点。由此可见,集群的主备namenode节点运行正常。
2.9.4 web访问oms-hadoop-hbase-node1:8088查看datanode运行情况
注释:由此可见3个datanode全部运行正常。