hadoop HA集群部署（HDFS&YARN）

最新推荐文章于 2022-10-05 19:08:40 发布

b1gx

最新推荐文章于 2022-10-05 19:08:40 发布

阅读量214

点赞数

分类专栏： hadoop 文章标签： hadoop HA 集群搭建

本文链接：https://blog.csdn.net/qq_40727267/article/details/99712641

版权

hadoop 专栏收录该内容

16 篇文章 2 订阅

订阅专栏

一、环境准备

阿里云按量付费方式购买三台同一区域的云主机
配置安全组策略

然后选择配置规则

选择添加安全组规则，在端口号和授权对象填入开放的端口和允许访问的ip
关闭防火墙（阿里云按量付费的不用管这个）

 # 停止防火墙
[hadoop@ruozedata001 software]$ systemctl stop firewalld    
# 关闭开机自启动防火墙
[hadoop@ruozedata001 software]$ systemctl disable firewalld
# 查看防火墙状态 
[hadoop@ruozedata001 software]$ systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

配置主机名
在创建云主机的时候既可以设置主机名，若未设置可使用如下命令设置，需要root

# 第一台机器
hostname ruozedata001

# 第二台机器
hostname ruozedata002

# 第二台机器
hostname ruozedata003

配置hosts文件
需要使用私网地址，三台机器都需要配置

[root@ruozedata001 ~]# vi /etc/hosts

172.31.92.236 ruozedata001
172.31.92.237 ruozedata002
172.31.92.235 ruozedata003

[root@ruozedata001 ~]# ping ruozedata001
PING ruozedata001 (172.31.92.236) 56(84) bytes of data.
64 bytes from ruozedata001 (172.31.92.236): icmp_seq=1 ttl=64 time=0.021 ms
64 bytes from ruozedata001 (172.31.92.236): icmp_seq=2 ttl=64 time=0.036 ms

配置ssh免密登录
方法一：有且知道用户密码的情况下

# 三台机器都执行
[root@ruozedata001 ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
74:35:be:8e:fc:4f:e5:94:90:fe:3b:e2:fe:bc:c7:7c root@ruozedata001
The key's randomart image is:
+--[ RSA 2048]----+
|            o    |
|           o ..  |
|        . . .o   |
|       . .  ... .|
|        S   .. .o|
|         . o  .+ |
|          o . .+.|
|           . o..E|
|            +++==|
+-----------------+
[root@ruozedata001 ~]# cat /root/.ssh/id_rsa.pub>> /root/.ssh/authorized_keys

# 将ruozedata002、ruozedata003 的 id_rsa.pub 发送到ruozedata001
# 如果没有配置hosts文件，须使用ip地址
[root@ruozedata002 ~]# cd .ssh/
[root@ruozedata002 .ssh]# scp id_rsa.pub ruozedata001:/root/.ssh/id_rsa.pub2
The authenticity of host 'ruozedata001 (172.31.92.236)' can't be established.
ECDSA key fingerprint is 5d:95:4e:69:2c:04:01:a0:3c:d3:c1:66:10:dd:d8:0a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ruozedata001,172.31.92.236' (ECDSA) to the list of known hosts.
root@ruozedata001's password: 
id_rsa.pub                                                            100%  399     0.4KB/s   00:00    

[root@ruozedata003 ~]# cd .ssh/
[root@ruozedata003 .ssh]# scp id_rsa.pub ruozedata001:/root/.ssh/id_rsa.pub3
root@ruozedata001's password: 
id_rsa.pub                                                            100%  399     0.4KB/s   00:00    

# 将ruozedata001的 id_rsa.pub2、id_rsa.pub3 合并到 authorized_keys，再将 authorized_keys 发送给另外两台机器
[root@ruozedata001 .ssh]# cat id_rsa.pub2 >> authorized_keys
[root@ruozedata001 .ssh]# cat id_rsa.pub3 >> authorized_keys
[root@ruozedata001 .ssh]# scp authorized_keys ruozedata002:/root/.ssh/
root@ruozedata002's password: 
authorized_keys                                                       100% 1197     1.2KB/s   00:00    
[root@ruozedata001 .ssh]# scp authorized_keys ruozedata003:/root/.ssh/
root@ruozedata003's password: 
authorized_keys                                                       100% 1197     1.2KB/s   00:00    

# 验证
[root@ruozedata001 .ssh]# ssh root@ruozedata001 date
Mon Aug 19 10:13:57 CST 2019
[root@ruozedata001 .ssh]# ssh root@ruozedata002 date
Mon Aug 19 10:14:01 CST 2019
[root@ruozedata001 .ssh]# ssh root@ruozedata003 date
Mon Aug 19 10:14:05 CST 2019

方法二：没有用户密码的情况

# 三台机器都执行如下命令
[hadoop@ruozedata001 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
b9:3f:25:b2:58:3d:20:ef:cb:02:21:dd:76:72:94:9f hadoop@ruozedata001
The key's randomart image is:
+--[ RSA 2048]----+
|         .       |
|        o        |
|   . . . . .     |
|  . o = +.E      |
|   . o *So       |
|    .   +.+ .    |
|     . +.o +     |
|      o.o..      |
|       .o...     |
+-----------------+

# 然后利用 sz 命令将ruozedata002、ruozedata003 的 id_rsa.pub 文件下载下来，再利用 rz 命令上传到ruozedata001 的相应目录，
# sz 与 rz 需要安装    yum install lrzsz
[hadoop@ruozedata002 .ssh]$ sz id_rsa.pub
[hadoop@ruozedata003 .ssh]$ sz id_rsa.pub

# 执行 下面的命令选择对应的文件
[hadoop@ruozedata001 .ssh]$ rz

# 将三台机器的 id_rsa.pub 文件 合并到 authorized_keys
[hadoop@ruozedata001 .ssh]$ cat id_rsa1.pub >> authorized_keys
[hadoop@ruozedata001 .ssh]$ cat id_rsa.pub >> authorized_keys
[hadoop@ruozedata001 .ssh]$ cat id_rsa_\(2\).pub >> authorized_keys
#修改权限！！！！！！！
[hadoop@ruozedata001 .ssh]$ chmod 600 authorized_keys

# 将 authorized_keys 下载到本地   再上传到另外两台机器
[hadoop@ruozedata001 .ssh]$ sz authorized_keys
[hadoop@ruozedata002 .ssh]$ rz
[hadoop@ruozedata003 .ssh]$ rz

# 验证
ssh hadoop@ruozedata001 date
ssh hadoop@ruozedata002 date
ssh hadoop@ruozedata003 date

二、安装JDK

解压
注：需要使用root用户在/usr目录下创建名为java的子目录，再将其解压到/usr/java/目录下，避免后期出现其它错误

[root@ruozedata001 software]# mkdir /usr/java/
[root@ruozedata001 software]# tar -zxvf jdk-8u45-linux-x64.gz -C /usr/java/

配置全局环境变量

[root@ruozedata001 software]# cd /usr/java/jdk1.8.0_45/
[root@ruozedata001 jdk1.8.0_45]# pwd
/usr/java/jdk1.8.0_45
[root@ruozedata001 jdk1.8.0_45]# vi /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=$JAVA_HOME/bin:$PATH

[root@ruozedata001 jdk1.8.0_45]# source /etc/profile
[root@ruozedata001 jdk1.8.0_45]# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

三台机器均需配置

三、安装zookeeper

三台机器都要配置

解压

[hadoop@ruozedata001 software]$ tar -zxvf zookeeper-3.4.6.tar.gz -C ~/app/

建立软连接

[hadoop@ruozedata001 software]$ cd ../app/
[hadoop@ruozedata001 app]$ ln -s zookeeper-3.4.6/ zookeeper

配置用户环境变量

[hadoop@ruozedata001 app]$ vi ~/.bash_profile 

export ZOOKEEPER_HOME=/home/hadoop/app/zookeeper
export PATH=$ZOOKEEPER_HOME/bin:$PATH

[hadoop@ruozedata001 app]$ source ~/.bash_profile

修改配置文件

# zoo.cfg
[hadoop@ruozedata001 conf]$ cp zoo_sample.cfg zoo.cfg
[hadoop@ruozedata001 conf]$ mkdir /home/hadoop/data/zookeeper
[hadoop@ruozedata001 conf]$ vi zoo.cfg

# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/home/hadoop/data/zookeeper
# the port at which the clients will connect
clientPort=2181

# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

server.1=ruozedata001:2888:3888
server.2=ruozedata002:2888:3888
server.3=ruozedata003:2888:3888


[hadoop@ruozedata001 conf]$ cd ~/data/
[hadoop@ruozedata001 data]$ cd zookeeper/
[hadoop@ruozedata001 zookeeper]$ echo 1 > myid

[hadoop@ruozedata002 zookeeper]$ echo 2 > myid

[hadoop@ruozedata003 zookeeper]$ echo 3 > myid
# 这个地方需要注意  >  左右的空格不能省略，省略了对应的内容写不进去

四、安装hadoop

解压

[hadoop@ruozedata001 software]$ tar -zxvf hadoop-2.6.0-cdh5.15.1.tar.gz -C ~/app/

建立软连接

[hadoop@ruozedata001 app]$ ln -s hadoop-2.6.0-cdh5.15.1 hadoop

配置用户环境变量

[hadoop@ruozedata001 app]$ vi ~/.bash_profile 
export HADOOP_HOME=/home/hadoop/app/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
[hadoop@ruozedata001 app]$ source ~/.bash_profile

bin目录存放的适是和系统管理相关的命令，sbin存放的是启停相关进程的命令

修改配置文件

hadoop-env.sh

JAVA_HOME=/usr/java/jdk1.8.0_45

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
	<!--Yarn 需要使用 fs.defaultFS 指定NameNode URI -->
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://ruozeclusterg7</value>
        </property>
        <!--==============================Trash机制======================================= -->
        <property>
                <!--多长时间创建CheckPoint NameNode截点上运行的CheckPointer 从Current文件夹创建CheckPoint;默认：0 由fs.trash.interval项指定 -->
                <name>fs.trash.checkpoint.interval</name>
                <value>0</value>
        </property>
        <property>
                <!--多少分钟.Trash下的CheckPoint目录会被删除,该配置服务器设置优先级大于客户端，默认：0 不删除 -->
                <name>fs.trash.interval</name>
                <value>1440</value>
        </property>

         <!--指定hadoop临时目录, hadoop.tmp.dir 是hadoop文件系统依赖的基础配置，很多路径都依赖它。如果hdfs-site.xml中不配 置namenode和datanode的存放位置，默认就放在这>个路径中 -->
        <property>   
                <name>hadoop.tmp.dir</name>
                <value>/home/hadoop/tmp/hadoop</value>
        </property>

         <!-- 指定zookeeper地址 -->
        <property>
                <name>ha.zookeeper.quorum</name>
                <value>ruozedata001:2181,ruozedata002:2181,ruozedata003:2181</value>
        </property>
         <!--指定ZooKeeper超时间隔，单位毫秒 -->
        <property>
                <name>ha.zookeeper.session-timeout.ms</name>
                <value>2000</value>
        </property>

        <property>
           <name>hadoop.proxyuser.hadoop.hosts</name>
           <value>*</value> 
        </property> 
        <property> 
            <name>hadoop.proxyuser.hadoop.groups</name> 
            <value>*</value> 
       </property> 


      <property>
		  <name>io.compression.codecs</name>
		  <value>org.apache.hadoop.io.compress.GzipCodec,
			org.apache.hadoop.io.compress.DefaultCodec,
			org.apache.hadoop.io.compress.BZip2Codec,
			org.apache.hadoop.io.compress.SnappyCodec
		  </value>
      </property>
</configuration>

hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
	<!--HDFS超级用户 -->
	<property>
		<name>dfs.permissions.superusergroup</name>
		<value>hadoop</value>
	</property>

	<!--开启web hdfs -->
	<property>
		<name>dfs.webhdfs.enabled</name>
		<value>true</value>
	</property>
	<property>
		<name>dfs.namenode.name.dir</name>
		<value>/home/hadoop/data/dfs/name</value>
		<description> namenode 存放name table(fsimage)本地目录（需要修改）</description>
	</property>
	<property>
		<name>dfs.namenode.edits.dir</name>
		<value>${dfs.namenode.name.dir}</value>
		<description>namenode粗放 transaction file(edits)本地目录（需要修改）</description>
	</property>
	<property>
		<name>dfs.datanode.data.dir</name>
		<value>/home/hadoop/data/dfs/data</value>
		<description>datanode存放block本地目录（需要修改）</description>
	</property>
	<property>
		<name>dfs.replication</name>
		<value>3</value>
	</property>
	<!-- 块大小128M （默认128M） -->
	<property>
		<name>dfs.blocksize</name>
		<value>134217728</value>
	</property>
	<!--======================================================================= -->
	<!--HDFS高可用配置 -->
	<!--指定hdfs的nameservice为ruozeclusterg7,需要和core-site.xml中的保持一致 -->
	<property>
		<name>dfs.nameservices</name>
		<value>ruozeclusterg7</value>
	</property>
	<property>
		<!--设置NameNode IDs 此版本最大只支持两个NameNode -->
		<name>dfs.ha.namenodes.ruozeclusterg7</name>
		<value>nn1,nn2</value>
	</property>

	<!-- Hdfs HA: dfs.namenode.rpc-address.[nameservice ID] rpc 通信地址 -->
	<property>
		<name>dfs.namenode.rpc-address.ruozeclusterg7.nn1</name>
		<value>ruozedata001:8020</value>
	</property>
	<property>
		<name>dfs.namenode.rpc-address.ruozeclusterg7.nn2</name>
		<value>ruozedata002:8020</value>
	</property>

	<!-- Hdfs HA: dfs.namenode.http-address.[nameservice ID] http 通信地址 -->
	<property>
		<name>dfs.namenode.http-address.ruozeclusterg7.nn1</name>
		<value>ruozedata001:50070</value>
	</property>
	<property>
		<name>dfs.namenode.http-address.ruozeclusterg7.nn2</name>
		<value>ruozedata002:50070</value>
	</property>

	<!--==================Namenode editlog同步 ============================================ -->
	<!--保证数据恢复 -->
	<property>
		<name>dfs.journalnode.http-address</name>
		<value>0.0.0.0:8480</value>
	</property>
	<property>
		<name>dfs.journalnode.rpc-address</name>
		<value>0.0.0.0:8485</value>
	</property>
	<property>
		<!--设置JournalNode服务器地址，QuorumJournalManager 用于存储editlog -->
		<!--格式：qjournal://<host1:port1>;<host2:port2>;<host3:port3>/<journalId> 端口同journalnode.rpc-address -->
		<name>dfs.namenode.shared.edits.dir</name>
		<value>qjournal://ruozedata001:8485;ruozedata002:8485;ruozedata003:8485/ruozeclusterg7</value>
	</property>

	<property>
		<!--JournalNode存放数据地址 -->
		<name>dfs.journalnode.edits.dir</name>
		<value>/home/hadoop/data/dfs/jn</value>
	</property>
	<!--==================DataNode editlog同步 ============================================ -->
	<property>
		<!--DataNode,Client连接Namenode识别选择Active NameNode策略 -->
                             <!-- 配置失败自动切换实现方式 -->
		<name>dfs.client.failover.proxy.provider.ruozeclusterg7</name>
		<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
	</property>
	<!--==================Namenode fencing：=============================================== -->
	<!--Failover后防止停掉的Namenode启动，造成两个服务 -->
	<property>
		<name>dfs.ha.fencing.methods</name>
		<value>sshfence</value>
	</property>
	<property>
		<name>dfs.ha.fencing.ssh.private-key-files</name>
		<value>/home/hadoop/.ssh/id_rsa</value>
	</property>
	<property>
		<!--多少milliseconds 认为fencing失败 -->
		<name>dfs.ha.fencing.ssh.connect-timeout</name>
		<value>30000</value>
	</property>

	<!--==================NameNode auto failover base ZKFC and Zookeeper====================== -->
	<!--开启基于Zookeeper  -->
	<property>
		<name>dfs.ha.automatic-failover.enabled</name>
		<value>true</value>
	</property>
	<!--动态许可datanode连接namenode列表 -->
	 <property>
	   <name>dfs.hosts</name>
	   <value>/home/hadoop/app/hadoop/etc/hadoop/slaves</value>
	 </property>
</configuration>

mapred-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
	<!-- 配置 MapReduce Applications -->
	<property>
		<name>mapreduce.framework.name</name>
		<value>yarn</value>
	</property>
	<!-- JobHistory Server ============================================================== -->
	<!-- 配置 MapReduce JobHistory Server 地址 ，默认端口10020 -->
	<property>
		<name>mapreduce.jobhistory.address</name>
		<value>ruozedata001:10020</value>
	</property>
	<!-- 配置 MapReduce JobHistory Server web ui 地址， 默认端口19888 -->
	<property>
		<name>mapreduce.jobhistory.webapp.address</name>
		<value>ruozedata001:19888</value>
	</property>

<!-- 配置 Map段输出的压缩,snappy-->
  <property>
      <name>mapreduce.map.output.compress</name> 
      <value>true</value>
  </property>
              
  <property>
      <name>mapreduce.map.output.compress.codec</name> 
      <value>org.apache.hadoop.io.compress.SnappyCodec</value>
   </property>

</configuration>

yarn-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
	<!-- nodemanager 配置 ================================================= -->
	<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
	</property>
	<property>
		<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
		<value>org.apache.hadoop.mapred.ShuffleHandler</value>
	</property>
	<property>
		<name>yarn.nodemanager.localizer.address</name>
		<value>0.0.0.0:23344</value>
		<description>Address where the localizer IPC is.</description>
	</property>
	<property>
		<name>yarn.nodemanager.webapp.address</name>
		<value>0.0.0.0:23999</value>
		<description>NM Webapp address.</description>
	</property>

	<!-- HA 配置 =============================================================== -->
	<!-- Resource Manager Configs -->
	<property>
		<name>yarn.resourcemanager.connect.retry-interval.ms</name>
		<value>2000</value>
	</property>
	<property>
		<name>yarn.resourcemanager.ha.enabled</name>
		<value>true</value>
	</property>
	<property>
		<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
		<value>true</value>
	</property>
	<!-- 使嵌入式自动故障转移。HA环境启动，与 ZKRMStateStore 配合 处理fencing -->
	<property>
		<name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
		<value>true</value>
	</property>
	<!-- 集群名称，确保HA选举时对应的集群 -->
	<property>
		<name>yarn.resourcemanager.cluster-id</name>
		<value>yarn-cluster</value>
	</property>
	<property>
		<name>yarn.resourcemanager.ha.rm-ids</name>
		<value>rm1,rm2</value>
	</property>


    <!--这里RM主备结点需要单独指定,（可选）
	<property>
		 <name>yarn.resourcemanager.ha.id</name>
		 <value>rm2</value>
	 </property>
	 -->

	<property>
		<name>yarn.resourcemanager.scheduler.class</name>
		<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
	</property>
	<property>
		<name>yarn.resourcemanager.recovery.enabled</name>
		<value>true</value>
	</property>
	<property>
		<name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
		<value>5000</value>
	</property>
	<!-- ZKRMStateStore 配置 -->
	<property>
		<name>yarn.resourcemanager.store.class</name>
		<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
	</property>
	<property>
		<name>yarn.resourcemanager.zk-address</name>
		<value>ruozedata001:2181,ruozedata002:2181,ruozedata003:2181</value>
	</property>
	<property>
		<name>yarn.resourcemanager.zk.state-store.address</name>
		<value>ruozedata001:2181,ruozedata002:2181,ruozedata003:2181</value>
	</property>
	<!-- Client访问RM的RPC地址 (applications manager interface) -->
	<property>
		<name>yarn.resourcemanager.address.rm1</name>
		<value>ruozedata001:23140</value>
	</property>
	<property>
		<name>yarn.resourcemanager.address.rm2</name>
		<value>ruozedata002:23140</value>
	</property>
	<!-- AM访问RM的RPC地址(scheduler interface) -->
	<property>
		<name>yarn.resourcemanager.scheduler.address.rm1</name>
		<value>ruozedata001:23130</value>
	</property>
	<property>
		<name>yarn.resourcemanager.scheduler.address.rm2</name>
		<value>ruozedata002:23130</value>
	</property>
	<!-- RM admin interface -->
	<property>
		<name>yarn.resourcemanager.admin.address.rm1</name>
		<value>ruozedata001:23141</value>
	</property>
	<property>
		<name>yarn.resourcemanager.admin.address.rm2</name>
		<value>ruozedata002:23141</value>
	</property>
	<!--NM访问RM的RPC端口 -->
	<property>
		<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
		<value>ruozedata001:23125</value>
	</property>
	<property>
		<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
		<value>ruozedata002:23125</value>
	</property>
	<!-- RM web application 地址 -->
	<property>
		<name>yarn.resourcemanager.webapp.address.rm1</name>
		<value>ruozedata001:8088</value>
	</property>
	<property>
		<name>yarn.resourcemanager.webapp.address.rm2</name>
		<value>ruozedata002:8088</value>
	</property>
	<property>
		<name>yarn.resourcemanager.webapp.https.address.rm1</name>
		<value>ruozedata001:23189</value>
	</property>
	<property>
		<name>yarn.resourcemanager.webapp.https.address.rm2</name>
		<value>ruozedata002:23189</value>
	</property>



	<property>
	   <name>yarn.log-aggregation-enable</name>
	   <value>true</value>
	</property>
	<property>
		 <name>yarn.log.server.url</name>
		 <value>http://ruozedata001:19888/jobhistory/logs</value>
	</property>


	<property>
		<name>yarn.nodemanager.resource.memory-mb</name>
		<value>2048</value>
	</property>
	<property>
		<name>yarn.scheduler.minimum-allocation-mb</name>
		<value>1024</value>
		<discription>单个任务可申请最少内存，默认1024MB</discription>
	 </property>

  
  <property>
	<name>yarn.scheduler.maximum-allocation-mb</name>
	<value>2048</value>
	<discription>单个任务可申请最大内存，默认8192MB</discription>
  </property>

   <property>
       <name>yarn.nodemanager.resource.cpu-vcores</name>
       <value>2</value>
    </property>

</configuration>

yarn-env.sh

slaves

ruozedata001
ruozedata002
ruozedata003

五、启动集群

启动zk

# 三台都需要
[hadoop@ruozedata001 ~]$ zkServer.sh start

启动journalnode

# 三台都需要 
[hadoop@ruozedata001 ~]$ hadoop-daemon.sh start journalnode

格式化namenode

# 只需在第一台格式化
[hadoop@ruozedata001 ~]$ hadoop namenode -format

# 格式化成功之后将元数据拷贝到另一个namenode所在的机器上，目录是  dfs.namenode.name.dir  指定的目录
[hadoop@ruozedata001 ~]$ scp -r /home/hadoop/data/dfs/name/ hadoop@ruozedata002:/home/hadoop/data/dfs/

格式化ZKFC

[hadoop@ruozedata001 ~]$ hdfs zkfc -formatZK

启动dfs

[hadoop@ruozedata001 ~]$ start-dfs.sh
Starting namenodes on [ruozedata001 ruozedata002]
ruozedata001: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-namenode-ruozedata001.out
ruozedata002: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-namenode-ruozedata002.out
ruozedata003: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-datanode-ruozedata003.out
ruozedata001: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-datanode-ruozedata001.out
ruozedata002: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-datanode-ruozedata002.out
Starting journal nodes [ruozedata001 ruozedata002 ruozedata003]
ruozedata002: journalnode running as process 21856. Stop it first.
ruozedata001: journalnode running as process 24187. Stop it first.
ruozedata003: journalnode running as process 20643. Stop it first.
Starting ZK Failover Controllers on NN hosts [ruozedata001 ruozedata002]
ruozedata001: starting zkfc, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-zkfc-ruozedata001.out
ruozedata002: starting zkfc, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-zkfc-ruozedata002.out

启动yarn

[hadoop@ruozedata001 ~]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/yarn-hadoop-resourcemanager-ruozedata001.out
ruozedata001: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/yarn-hadoop-nodemanager-ruozedata001.out
ruozedata003: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/yarn-hadoop-nodemanager-ruozedata003.out
ruozedata002: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/yarn-hadoop-nodemanager-ruozedata002.out

查看进程

[hadoop@ruozedata001 ~]$ jps
19153 QuorumPeerMain
24850 DFSZKFailoverController
24563 DataNode
24187 JournalNode
24989 ResourceManager
24429 NameNode
25455 Jps
25087 NodeManager

[hadoop@ruozedata002 dfs]$ jps
21856 JournalNode
22513 ResourceManager
22085 DataNode
22710 Jps
22215 DFSZKFailoverController
19101 QuorumPeerMain
21982 NameNode
22367 NodeManager

[hadoop@ruozedata003 dfs]$ jps
20752 DataNode
19057 QuorumPeerMain
20643 JournalNode
21029 Jps
20894 NodeManager

访问web界面

# 如果访问不了 检查一下安全组设置有没有开放这两个端口，或者对应的服务有没有启动成功
# HDFS
http://ip:50070

#YARN
http://ip:8088
#这里需要注意
standby的resourcemanager节点的web页面需要使用如下地址
http://ip:8088/cluster/cluster

b1gx

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
hadoop HA集群部署（HDFS&YARN）

一、环境准备阿里云按量付费方式购买三台同一区域的云主机配置安全组策略然后选择配置规则选择添加安全组规则，在端口号和授权对象填入开放的端口和允许访问的ip关闭防火墙（阿里云按量付费的不用管这个） # 停止防火墙[hadoop@ruozedata001 software]$ systemctl stop firewalld # 关闭开机自启动防火墙[hadoop@r...
复制链接

扫一扫

专栏目录