集群安装：HA与Yarn（测试）

最新推荐文章于 2023-03-31 19:51:06 发布

**码上人生**

最新推荐文章于 2023-03-31 19:51:06 发布

阅读量1.2k

点赞数

分类专栏： hadoop相关文章标签： hadoop集群安装

本文链接：https://blog.csdn.net/qq_16365849/article/details/50642737

版权

hadoop相关专栏收录该内容

11 篇文章 0 订阅

订阅专栏

准备工作：

修改网卡和主机名（在三个节点同时操作）

1）、修改网卡1：

注意：此处只用一个网卡就可以了，本地IP（10.10.10.1）可以不用配置

Vi/etc/sysconfig/network-scripts/ifcfg-eth0

修改网卡2：

设置好后重启网络服务：

看网卡的配置信息：

2）、修改主机名（修改/etc/hosts文件）

（在三个节点同时操作）

[root@hadoop2 ~]# vi /etc/hosts

# Do not remove the following line, orvarious programs

# that require network functionality willfail.

127.0.0.1 localhost

192.168.6.52 hadoop2

10.10.10.2 hadoop2priv

192.168.6.51 hadoop1

10.10.10.1 hadoop1priv

192.168.6.53 hadoop3

10.10.10.3 hadoop3priv

注意：可以把此文件拷贝到节点2和节点3

修改完文件后，最后记得在相应的机器上执行hostname master(你修改后的名字) ，hostname slave1等；（此步可以省略）

[root@hadoop2 ~]# vi /etc/hostname

hadoop2

[root@hadoop2 ~]# hostname hadoop2

3）、关闭防火墙

需要关闭SELINX，执行：/usr/sbin/setenforce0

注意：最好是手动关闭。

还有：要把各个服务器的防火墙给关闭了，不然，后面运行时会报错。

Linux关闭防火墙命令：1）永久性生效，重启后不会复原开启：chkconfigiptables on 关闭：chkconfig iptables off 2）即时生效，重启后复原开启：serviceiptables start 关闭：service iptables stop

2、创建用户和组，并配置信任关系来实现免密码登陆

1）、创建用户和组（在三个节点同时操作）

[root@hadoop ~]# groupadd -g 200 hadoop

[root@hadoop ~]# useradd -u 200 -g hadoophadoop

[root@hadoop ~]# passwd hadoop

Changing password for user hadoop.

New UNIX password:

BAD PASSWORD: it is based on a dictionaryword

Retype new UNIX password:

passwd: all authentication tokens updatedsuccessfully.

[root@hadoop ~]# su - hadoop

2）、配置信任关系

在节点1

在hadoop用户下生成密钥：rsa格式的密钥都选择默认格式

[hadoop@hadoop ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key(/home/hadoop/.ssh/id_rsa):

Enter passphrase (empty for nopassphrase):

Enter same passphrase again:

Your identification has been saved in/home/hadoop/.ssh/id_rsa.

Your public key has been saved in/home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

1a:d9:48:f8:de:5b:be:e7:1f:5b:fd:48:df:59:59:94hadoop@hadoop

[hadoop@hadoop ~]$ cd .ssh

[hadoop@hadoop .ssh]$ ls

id_rsa id_rsa.pub

把公钥添加到密钥中去

[hadoop@hadoop ~]$ cat .ssh/id_rsa.pub >>.ssh/authorized_keys

#修改master 密钥权限,非常容易错误的地方。

chmod go-rwx /home/hadoop/.ssh/authorized_keys

[hadoop@h1 .ssh]$ ll

total 24

-rw------- 1 hadoop hadoop 391 Jun 7 17:07 authorized_keys 注意：权限为600

-rw------- 1 hadoop hadoop 1675 Jun 7 17:06 id_rsa

-rw-r--r-- 1 hadoop hadoop 391 Jun 7 17:06 id_rsa.pub

在节点2：（注意：在hadoop用户下）

[root@hadoop2 ~]# su - hadoop

[hadoop@hadoop2 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key(/home/hadoop/.ssh/id_rsa):

Created directory '/home/hadoop/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in/home/hadoop/.ssh/id_rsa.

Your public key has been saved in/home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

43:c8:05:b9:6d:44:2c:b9:f3:c9:da:2d:64:b7:e9:83hadoop@hadoop2

[hadoop@hadoop2 ~]$ cd .ssh

[hadoop@hadoop2 .ssh]$ ls

id_rsa id_rsa.pub

[hadoop@hadoop2 .ssh]$

在节点3：（注意：在hadoop用户下）

[root@hadoop3 ~]# su - hadoop

[hadoop@hadoop3 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key(/home/hadoop/.ssh/id_rsa):

Created directory '/home/hadoop/.ssh'.

Enter passphrase (empty for nopassphrase):

Enter same passphrase again:

Your identification has been saved in/home/hadoop/.ssh/id_rsa.

Your public key has been saved in/home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

be:2b:89:28:99:e2:46:1d:86:3c:cf:01:78:eb:5d:f3hadoop@hadoop3

[hadoop@hadoop3 ~]$ cd .ssh

[hadoop@hadoop3 .ssh]$ ls

id_rsa id_rsa.pub

[hadoop@hadoop3 .ssh]$

在节点1上，添加节点2和节点3的公钥到节点1的authorized_keys中：

[hadoop@hadoop1 ~]$ ssh hadoop2 cat.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The authenticity of host 'hadoop2(192.168.6.52)' can't be established.

RSA key fingerprint isbe:ac:97:91:50:9c:63:b6:4d:35:3f:60:be:e1:ab:3d.

Are you sure you want to continue connecting(yes/no)? yes

Warning: Permanently added'hadoop2,192.168.6.52' (RSA) to the list of known hosts.

hadoop@hadoop2's password:

[hadoop@hadoop1 ~]$ ssh hadoop3 cat.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The authenticity of host 'hadoop3 (192.168.6.53)'can't be established.

RSA key fingerprint isbe:ac:97:91:50:9c:63:b6:4d:35:3f:60:be:e1:ab:3d.

Are you sure you want to continueconnecting (yes/no)? yes

Warning: Permanently added'hadoop3,192.168.6.53' (RSA) to the list of known hosts.

hadoop@hadoop3's password:

把节点上的authorized_keys 文件拷贝到节点2和节点3上：

[hadoop@hadoop1 ~]$ scp.ssh/authorized_keys hadoop2:~/.ssh

hadoop@hadoop2's password:

authorized_keys 100% 792 0.8KB/s 00:00

[hadoop@hadoop1 ~]$ scp.ssh/authorized_keys hadoop3:~/.ssh

hadoop@hadoop3's password:

authorized_keys 100% 1188 1.2KB/s 00:00

注意：拷贝到节点2和节点3上的authorized_keys文件的权限和节点1修改后的一样，所以，在节点2和节点3上就不用修改了

[hadoop@hadoop2 .ssh]$ ll

total 32

-rw------- 1 hadoop hadoop 1188 Dec 1914:20 authorized_keys

验证节点1能否连通节点1、节点2和节点3

[hadoop@hadoop1 ~]$ ssh hadoop1

Last login: Thu Dec 19 14:14:24 2013 fromhadoop1

[hadoop@hadoop1 ~]$ ssh hadoop2 date

Thu Dec 19 14:16:09 CST 2013

[hadoop@hadoop1 ~]$ ssh hadoop3 date

Thu Dec 19 14:22:31 CST 2013

验证节点2能否连通节点1、节点2和节点3

[hadoop@hadoop2 ~]$ ssh hadoop2

Last login: Thu Dec 19 14:15:42 2013 fromhadoop2

[hadoop@hadoop2 ~]$ ssh hadoop1 date

Thu Dec 19 14:23:12 CST 2013

[hadoop@hadoop2 ~]$ ssh hadoop3 date

The authenticity of host 'hadoop3(192.168.6.53)' can't be established.

RSA key fingerprint isbe:ac:97:91:50:9c:63:b6:4d:35:3f:60:be:e1:ab:3d.

Are you sure you want to continueconnecting (yes/no)? yes

Warning: Permanently added'hadoop3,192.168.6.53' (RSA) to the list of known hosts.

Thu Dec 19 14:23:18 CST 2013

验证节点3能否连通节点1、节点2和节点3

hadoop@hadoop3 .ssh]$ ssh hadoop3

The authenticity of host 'hadoop3(192.168.6.53)' can't be established.

RSA key fingerprint is be:ac:97:91:50:9c:63:b6:4d:35:3f:60:be:e1:ab:3d.

Are you sure you want to continueconnecting (yes/no)? yes

Warning: Permanently added'hadoop3,192.168.6.53' (RSA) to the list of known hosts.

Last login: Thu Dec 19 14:22:03 2013 fromhadoop1

[hadoop@hadoop3 ~]$ ssh hadoop2 date

The authenticity of host 'hadoop2(192.168.6.52)' can't be established.

RSA key fingerprint isbe:ac:97:91:50:9c:63:b6:4d:35:3f:60:be:e1:ab:3d.

Are you sure you want to continueconnecting (yes/no)? yes

Warning: Permanently added 'hadoop2,192.168.6.52'(RSA) to the list of known hosts.

Thu Dec 19 14:24:08 CST 2013

[hadoop@hadoop3 ~]$ ssh hadoop2 date

Thu Dec 19 14:24:16 CST 2013

3、下载并安装 JAVA JDK系统软件

（注意：在root用户下安装，不然会报错）

请参考linux中安装jdk（在三个节点同时操作）

#下载jdk

wget http://60.28.110.228/source/package/jdk-6u21-linux-i586-rpm.bin

#安装jdk

chmod +x jdk-6u21-linux-i586-rpm.bin 别忘了赋权限

./jdk-6u21-linux-i586-rpm.bin

安装,执行命令

[root@hn ~]# rpm -ivh jdk-6u17-linux-i586.rpm

(jdk的默认路径为/usr/java/jdk1.6.0_17)

#配置环境变量

注意：此处可以修改.bash_profile，也可以修改/etc/profile，也可以修改 /etc/profile.d/java.sh。通常修改.bash_profile，下面以此为例。（此处最好修改/etc/profile文件）

[root@linux64 ~]# vi .bash_profile

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

# User specific environment and startupprograms

PATH=$PATH:$HOME/bin

export PATH

unset USERNAME

添加java的参数

exportJAVA_HOME=/home/hadoop/java/jdk1.7.0_45

exportHADOOP_HOME=/home/hadoop/hadoop/hadoop-1.1.2

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

exportPATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH

vi /etc/profile

#复制粘贴一下内容到 vi 中。

exportJAVA_HOME=/home/hadoop/java/jdk1.8.0_25

export HADOOP_HOME=/home/hadoop/hadoop/hadoop-2.2.0

exportPATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_YARN_HOME=$HADOOP_HOME

exportHADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

exportHADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

exportHADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

#手动立即生效

source /etc/profile

#测试：下面两个命令都可能报下面的错误

[hadoop@ha2 ~]$ java

[hadoop@ha2 ~]$ jps

Error: dl failure on line 863

Error: failed/home/hadoop/java/jdk1.7.0_45/jre/lib/i386/client/libjvm.so, because/home/hadoop/java/jdk1.7.0_45/jre/lib/i386/client/libjvm.so: cannot restoresegment prot after reloc: Permission denied

这是因为SELINUX的问题，需要关闭SELINX，执行：/usr/sbin/setenforce 0

注意：最好是手动关闭。

还有：要把各个服务器的防火墙给关闭了，不然，后面运行时会报错。

4、检查基础环境

/sbin/ifconfig

[hadoop@master root]$ /sbin/ifconfig

eth0 Link encap:Ethernet HWaddr 00:0C:29:7A:DE:12

inet addr:192.168.1.100 Bcast:192.168.1.255 Mask:255.255.255.0

inet6 addr: fe80::20c:29ff:fe7a:de12/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:14 errors:0 dropped:0 overruns:0 frame:0

TX packets:821 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:1591 (1.5 KiB) TX bytes:81925 (80.0 KiB)

Interrupt:67 Base address:0x2024

ping master

ssh master

jps

echo $JAVA_HOME

echo $HADOOP_HOME

Hadoop

5、在节点1解压安装hadoop

创建hadoop的安装目录（在三个节点同时操作）

[root@hadoop1 hadoop]# mkdir -p/opt/hadoop

[root@hadoop1 hadoop]# chown -Rhadoop:hadoop /opt/hadoop/

[root@hadoop1 hadoop]# chmod 755 *

[root@hadoop1 hadoop]# ll

total 61036

-rwxr-xr-x 1 hadoop hadoop 62428860 Dec 1722:48 hadoop-1.0.3.tar.gz

解压hadoop软件（只需要在节点1上安装，配置好后拷贝到其他节点即可）

tar -xzvf hadoop-1.0.3.tar.gz

6、安装Hadoop2.2，搭建集群

在hadoopMaster上安装hadoop

首先到Apache官网上下载hadoop2.2的压缩文件，将其解压到当前用户的根文件夹中（home/fyzwjd/），将解压出的文件夹改名为hadoop。

$ sudo mv hadoop-2.2.0 hadoop

配置之前，先在本地文件系统创建以下文件夹：~/hadoop/tmp、~/dfs/data、~/dfs/name。主要涉及的配置文件有7个：都在/hadoop/etc/hadoop文件夹下，可以用gedit命令对其进行编辑。

~/hadoop/etc/hadoop/hadoop-env.sh

~/hadoop/etc/hadoop/yarn-env.sh

~/hadoop/etc/hadoop/slaves

~/hadoop/etc/hadoop/core-site.xml

~/hadoop/etc/hadoop/hdfs-site.xml

~/hadoop/etc/hadoop/mapred-site.xml

~/hadoop/etc/hadoop/yarn-site.xml

(1) 配置文件1：hadoop-env.sh

修改JAVA_HOME值（export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_51）

(2) 配置文件2：yarn-env.sh

注意：这个可以不用配置

修改JAVA_HOME值（exportJAVA_HOME=/usr/lib/jvm/jdk1.7.0_51）

(3) 配置文件3：slaves，指定dataNode的列表。

SY-0226

SY-0225

SY-0355

(4) 配置文件4：core-site.xml

<name>fs.defaultFS</name>

<description>The name of the default file system. A URI whose

scheme and authority determine the FileSystem implementation. The

uri's scheme determines the config property (fs.SCHEME.impl) naming

the FileSystem implementation class. The uri's authority is used to

determine the host, port, etc. for a filesystem.</description>

</property>

</configuration>

(5) 配置文件5：hdfs-site.xml

<name>dfs.nameservices</name>

<value>hadoop-test</value>

Comma-separated list of nameservices.

</description>

</property>

配置HDFS的命名空间：hadoop-cluster1,hadoop-cluster2这两个名字可以自己起

<name>dfs.ha.namenodes.hadoop-test</name>

The prefix for a given nameservice, contains a comma-separated

list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE).

</description>

</property>

这两个名字是可以自己起的

<name>dfs.namenode.rpc-address.hadoop-cluster1.nn1</name>

RPC address for nomenode1 of hadoop-cluster1

</description>

</property>

<name>dfs.namenode.rpc-address.hadoop-cluster1.nn2</name>

RPC address for nomenode2 of hadoop-test

</description>

</property>

<name>dfs.namenode.http-address.hadoop-test.nn1</name>

The address and the base port where the dfs namenode1 web ui will listenon.

</description>

</property>

<name>dfs.namenode.http-address.hadoop-test.nn2</name>

The address and the base port where the dfs namenode2 web ui will listenon.

</description>

</property>

<name>dfs.namenode.name.dir</name>

<value>file:///home/dongxicheng/hadoop/hdfs/name</value>

<description>Determines where on the local filesystem the DFS namenode

should store the name table(fsimage). If this is a comma-delimited list

of directories then the name table is replicated in all of the

directories, for redundancy. </description>

</property>

namenode元数据信息存放地址，配置一个本地地址；可以配置多个，用逗号分隔

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://SY-0355:8485;SY-0225:8485;SY-0226:8485/hadoop-test</value>

<description>A directory on shared storage between the multiplenamenodes

inan HA cluster. This directory will be written by the active and read

bythe standby in order to keep the namespaces synchronized. This directory

does not need to be listed in dfs.namenode.edits.dir above. It should be

left empty in a non-HA cluster.

</description>

</property>

注意：配置每个命名空间的journalNode，JournalNode应配置为奇数个，用分号隔开；每个NameNode的命名空间有active和standby两个NameNode节点，并且这两个节点共用相同的JournalNode；千万要注意，在不同的NameNode配置文件中，NameNode命名空间需要指定不同的名字，active和standby两个NameNode节点的命名空间名字相同。例如，NameNode节点nn1和nn2为上面的hadoop-cluster1。

<name>dfs.datanode.data.dir</name>

<value>file:///home/dongxicheng/hadoop/hdfs/data</value>

<description>Determines where on the local filesystem an DFS datanode

should store its blocks. If thisis a comma-delimited

list of directories, then data will be stored in all named

directories, typically on different devices.

Directories that do not exist are ignored.

</description>

</property>

dateNode的本地目录：建议也用多个，可以提高并发。

<name>dfs.ha.automatic-failover.enabled</name>

<value>false</value>

Whether automatic failover is enabled. See the HDFS High

Availability documentation for details on automatic HA

configuration.

</description>

</property>

<name>dfs.journalnode.edits.dir</name>

<value>/home/dongxicheng/hadoop/hdfs/journal/</value>

</property>

</configuration>

注意：最好在/home/dongxicheng/hadoop/hdfs下创建好以下目录[hadoop@h11 hdfs]$ mkdir data name journal

(6) 配置文件6：mapred-site.xml

<name>mapreduce.framework.name</name>

<description>The runtime framework for executing MapReduce jobs.

Can be one of local, classic or yarn.

</description>

</property>

<name>mapreduce.jobhistory.address</name>

<description>MapReduce JobHistory Server IPChost:port</description>

</property>

<name>mapreduce.jobhistory.webapp.address</name>

<description>MapReduce JobHistory Server Web UIhost:port</description>

</property>

</configuration>

(7) 配置文件7：yarn-site.xml

<description>The hostname of the RM.</description>

<name>yarn.resourcemanager.hostname</name>

</property>

配置.resourcemanager的节点，修改为相应的主机名

<description>The address of the applications manager interface inthe RM.</description>

<name>yarn.resourcemanager.address</name>

<value>${yarn.resourcemanager.hostname}:8032</value>

</property>

<description>The address of the schedulerinterface.</description>

<name>yarn.resourcemanager.scheduler.address</name>

<value>${yarn.resourcemanager.hostname}:8030</value>

</property>

<description>The http address of the RM webapplication.</description>

<name>yarn.resourcemanager.webapp.address</name>

<value>${yarn.resourcemanager.hostname}:8088</value>

</property>

<description>The https adddress of the RM webapplication.</description>

<name>yarn.resourcemanager.webapp.https.address</name>

<value>${yarn.resourcemanager.hostname}:8090</value>

</property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>${yarn.resourcemanager.hostname}:8031</value>

</property>

<description>The address of the RM admininterface.</description>

<name>yarn.resourcemanager.admin.address</name>

<value>${yarn.resourcemanager.hostname}:8033</value>

</property>

<description>The class to use as the resourcescheduler.</description>

<name>yarn.resourcemanager.scheduler.class</name>

<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>

</property>

此时用的调度器是FairScheduler，默认CapacitySchedule；

<description>fair-scheduler conf location</description>

<name>yarn.scheduler.fair.allocation.file</name>

<value>${yarn.home.dir}/etc/hadoop/fairscheduler.xml</value>

</property>

需要创建fairscheduler.xml，并拷贝到${yarn.home.dir}目录下，fairscheduler.xml的配置在下面

<description>List of directories to store localized files in. An

application's localized file directory will be found in:

${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}.

Individual containers' work directories, called container_${contid},will

be subdirectories of this.

</description>

<name>yarn.nodemanager.local-dirs</name>

<value>/home/dongxicheng/hadoop/yarn/local</value>

</property>

yarn.nodemanager的本地目录，需要创建；最好是配置多个目录，用逗号分隔

<description>Whether to enable log aggregation</description>

<name>yarn.log-aggregation-enable</name>

</property>

<description>Where to aggregate logs to.</description>

<name>yarn.nodemanager.remote-app-log-dir</name>

</property>

<description>Amount of physical memory, in MB, that can beallocated

for containers.</description>

<name>yarn.nodemanager.resource.memory-mb</name>

</property>

注意：我的NM虚拟机是1G内存，1核CPU，当该值配置小于1024时，NM是无法启动的！会报错：

NodeManager from slavenode2 doesn't satisfy minimum allocations, Sending SHUTDOWN signalto the NodeManager.

所以，在配置此参数时，应按照NameManager虚拟机来配置。

<description>Number of CPU cores that can be allocated

for containers.</description>

<name>yarn.nodemanager.resource.cpu-vcores</name>

</property>

<description>the valid service name should only contain a-zA-Z0-9_and can not start with numbers</description>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

（8）、配置文件8：fairscheduler.xml的配置

<?xml version="1.0"?>

<minResources>102400 mb, 50 vcores </minResources> 指定使用的内存数和cpu核数

<maxResources>153600 mb, 100 vcores </maxResources>

<aclSubmitApps>root,yarn,search,hdfs</aclSubmitApps> 允许访问的用户

</queue>

<minResources>102400 mb, 30 vcores</minResources>

<maxResources>153600 mb, 50 vcores</maxResources>

</queue>

<minResources>102400 mb, 30 vcores</minResources>

<maxResources>153600 mb, 50vcores</maxResources>

</queue>

</allocations>

7、将hadoop文件夹拷贝到其他节点上。

注意：修改不同NameNode命名空间下active NameNode、standby NameNode的hdfs-site.xml配置文件的下列参数，需要指定不同的命名空间。如，hadoop-cluster1需要两两一样

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://SY-0355:8485;SY-0225:8485;SY-0226:8485/hadoop-cluster1</value>

scp –r/home/fyzwjd/hadoopfyzwjd@hadoopSlave1:~/

scp –r/home/fyzwjd/hadoopfyzwjd@hadoopSlave2:~/

8、启动与停止集群

注意：所有操作均在Hadoop部署目录下进行。

启动Hadoop集群：

Step1 :

在各个JournalNode节点上，输入以下命令启动journalnode服务：

sbin/hadoop-daemon.sh start journalnode

Step2:

在[nn1]上，对其进行格式化，并启动：

bin/hdfs namenode -format

sbin/hadoop-daemon.sh start namenode

Step3:

在[nn2]上，同步nn1的元数据信息：

bin/hdfs namenode -bootstrapStandby

Step4:

启动[nn2]：

sbin/hadoop-daemon.sh start namenode

经过以上四步操作，nn1和nn2均处理standby状态

Step5:

将[nn1]切换为Active

bin/hdfs haadmin -transitionToActive nn1

Step6:

在[nn1]上，启动所有datanode

sbin/hadoop-daemons.sh start datanode

然后启动yarn：

Sbin/start-yarn.sh

注意：如果，不启动yarn，无法跑MR程序。

在运行MRJS的slave1上执行以下命令启动MR JobHistory Server：

sbin/mr-jobhistory-daemon.sh starthistoryserver

停止集群

在RM和NN所在节点master执行：

先停止yarn：

sbin/stop-yarn.sh

再停止hdfs：

sbin/stop-dfs.sh

在运行JobHistoryServer的slave1上执行：

停止JobHistoryServer：

sbin/mr-jobhistory-daemon.sh stophistoryserver

第六步：再次启动

注意，再次启动时，所有的格式化命令都不用执行了！！！

**码上人生**

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
集群安装：HA与Yarn（测试）

准备工作：修改网卡和主机名（在三个节点同时操作） 1）、修改网卡1：注意：此处只用一个网卡就可以了，本地IP（10.10.10.1）可以不用配置Vi/etc/sysconfig/network-scripts/ifcfg-eth0 修改网卡2：设置好后重启网络服务：看网卡的配置信息： 2）、修改主机名
复制链接

扫一扫

专栏目录