linux完全分布式作用,hadoop 完全分布式部署

一.实验环境准备

需要准备四台linux操作系统的服务器，配置参数最好一样，由于我的虚拟机是之前伪分布式部署而来的，因此我的环境都一致，并且每天虚拟机默认都是hadoop伪分布式哟！

1>.namenode服务器(172.20.20.228)

2>.datanode服务器(172.20.20.226-220)

二.修改hadoop的配置文件

修改的配置文件路径是我之前拷贝的full目录，绝对路径是：“/tosp/opt/hadoop”,修改这个目录下的文件之后，我们将hadoop目录连接过来即可，当你需要伪分布式或者本地模式的时候只需要改变软连接指向的目录即可，这样就轻松实现了三种模式配置文件和平相处的局面。

1>.core-site.xml 配置文件

[root@cdh14 ~]$ more /tosp/opt/hadoop/etc/hadoop/core-site.xml

fs.defaultfs

hdfs://cdh14:9000

hadoop.tmp.dir

/tosp/opt/hadoop

[root@cdh14 ~]$

2>.hdfs-site.xml 配置文件

[root@cdh14 ~]$ more /tosp/opt/hadoop/etc/hadoop/hdfs-site.xml

dfs.replication

2

[root@cdh14 ~]$

3>.mapred-site.xml 配置文件

[root@cdh14 ~]$ more /tosp/opt/hadoop/etc/hadoop/mapred-site.xml

mapreduce.framework.name

yarn

[root@cdh14 ~]$

4>.yarn-site.xml配置文件

[root@cdh14 ~]$ more /tosp/opt/hadoop/etc/hadoop/yarn-site.xml

yarn.resourcemanager.hostname

cdh14

yarn.nodemanager.aux-services

mapreduce_shuffle

[root@cdh14 ~]$

5>.slaves配置文件

[root@cdh14 ~]$ more /tosp/opt/hadoop/etc/hadoop/slaves

#该配置文件的作用：是namenode用与记录需要连接哪些datanode服务器节点，用与启动或停止服务时发送远程命令指令的目标主机。

cdh14

cdh12

cdh11

cdh10

cdh9

cdh8

cdh7

[root@cdh14 ~]$

三.在namenode节点上配置免密码登录各datanode节点

1>.在本地上生成公私秘钥对(生成之前，把上次部署伪分布式的秘钥删除掉)

[root@cdh14 ~]$ rm -rf ~/.ssh/*

[root@cdh14 ~]$ ssh-keygen -t rsa -p '' -f ~/.ssh/id_rsa

generating public/private rsa key pair.

your identification has been saved in /home/root/.ssh/id_rsa.

your public key has been saved in /home/root/.ssh/id_rsa.pub.

the key fingerprint is:

a3:a4:ae:d8:f7:7f:a2:b6:d6:15:74:29:de:fb:14:08 root@cdh14

the key's randomart image is:

+--[ rsa 2048]----+

| . |

| e o |

| o = . |

| o o . |

| . s . . . |

| o . .. . . |

| . .. . o |

| o .. o o . . |

|. oo.+++.o |

+-----------------+

[root@cdh14 ~]$

2>.使用ssh-copy-id命令分配公钥到datanode服务器(172.20.20.228)

[root@cdh14 ~]$ ssh-copy-id root@cdh14

the authenticity of host 'cdh14 (172.16.30.101)' can't be established.

ecdsa key fingerprint is fa:25:bc:03:7e:99:eb:12:1e:bc:a8:c9:ce:39:ba:7b.

are you sure you want to continue connecting (yes/no)? yes

/usr/bin/ssh-copy-id: info: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: info: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

root@cdh14's password:

number of key(s) added: 1

now try logging into the machine, with: "ssh 'root@cdh14'"

and check to make sure that only the key(s) you wanted were added.

[root@cdh14 ~]$ ssh cdh14

last login: fri may 25 18:35:40 2018 from 172.16.30.1

[root@cdh14 ~]$ who

root pts/0 2018-05-25 18:35 (172.16.30.1)

root pts/1 2018-05-25 19:17 (cdh14)

[root@cdh14 ~]$ exit

logout

connection to cdh14 closed.

[root@cdh14 ~]$ who

root pts/0 2018-05-25 18:35 (172.16.30.1)

[root@cdh14 ~]$

3>.使用ssh-copy-id命令分配公钥到datanode服务器(172.20.20.226-220)

[root@cdh14 ~]$ ssh-copy-id root@chd12-cdh7

/usr/bin/ssh-copy-id: info: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: info: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

root@s102's password:

number of key(s) added: 1

now try logging into the machine, with: "ssh 'root@s102'"

and check to make sure that only the key(s) you wanted were added.

[root@cdh14 ~]$ ssh s102

last login: fri may 25 18:35:42 2018 from 172.16.30.1

[root@s102 ~]$ who

root pts/0 2018-05-25 18:35 (172.16.30.1)

root pts/1 2018-05-25 19:19 (cdh14)

[root@s102 ~]$ exit

logout

connection to s102 closed.

[root@cdh14 ~]$ who

root pts/0 2018-05-25 18:35 (172.16.30.1)

[root@cdh14 ~]$

注意：以上是普通使配置免密登录，root用户配置方法一致，最好也配置上root用户的免密登录，因为下文我会执行相应的shell脚本。

五.启动服务并验证是否成功

1>.格式化文件系统

2>.启动hadoop

3>.用自定义脚本验证namenode和datanode是否已经正常启动

[root@cdh14 ~]$ jps