集群搭建与启动

目录

1.克隆虚拟机

2.设置免密登录

3.启动集群

4. web端集群查看

---

1.克隆虚拟机

       关闭master主机，在VM软件左侧系统资源中单击master,选择“管理”列表下的克隆选项，如图1-1所示。在弹出克隆虚拟机向导界面上，点击下一步，在选中"虚拟机中当前状态"下点击下一页，选择克隆类型。完整克隆的虚拟机文件相对独立并且安全，采用此处完整克隆方式。对主机命名为slave1,并确定存放地址（例：D:\CentOS\slave1,与master主机放在同一个文件下，方便管理），进度条完成说明克隆成功。

 图1-1  虚拟机克隆选项向导

        开启克隆的slave1主机，登录后，会发现主机名、ip地址与master主机一致，为了避免IP冲突，需要更改slave1主机名和静态IP，可参考之前设置master主机名和静态IP地址步骤。之前映射master主机名和IP地址时，已确定slave1主机的ip地址为192.168.38.129，因此此次设置应保持一致。

[zhao@master ~]$ su - root
Password: 
Last login: Wed Sep 29 19:57:28 CST 2021 from 192.168.38.1 on pts/2
[root@master ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:96:8a:db brd ff:ff:ff:ff:ff:ff
    inet 192.168.38.128/24 brd 192.168.38.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::268d:bf52:8b8f:b5be/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::d584:59a1:6240:d215/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether 52:54:00:ca:34:4d brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
    link/ether 52:54:00:ca:34:4d brd ff:ff:ff:ff:ff:ff
[root@master ~]# hostnamectl set-hostname slave1
[root@master ~]# bash
[root@slave1 ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33
[root@slave1 ~]# service network restart
Restarting network (via systemctl):                        [  OK  ]
[root@slave1 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:27:31:d8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.38.129/24 brd 192.168.38.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::d584:59a1:6240:d215/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether 52:54:00:ca:34:4d brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
    link/ether 52:54:00:ca:34:4d brd ff:ff:ff:ff:ff:ff
[root@slave1 ~]#

       若以上步骤是利用远程连接软件操作，网络服务重新启动后，连接会中断，是因为slave1主机IP已经改为192.168.38.129，而之前连接是通过192.168.38.128IP连接的。

         重复以上步骤，可克隆slave2主机，该主机的静态IP为192.168.38.130。

2.设置免密登录

       下面以master主机为例，让主机master免密进入master、slave1和slave2主机。确保三台主机都开启，并利用远程连接软件连接master主机。

        输入ssh-keygen -t rsa命令，并按四次回车，/root目录下生成密钥文件。执行命令ssh-copy-id slave1，将生成的公钥复制到slave1，实现master主机到slave1主机的单向免密登录。

[root@master ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:OAaMu3EAxiiEnbuepVUHdde1tW5QV9W6RtL9B/oI+qI root@master
The key's randomart image is:
+---[RSA 2048]----+
|*+ .   .. . .. oX|
|=ooo  .  . .  o *|
|. o.o  .     o = |
|  .o ....   . B .|
|  o...+.S    + =.|
|  .+o. .  . . + o|
| ..=     . . +  .|
|  +     o   . .  |
|      E. o.      |
+----[SHA256]-----+
[root@master ~]# ssh-copy-id slave1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'slave1 (192.168.38.129)' can't be established.
ECDSA key fingerprint is SHA256:CYMn0hmNUEip41Lu/4wLlpu0HI/CTfEWInMellcOKk4.
ECDSA key fingerprint is MD5:6c:fe:5c:9b:0e:bb:9f:85:dd:0f:3c:27:0a:e2:05:8b.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@slave1's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'slave1'"
and check to make sure that only the key(s) you wanted were added.

[root@master ~]# ssh slave1
Last login: Thu Sep 30 19:44:40 2021 from 192.168.38.1
[root@slave1 ~]#

        实现master主机到slave2以及master主机的单向免密登录，执行命令ssh-copy-id slave2和ssh-copy-id master即可。注意以上命令都需要在mstart主机上。

[root@slave1 ~]# ssh master
[root@master ~]# ssh-copy-id slave2
[root@master ~]# ssh-copy-id master

       主机slave1和slave2主机实现免密登录，重复上述步骤即可。所以，要实现3台主机自由登录，需要执行ssh-copy-id 这条命令9次。

3.启动集群

        如果集群是第一次启动，需要格式化NameNode。

[root@master ~]# cd /export/servers/hadoop-2.6.4/
[root@master hadoop-2.6.4]# bin/hdfs namenode -format

        输入对应命令后，若出现图3-1所示标志，说明NameNode格式化成功。后续若出现节点集群不能正常启动时，需要重新格式化NameNode时，一定要先停止上次启动的所有Namenode和datanode进程，然后再删除tmp和log数据。

 图3-1 格式化NameNode成功界面

        整体启动hdfs和yarn。NameNode和ResourceManger不在同一台主机上，应在拥有NameNode节点上的master主机上启动hdfs，在拥有ResouceManager节点的slave2主机上启动yarn。

[root@master hadoop-2.6.4]# start-dfs.sh

[root@slave2 ~]# start-yarn.sh

         输入jps命令，查看主机节点启动情况。

[root@master ~]# jps
2806 Jps
2490 NameNode

[root@slave1 ~]# jps
3973 DataNode
4071 SecondaryNameNode
4264 NodeManager
4505 Jps

[root@slave2 ~]# jps
4466 ResourceManager
4167 NodeManager
4042 DataNode
4767 Jps

        若需关闭集群，在各主机上输入相应关闭的命令。启动集群是先hdfs后yarn，关闭集群是先yarn后hdfs。

4. web端集群查看

在浏览器输入http://192.168.38.128:50070,查看hdfs信息，如图4-1所示。

 图4-1 hdfs的web端界面

在浏览器输入http://192.168.38.130:8088,查看yarn信息，如图4-2所示。

  图4-2 yarn的web端界面

在浏览器输入http://192.168.38.129:50090,查看SecondaryNameNode信息，如图4-3所示。

 

   图4-3 SecondaryNameNode的web端界面