hadoop 集群搭建

最新推荐文章于 2022-11-23 11:31:05 发布

就叫阿坤吧

最新推荐文章于 2022-11-23 11:31:05 发布

阅读量811

点赞数

分类专栏： hadoop

本文链接：https://blog.csdn.net/weixin_45019914/article/details/90599397

版权

hadoop 专栏收录该内容

3 篇文章 0 订阅

订阅专栏

1、hadoop概述

hadoop是开源的，分布式存储+分布式计算平台。hadoop可以用来搭建大型数据仓库，PB级数据的存储、处理、分析、统计等业务，具有高扩展、低成本、成熟的生态圈的特点，本文档介绍搭建hadoop的步骤，以及搭建中的注意事项。

2、文档规划

搭建hadoop集群需要两台虚拟机，所以有节点一，节点二，后面再动态添加一个节点。

	节点一	节点二	节点三
操作系统	Linux	Linux	Linux
操作系统版本	Centos 64位	Cebtos 64位	Cebtos 64位
IP地址	192.168.33.132	192.168.33.131	192.168.56.130
安装包版本	hadoop-2.7.7.tar.gz	hadoop-2.7.7.tar.gz	hadoop-2.7.7.tar.gz
Linux普通用户	lan	lan	lan
安装包路径	/home/lan	/home/lan	/home/lan

3、配置hadoop安装条件

3.1、检查时钟

查看时钟是否同步

date
Sun May  5 15:27:08 CST 2019
date
Sun May  5 15:28:10 CST 2019

3.2、查看主机名

查看两个节点的主机名，节点一为主要的数据库，所以为master，节点二为辅助节点一所以名为slave。

more /etc/hostname
master
more /etc/hostname
slave

3.3、关闭防火墙

搭建hadoop需要关闭防火墙，两个节点都要关闭，让两个节点相通。

systemctl stop firewalld.service
禁止开机启动
[root@master ~] systemctl disable firewalld.service
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

[root@slave ~] systemctl disable firewalld.service
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

3.4、查看IP地址

节点一
[root@master ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.33.128  netmask 255.255.255.0  broadcast 192.168.33.255
        inet6 fe80::a3d8:1d91:6a85:cfcb  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:bb:0f:af  txqueuelen 1000  (Ethernet)
        RX packets 445  bytes 47282 (46.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 364  bytes 46987 (45.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 84  bytes 9156 (8.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 84  bytes 9156 (8.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:99:ed:d1  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
节点二
[root@slave ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.33.129  netmask 255.255.255.0  broadcast 192.168.33.255
        inet6 fe80::ef11:ce2a:f40e:3a50  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:73:79:f5  txqueuelen 1000  (Ethernet)
        RX packets 173  bytes 20680 (20.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 197  bytes 29605 (28.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 84  bytes 9156 (8.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 84  bytes 9156 (8.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:48:c6:97  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

3.5、配置hosts

配置两个节点的hosts

节点一
[root@master ~]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.33.128 master
192.168.33.129 slave
节点二
[root@slave ~]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.33.128 master
192.168.33.129 slave

检查配置效果

节点一
[root@master ~]# ping slave
PING slave (192.168.33.129) 56(84) bytes of data.
64 bytes from slave (192.168.33.129): icmp_seq=1 ttl=64 time=0.334 ms
64 bytes from slave (192.168.33.129): icmp_seq=2 ttl=64 time=0.293 ms
64 bytes from slave (192.168.33.129): icmp_seq=3 ttl=64 time=0.314 ms
64 bytes from slave (192.168.33.129): icmp_seq=4 ttl=64 time=0.205 ms
64 bytes from slave (192.168.33.129): icmp_seq=5 ttl=64 time=0.293 ms
^C
--- slave ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7000ms
rtt min/avg/max/mdev = 0.185/0.292/0.378/0.065 ms
节点二
[root@slave ~]# ping master
PING master (192.168.33.128) 56(84) bytes of data.
64 bytes from master (192.168.33.128): icmp_seq=1 ttl=64 time=0.400 ms
64 bytes from master (192.168.33.128): icmp_seq=2 ttl=64 time=0.851 ms
64 bytes from master (192.168.33.128): icmp_seq=3 ttl=64 time=0.272 ms
64 bytes from master (192.168.33.128): icmp_seq=4 ttl=64 time=0.551 ms
64 bytes from master (192.168.33.128): icmp_seq=5 ttl=64 time=0.285 ms
^C
--- master ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5002ms
rtt min/avg/max/mdev = 0.272/0.443/0.851/0.206 ms

3.6、创建目录

创建目录在root用户下，两个节点都要创建。

[root@master ~]# mkdir /usr/java

移动安装包到指定目录下

[root@master ~]Mv /home/lan/jdk-8u144-linux-x64.tar.gz /usr/java

然后解压

tar -zxvf jdk-8u144-linux-x64.tar.gz

3.7、配置环境变量

在普通用户下，配置环境标量，两个节点都要配置。

cat .bash_profile 
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/.local/bin:$HOME/bin
export PATH
export PATH
export JAVA_HOME=/usr/java/jdk1.8.0_144
export PATH=$JAVA_HOME/bin:$PATH

验证环境变量配置文件

java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

3.8、配置秘钥

在普通用户下执行，生成秘钥，配置秘钥之后，两个主机就能来回切换。两个节点都要执行此命令。

[lan@master ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/lan/.ssh/id_rsa): 
Created directory '/home/lan/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/lan/.ssh/id_rsa.
Your public key has been saved in /home/lan/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:C4JUA33ksqrZ9ayKJ92sU8GgzKr9Oligma6f6U6lCdA lan@master
The key's randomart image is:
+---[RSA 2048]----+
|  .oo..          |
| ....o.          |
|+.Eo...          |
|++ .oo           |
|++. +.. S        |
|=..=.. . .       |
|o==+.   .        |
|+B*+oo           |
|**@Bo.o          |
+----[SHA256]-----+
查看生成情况
[lan@master ~]$ cd .ssh/
[lan@master .ssh]$ ll
total 8
-rw-------. 1 lan lan 1679 May  5 16:45 id_rsa（秘钥私有）
-rw-r--r--. 1 lan lan  392 May  5 16:45 id_rsa.pub（秘钥公有）
将秘钥公有文件内容写到文件下
[lan@master .ssh]$ cat id_rsa.pub >> authorized_keys
[lan@master .ssh]$ ll
total 12
-rw-rw-r--. 1 lan lan  392 May  5 16:49 authorized_keys
-rw-------. 1 lan lan 1679 May  5 16:45 id_rsa
-rw-r--r--. 1 lan lan  392 May  5 16:45 id_rsa.pub

修改权限

[lan@master .ssh]$ chmod 600 authorized_keys

传送文件，到普通用户的家目录下。

[lan@master .ssh]$ scp authorized_keys lan@slave:~/
The authenticity of host 'slave (192.168.33.129)' can't be established.
ECDSA key fingerprint is SHA256:BQJkKx7SdhgvffUIuYlN+5R28nvou60/ZDSpsKCM69U.
ECDSA key fingerprint is MD5:8e:d2:43:bc:02:f9:96:3c:67:ab:54:86:3f:82:7b:5c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave,192.168.33.129' (ECDSA) to the list of known hosts.
lan@slave's password: 
authorized_keys

生成目录，将传送到节点二的文件传送到目录下，查看

[lan@slave ~]$ cd .ssh/
[lan@slave .ssh]$ ll
total 12
-rw-------. 1 lan lan  392 May  5 16:51 authorized_keys
-rw-------. 1 lan lan 1679 May  5 16:54 id_rsa
-rw-r--r--. 1 lan lan  391 May  5 16:54 id_rsa.pub

4、配置hadoop文件

4.1、解压文件

这个只在节点一执行，用普通用户解压hadoop文件

tar -zxvf hadoop-2.7.7.tar.gz

修改配置文件

[lan@master hadoop]$ pwd
/home/lan/hadoop-2.7.7/etc/hadoop
[lan@master hadoop]$ vim hadoop-env.sh

4.2、配置文件

将jdk的路径添加到hadoop-env.sh文件中，因为hadoop是Java语言搭建的，该文件的目的是说明Java文件去哪个路径下可以找到。

vim hadoop-env.sh

修改yarn-env.sh，该文件的作用和上面一样。

[lan@master hadoop]$ vim yarn-env.sh

在文件中添加以下内容，该文件是修改进程的路径。

[lan@master hadoop]$ vim core-site.xml 
<configuration>
 <property> 
 <name>fs.defaultFS</name>
 <value>hdfs://master:9000</value>
 </property>
 <property>
 <name>hadoop.tmp.dir</name>
 <value>/home/lan/hadoopdata</value>
 </property>
</configuration>

在文件中添加以下内容

[lan@master hadoop]$ vim hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

修改文件

[lan@master hadoop]$ vim yarn-site.xml 
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:18040</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:18030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:18025</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:18141</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:18088</value>
</property>

修改文件，该文件是说明这是几个节点。

[lan@master hadoop]$ cp mapred-site.xml.template mapred-site.xml 
[lan@master hadoop]$ vim mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>1</value>
</property>

将从节点的主机名添加到文件中。

[lan@master hadoop]$ vim slaves
Slave

5、配置环境变量

5.1、传送文件

传送整个文件夹到第二节点

[lan@master ~]$ scp -r hadoop-2.7.7 lan@slave:~/

5.2、修改环境变量

修改普通用户配置文件将以下内容添加到配置文件中

export HADOOP_HOME=/home/lan/hadoop-2.7.7
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

5.3、格式化磁盘

格式化磁盘

[lan@master ~]$ hdfs namenode -format
19/05/05 17:45:28 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = master/192.168.33.128
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.7.7

6、检查搭建是否成功

6.1、启动集群数据库

[lan@master ~]$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
The authenticity of host 'master (192.168.33.128)' can't be established.
ECDSA key fingerprint is SHA256:LIyf64qMkgdCtl6os61aNDcI5yf3uD2tN1EAacXvG9k.
ECDSA key fingerprint is MD5:10:6c:c7:f4:35:fa:87:30:87:5a:4f:3e:32:87:eb:f1.
Are you sure you want to continue connecting (yes/no)? yes
master: Warning: Permanently added 'master,192.168.33.128' (ECDSA) to the list of known hosts.
master: starting namenode, logging to /home/lan/hadoop-2.7.7/logs/hadoop-lan-namenode-master.out
slave: starting datanode, logging to /home/lan/hadoop-2.7.7/logs/hadoop-lan-datanode-slave.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is SHA256:LIyf64qMkgdCtl6os61aNDcI5yf3uD2tN1EAacXvG9k.
ECDSA key fingerprint is MD5:10:6c:c7:f4:35:fa:87:30:87:5a:4f:3e:32:87:eb:f1.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /home/lan/hadoop-2.7.7/logs/hadoop-lan-secondarynamenode-master.out
starting yarn daemons
starting resourcemanager, logging to /home/lan/hadoop-2.7.7/logs/yarn-lan-resourcemanager-master.out
slave: starting nodemanager, logging to /home/lan/hadoop-2.7.7/logs/yarn-lan-nodemanager-slave.out

6.2、显示集群进程

[lan@master ~]$ jps
20624 ResourceManager
20274 NameNode
20475 SecondaryNameNode
20879 Jps

6.3、执行运算

查看是否搭建集群成功，输入，运算脚本。

 hadoop jar hadoop-2.7.7/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.7.jar pi 5 5

7、添加节点

为集群动态添加一个节点，首先要再见一台虚拟机，内存为1GB，磁盘空间为20GB。

7.1、配置Linux

7.1.1、同步时间

节点一

[root@master ~]# date
Mon May  6 19:44:57 CST 2019

节点二

[root@slave network-scripts]# date
Mon May  6 19:44:59 CST 2019

节点三

[root@slave01 ~]# date
Mon May  6 19:45:13 CST 2019

7.1.2、查看主机名

配置Linux都要在超级用户下执行，普通用户没有权限，首先查看细节点的主机名。

[root@slave01 network-scripts]# hostname
slave01

7.1.3、查看IP地址

查看IP地址，连接到xshell工具。

[root@slave01 network-scripts]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.33.130  netmask 255.255.255.0  broadcast 192.168.33.255
        inet6 fe80::dc39:d32f:82a0:c1cf  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:22:77:5f  txqueuelen 1000  (Ethernet)
        RX packets 465  bytes 47003 (45.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 418  bytes 102765 (100.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 116  bytes 12676 (12.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 116  bytes 12676 (12.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:7f:b0:f6  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

7.1.4、关闭防火墙

关闭防火墙，让三个节点能够畅通。

[root@slave01 ~]# systemctl stop firewalld.service

关闭开机启动防火墙，永久生效。

[root@slave01 ~]# systemctl disable firewalld.service

7.1.5、修改主机名

三个节点都要修改
节点一

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.33.132 master
192.168.33.131 slave
192.168.33.130 slave01

节点二

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.33.132 master
192.168.33.131 slave
192.168.33.130 slave01

节点三

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.33.132 master
192.168.33.131 slave
192.168.33.130 slave01

修改后验证
节点一

[root@master ~]# ping slave01
PING slave01 (192.168.33.130) 56(84) bytes of data.
64 bytes from slave01 (192.168.33.130): icmp_seq=1 ttl=64 time=1.10 ms
64 bytes from slave01 (192.168.33.130): icmp_seq=2 ttl=64 time=0.336 ms
64 bytes from slave01 (192.168.33.130): icmp_seq=3 ttl=64 time=0.362 ms
^C
--- slave01 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.336/0.599/1.101/0.355 ms

节点二

[root@slave ~]# ping slave01
PING slave01 (192.168.33.130) 56(84) bytes of data.
64 bytes from slave01 (192.168.33.130): icmp_seq=1 ttl=64 time=0.960 ms
64 bytes from slave01 (192.168.33.130): icmp_seq=2 ttl=64 time=0.324 ms
64 bytes from slave01 (192.168.33.130): icmp_seq=3 ttl=64 time=0.299 ms
^C
--- slave01 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.299/0.527/0.960/0.307 ms

节点三

[root@slave01 ~]# ping master
PING master (192.168.33.132) 56(84) bytes of data.
64 bytes from master (192.168.33.132): icmp_seq=1 ttl=64 time=0.346 ms
64 bytes from master (192.168.33.132): icmp_seq=2 ttl=64 time=1.10 ms
64 bytes from master (192.168.33.132): icmp_seq=3 ttl=64 time=0.344 ms
^C
--- master ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2005ms
rtt min/avg/max/mdev = 0.344/0.598/1.106/0.359 ms

7.1.6、创建目录

在root用户下创建目录

[root@slave01 ~]# mkdir /usr/java

移动安装包到指定目录下

[root@slave01~]Mv /home/lan/jdk-8u144-linux-x64.tar.gz /usr/java

然后解压

[root@slave01~]tar -zxvf jdk-8u144-linux-x64.tar.gz

7.1.7、配置环境变量

在普通用户下，配置环境标量。

[root@slave01~]vim .bash_profile 
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/.local/bin:$HOME/bin
export PATH
export PATH
export JAVA_HOME=/usr/java/jdk1.8.0_144
export PATH=$JAVA_HOME/bin:$PATH

验证环境变量配置文件

[root@slave01~]java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

7.1.8、配置秘钥

配置秘钥，已连接到主节点，让主节点可以登录。

[root@slave01~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/lan/.ssh/id_rsa): 
Created directory '/home/lan/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/lan/.ssh/id_rsa.
Your public key has been saved in /home/lan/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:C4JUA33ksqrZ9ayKJ92sU8GgzKr9Oligma6f6U6lCdA lan@master
The key's randomart image is:
+---[RSA 2048]----+
|  .oo..          |
| ....o.          |
|+.Eo...          |
|++ .oo           |
|++. +.. S        |
|=..=.. . .       |
|o==+.   .        |
|+B*+oo           |
|**@Bo.o          |
+----[SHA256]-----+

查看生成情况

[root@slave01~]$ cd .ssh/
[lan@master .ssh]$ ll
total 8
-rw-------. 1 lan lan 1679 May  5 16:45 id_rsa
-rw-r--r--. 1 lan lan  392 May  5 16:45 id_rsa.pub
将主节点之前生成的秘钥传送到新节点。
[lan@master .ssh]$ scp authorized_keys lan@slave01:~/

7.2、修改配置文件

7.2.1、修改文件

修改文件，写上这是第二个副节点

/home/lan/hadoop-2.7.7/etc/hadoop
[lan@slave01 hadoop]$  vim mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>2</value>
</property>
</configuration>
[lan@master ~]$ scp -r hadoop-2.7.7 lan@slave:~/

修改节点名称文件，三个节点都要修改，将两个副节点的名称写入到这个文件中。

节点一
vim slaves
slave
slave01
节点二
vim slaves
slave
slave01
节点三
vim slaves
slave
slave01

7.2.2、传送文件

传送整个文件夹到第二节点

[lan@master ~]$ scp -r hadoop-2.7.7 lan@slave01:~/

修改普通用户配置文件将以下内容添加到配置文件中

export HADOOP_HOME=/home/lan/hadoop-2.7.7
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

7.3、启动程序

因为是动态安装，所以不能关闭虚拟机，要首先执行两个程序，才能启动成功。

[lan@slave01 hadoop-2.7.7]$ sbin/hadoop-daemon.sh start datanode
starting datanode, logging to /home/lan/hadoop-2.7.7/logs/hadoop-lan-datanode-slave01.out
[lan@slave01 hadoop-2.7.7]$ sbin/yarn-daemon.sh start nodemanager
starting nodemanager, logging to /home/lan/hadoop-2.7.7/logs/yarn-lan-nodemanager-slave01.out

启动之后，输入命令查看。

[lan@slave01 hadoop-2.7.7]$ jps
10211 DataNode
10311 NodeManager
10463 Jps

启动成功。

8、遇见的问题

8.1、 IP地址

在使用的时候遇见的问题，就是网卡在是动态的情况下，总是发生改变，改变之后集群不能成功启动，无法连接到工具。将网卡配置改为静态网卡，问题解决，三个节点网卡都改为静态网卡。修改网卡必须要到root用户下，普通用户没有权限。
修改方式：将网卡改为静态，在最后面添加IP地址，网段，网关。

节点一
[root@master network-scripts]# vim /etc/sysconfig/network-scripts/ifcfg-ens33 
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="5a8ac1bd-3ec6-4a9b-8c89-f460043c4bde"
DEVICE="ens33"
ONBOOT="yes"
IPADDR=192.168.33.132
NETMASK=255.255.255.0
GATEWAY=192.168.33.1
节点二
[root@slave ~]#  vim /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="648c35d9-c72c-4a9a-b374-cccd228af3f1"
DEVICE="ens33"
ONBOOT="yes"
IPADDR=192.168.33.131
NETMASK=255.255.255.0
GATEWAY=192.168.33.1
节点三
[root@slave01 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="32f15441-4746-43ed-88ce-a320d7cf81b2"
DEVICE="ens33"
ONBOOT="yes"
IPADDR=192.168.33.130
NETMASK=255.255.255.0
GATEWAY=192.168.33.1

就叫阿坤吧

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
hadoop 集群搭建

1、hadoop概述hadoop是开源的，分布式存储+分布式计算平台。hadoop可以用来搭建大型数据仓库，PB级数据的存储、处理、分析、统计等业务，具有高扩展、低成本、成熟的生态圈的特点，本文档介绍搭建hadoop的步骤，以及搭建中的注意事项。2、文档规划搭建hadoop集群需要两台虚拟机，所以有节点一，节点二，后面再动态添加一个节点。节点一节点二节点三操作系统...
复制链接

扫一扫