Hadoop全分布式集群配置指南

最新推荐文章于 2024-08-22 16:16:08 发布

冽夫

最新推荐文章于 2024-08-22 16:16:08 发布

阅读量2.4k

点赞数 1

分类专栏： Hadoop

本文链接：https://blog.csdn.net/helloworld0906/article/details/89190752

版权

Hadoop 专栏收录该内容

17 篇文章 1 订阅

订阅专栏

这里以centOS7为例

先配置好一台master，另外两台slave1和slave2克隆master即可

创建hadoop用户

必须在root用户下进行新用户的创建（或删除）：

[root@master hadoop]# useradd test

[root@master hadoop]# passwd test //为新用户修改密码

正式搭建

1、准备好需要的包：hadoop….tar.gz；jdk….tar.gz

2、解压到/home/hadoop/bigdata（~/bigdata）目录下

3、编辑环境变量：vim /etc/profie 和hadoop本地环境参数配置以及hadoop配置文件

Profie:

#JAVA

export JAVA_HOME=/home/hadoop/bigdata/jdk

export PATH=${JAVA_HOME}/bin:$PATH

export JRE_HOME=${JAVA_HOME}/jre

#HADOOP

export HADOOP_HOME=/home/hadoop/bigdata/hadoop

export PATH=${HADOOP_HOME}/bin:$PATH

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native

export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib"

hadoop-env.sh：

sudo vim bigdata/hadoop/etc/hadoop/hadoop-env.sh 在此文件中确保有下面这两句

export JAVA_HOME=${JAVA_HOME}

export JAVA_HOME=/home/hadoop/bigdata/jdk

修改 core-site.xml

<name>hadoop.tmp.dir</name>

<value>file:/home/user/bigdata/hadoop/tmp</value>

<description>Abase for other temporary directories.</description>

</property>

<name>fs.defaultFS</name>

<value>hdfs://localhost:9000</value>

</property>

hdfs-site.xml

<name>dfs.replication</name>

</property>

<name>dfs.namenode.name.dir</name>

<value>file:/home/user/bigdata/hadoop/tmp/dfs/name</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>file:/home/user/bigdata/hadoop/tmp/dfs/data</value>

</property>

<name>dfs.blocksize</name>

</property>

<name>dfs.namenode.handler.count</name>

</property>

yarn-site.xml

<name>yarn.resourcemanager.hostname</name>

<value>master<value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle<value>

</property>

<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler<value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle<value>

</property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler<value>

</property>

<name>yarn.resourcemanager.admin.address</name>

<value>master:8033<value>

</property>

<name>yarn.resourcemanager.address</name>

<value>master:8032<value>

</property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>master:8031<value>

</property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>master:8030<value>

</property>

<name>yarn.nodemanager.resource.memory-mb</name>

</property>

<name>yarn.nodemanager.resource.cpu-vcores</name>

</property>

mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

4、重启环境变量使其生效：source /etc/profie

确认：

[hadoop@master ~]$ java -version

java version "1.8.0_181"

Java(TM) SE Runtime Environment (build 1.8.0_181-b13)

Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

[hadoop@master ~]$ hadoop version

Hadoop 2.7.7

Subversion Unknown -r c1aad84bd27cd79c3d1a7dd58202a8c3ee1ed3ac

Compiled by stevel on 2018-07-18T22:47Z

Compiled with protoc 2.5.0

From source with checksum 792e15d20b12c74bd6f19a1fb886490

This command was run using /home/hadoop/bigdata/hadoop/share/hadoop/common/hadoop-common-2.7.7.jar

5、实现ssh无密码登陆：即master和slave可以随意切换 ssh master / ssh slave这样子

SSH协议在某些系统里不自带，所以在启动SSH协议前，需要进行ssh和rsync两个服务的检查，确认是否已经安装。Rsync是一个远程数据同步工具，可通过LAN/WAN快速同步多台主机间的文件：

[hadoop@master ~]$ rpm -qa | grep openssh

openssh-clients-7.4p1-16.el7.x86_64

openssh-server-7.4p1-16.el7.x86_64

openssh-7.4p1-16.el7.x86_64

[hadoop@master ~]$ rpm -qa | grep rsync

rsync-3.1.2-4.el7.x86_64

如果没有：yum install -y openssh-clients

openssh-server

rsync

ssh安装与配置：

[hadoop@master ~]$ sudo vim /etc/ssh/sshd_config

确保下面三句存在（没有就打上去hhhhhg）

[hadoop@master ~]$ hostname

master

[hadoop@master ~]$ ssh-key

ssh-keygen ssh-keyscan

[hadoop@master ~]$ ssh-keygen -t rsa -P ''

//出现问题时直接点回车，-P ‘’（中间有空格即认为密码是空）

Generating public/private rsa key pair.

Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):

Created directory '/home/hadoop/.ssh'.

Your identification has been saved in /home/hadoop/.ssh/id_rsa.

Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:gQ6c9NzxdQ8Amc36e8yo0UT/d8xj6ZSJEef2+g3pqg8 hadoop@master

The key's randomart image is:

+---[RSA 2048]----+

| . . .*o.o |

| o + o oo.o. o |

| + + o .... ..|

| o ... .+ |

| . S ....o |

| o. +==|

| E .*oBB|

| oo.Bo=|

| o+ooooo|

+----[SHA256]-----+

sshd_config中指定的公钥路径：authorized_keys

这是一对密钥对：

Your identification has been saved in /home/hadoop/.ssh/id_rsa.

Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.

这是错的：[hadoop@master ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/id_rsa

id_rsa id_rsa.pub

authorized_keys需要自己创建，但是这里只要直接输就可以

将生成的密钥写入sshd_config指定公钥文件路径中~/.ssh/authorized_keys

[hadoop@master ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

[hadoop@master ~]$ chmod 600 ~/.ssh/authorized_keys

[hadoop@master ~]$ sudo systemctl restart sshd.service //centOS7的服务命令有些不一样

[hadoop@master ~]$ ssh master

The authenticity of host 'master (fe80::d514:6ab7:d788:8294%ens33)' can't be established.

ECDSA key fingerprint is SHA256:E30LEGLP/hSoJ5Er+gjh5INzhzIf5OCCTmZad5+7yaU.

ECDSA key fingerprint is MD5:13:45:bc:c5:63:bd:87:55:b2:c3:72:f4:85:5c:91:18.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'master,fe80::d514:6ab7:d788:8294%ens33' (ECDSA) to the list of known hosts.

Last login: Fri Mar 29 19:05:41 2019 from 192.168.40.1

[hadoop@master ~]$

6、网络配置

①修改IP地址：

[hadoop@master ~]$ ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

link/ether 00:0c:29:f7:f4:11 brd ff:ff:ff:ff:ff:ff

inet 192.168.40.160/24 brd 192.168.40.255 scope global noprefixroute dynamic ens33

valid_lft 1218sec preferred_lft 1218sec

inet6 fe80::d514:6ab7:d788:8294/64 scope link noprefixroute

valid_lft forever preferred_lft forever

我们用的网卡是ens33；通过下面的步骤看网关

[hadoop@master bigdata]$ sudo cp /etc/sysconfig/network-scripts/ifcfg-ens33 /etc/sysconfig/network-scripts/ifcfg-ens33-beifen

[hadoop@master bigdata]$ sudo vim /etc/sysconfig/network-scripts/ifcfg-ens33

TYPE=Ethernet #配置为互联网网卡

BOOTPROTO=static #配置获取IP地址形式为静态获取

IPADDR=192.168.40.160 #配置网卡地址

NETMASK=255.255.255.0 #配置子网掩码

NETWORK=192.168.40.0 #配置网络地址（虚拟机中叫子网地址）

NAME=enp0s3 #配置网卡名字

DEVICE=ens33 #网卡硬件名字

ONBOOT=yes #开启自动重启

DNS1=114.114.114.114 #配置dns

GATEWAY=192.168.40.2 #配置网关

[hadoop@master bigdata]$ sudo systemctl restart network

[hadoop@master bigdata]$ ping www.baidu.com

PING www.a.shifen.com (111.13.100.92) 56(84) bytes of data.

64 bytes from 111.13.100.92 (111.13.100.92): icmp_seq=1 ttl=128 time=52.4 ms

64 bytes from 111.13.100.92 (111.13.100.92): icmp_seq=2 ttl=128 time=50.9 ms

7、时间同步

①所有机器时区一致的设置方法：

[hadoop@master bigdata]$ date

Sat Mar 30 09:35:54 CST 2019

[hadoop@master bigdata]$ tzselect

Please identify a location so that time zone rules can be set correctly.

Please select a continent or ocean.

1) Africa

2) Americas

3) Antarctica

4) Arctic Ocean

5) Asia

6) Atlantic Ocean

7) Australia

8) Europe

9) Indian Ocean

10) Pacific Ocean

11) none - I want to specify the time zone using the Posix TZ format.

1) Africa

2) Americas

3) Antarctica

4) Arctic Ocean

5) Asia

6) Atlantic Ocean

7) Australia

8) Europe

9) Indian Ocean

10) Pacific Ocean

11) none - I want to specify the time zone using the Posix TZ format.

#? y

Please enter a number in range.

#? 5

Please select a country.

1) Afghanistan 18) Israel 35) Palestine

2) Armenia 19) Japan 36) Philippines

3) Azerbaijan 20) Jordan 37) Qatar

4) Bahrain 21) Kazakhstan 38) Russia

5) Bangladesh 22) Korea (North) 39) Saudi Arabia

6) Bhutan 23) Korea (South) 40) Singapore

7) Brunei 24) Kuwait 41) Sri Lanka

8) Cambodia 25) Kyrgyzstan 42) Syria

9) China 26) Laos 43) Taiwan

10) Cyprus 27) Lebanon 44) Tajikistan

11) East Timor 28) Macau 45) Thailand

12) Georgia 29) Malaysia 46) Turkmenistan

13) Hong Kong 30) Mongolia 47) United Arab Emirates

14) India 31) Myanmar (Burma) 48) Uzbekistan

15) Indonesia 32) Nepal 49) Vietnam

16) Iran 33) Oman 50) Yemen

17) Iraq 34) Pakistan

#? 9

Please select one of the following time zone regions.

1) Beijing Time

2) Xinjiang Time

#? 1

The following information has been given:

China

Beijing Time

Therefore TZ='Asia/Shanghai' will be used.

Local time is now: Sat Mar 30 09:37:10 CST 2019.

Universal Time is now: Sat Mar 30 01:37:10 UTC 2019.

Is the above information OK?

1) Yes

2) No

#? yes

Please enter 1 for Yes, or 2 for No.

#? 1

You can make this change permanent for yourself by appending the line

TZ='Asia/Shanghai'; export TZ

to the file '.profile' in your home directory; then log out and log in again.

Here is that TZ value again, this time on standard output so that you

can use the /usr/bin/tzselect command in shell scripts:

Asia/Shanghai

[hadoop@master bigdata]$ hwclock -w

hwclock: Sorry, only the superuser can change the Hardware Clock.

[hadoop@master bigdata]$ sudo hwclock -w

②设置时间同步的主服务器

[hadoop@master bigdata]$ rpm -qa | grep ntp

fontpackages-filesystem-1.44-8.el7.noarch

ntp-4.2.6p5-28.el7.centos.x86_64

ntpdate-4.2.6p5-28.el7.centos.x86_64

[hadoop@master bigdata]$ sudo vim /etc/ntp.conf //追加

server 127.127.1.0

Fudge 127.127.1.0 stratum 10

[hadoop@master bigdata]$ sudo systemctl stop firewalld

==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===

Authentication is required to manage system services or units.

Authenticating as: root

Password:

==== AUTHENTICATION COMPLETE ===

[hadoop@master bigdata]$ sudo systemctl restart ntpd

==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===

Authentication is required to manage system services or units.

Authenticating as: root

Password:

==== AUTHENTICATION COMPLETE ===

[hadoop@master bigdata]$ watch ntpq -p

③节点机的配置

这里就可以开始克隆啦~

克隆之后的账号密码都与原来的一样

hadoop 123456

可以看到需要我们更改主机名master为node1以及IP地址

克隆出来的机器改：

改主机名：hostnamectl set-hostname node1

改ip：sudo vim /etc/sysconfig/network-scripts/ifcfg-ens33(通过ip addr查看网卡是哪一块)

以及配置文件…

所有机器都要改映射：

sudo vim /etc/hosts

192.168.40.160 master

192.168.40.40 node1

192.168.40.41 node2

改完之后互相ping 主机名和www.baidu.com

所有机器都执行（xshell发送键盘输入的所有会话）

[hadoop@master ~]$ sudo vim /etc/crontab

加最后一句实现定时自动同步时间

所有机器：

[hadoop@master hadoop]$ vim etc/hadoop/slaves

node1

node2

确保所有机器的防火墙都已关闭

[hadoop@master hadoop]$sudo systemctl disable firewalld.service

只在master上格式化和启动

[hadoop@master hadoop]$ hdfs namenode -format

[hadoop@master hadoop]$ sbin/start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

19/03/30 22:00:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on [master]

master: starting namenode, logging to /home/hadoop/bigdata/hadoop/logs/hadoop-hadoop-namenode-master.out

node2: starting datanode, logging to /home/hadoop/bigdata/hadoop/logs/hadoop-hadoop-datanode-node2.out

node1: starting datanode, logging to /home/hadoop/bigdata/hadoop/logs/hadoop-hadoop-datanode-node1.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /home/hadoop/bigdata/hadoop/logs/hadoop-hadoop-secondarynamenode-master.out

19/03/30 22:01:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

starting yarn daemons

starting resourcemanager, logging to /home/hadoop/bigdata/hadoop/logs/yarn-hadoop-resourcemanager-master.out

node2: starting nodemanager, logging to /home/hadoop/bigdata/hadoop/logs/yarn-hadoop-nodemanager-node2.out

node1: starting nodemanager, logging to /home/hadoop/bigdata/hadoop/logs/yarn-hadoop-nodemanager-node1.out

[hadoop@master hadoop]$ jps

7984 SecondaryNameNode

7687 NameNode

8135 ResourceManager

8392 Jps

Start-all.sh之后才会有logs

[hadoop@master hadoop]$ hdfs dfsadmin -report

19/03/30 22:02:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Configured Capacity: 36477861888 (33.97 GB)

Present Capacity: 30252290048 (28.17 GB)

DFS Remaining: 30252281856 (28.17 GB)

DFS Used: 8192 (8 KB)

DFS Used%: 0.00%

Under replicated blocks: 0

Blocks with corrupt replicas: 0

Missing blocks: 0

Missing blocks (with replication factor 1): 0

-------------------------------------------------

Live datanodes (2):

Name: 192.168.40.40:50010 (node1)

Hostname: node1

Decommission Status : Normal

Configured Capacity: 18238930944 (16.99 GB)

DFS Used: 4096 (4 KB)

Non DFS Used: 3112660992 (2.90 GB)

DFS Remaining: 15126265856 (14.09 GB)

DFS Used%: 0.00%

DFS Remaining%: 82.93%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Sat Mar 30 22:02:14 CST 2019

Name: 192.168.40.41:50010 (node2)

Hostname: node2

Decommission Status : Normal

Configured Capacity: 18238930944 (16.99 GB)

DFS Used: 4096 (4 KB)

Non DFS Used: 3112910848 (2.90 GB)

DFS Remaining: 15126016000 (14.09 GB)

DFS Used%: 0.00%

DFS Remaining%: 82.93%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Sat Mar 30 22:02:16 CST 2019

冽夫

关注

1
点赞
踩
3

收藏

觉得还不错? 一键收藏
0
评论
复制链接

分享到 QQ

分享到新浪微博

扫一扫

专栏目录