Hadoop伪分布式集群的环境搭建

最新推荐文章于 2024-09-08 19:02:41 发布

heady230

最新推荐文章于 2024-09-08 19:02:41 发布

阅读量308

点赞数

文章标签： hadoop 大数据

本文链接：https://blog.csdn.net/weixin_44228830/article/details/107365763

版权

本文详细介绍了如何在Linux环境下搭建Hadoop 2.6.0的伪分布式集群，包括关闭防火墙和禁用SELINUX、配置hostname与IP对应、创建Hadoop用户、配置SSH免密登录、安装JDK、下载并配置Hadoop、以及启动和检查集群进程。

摘要由CSDN通过智能技术生成

1.关闭防火墙和禁用SELINUX

防火墙会妨碍Hadoop集群间的相互通信，所以需要关闭防火墙，禁用SELINUX（Security Enhanced Linux)，SELINUX配置设置太严格，可能会与其他服务的功能相冲突，所以直接关闭

在控制台中输入chkconfig iptables off命令

[root@csdn ~]# chkconfig iptables off

上述命令执行完成后，在控制台中输入reboot 命令重启Linux系统，即可永久关闭防火墙

[root@csdn ~]# reboot

在控制台输入vi /etc/selinux/config命令，然后修改SELINUX的相关配置，将SELINUX的参数值修改为disabled(SELINUX=disabled)

[root@csdn ~]# vi /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

输入reboot重启系统让配置生效

[root@csdn ~]# reboot

在控制台输入service iptables status命令查看防火墙状态，如果防火墙未运行，说明防火墙关闭成功

[root@csdn ~]# service iptables status
iptables: Firewall is not running.
[root@csdn ~]#

2.配置hostname与IP地址之间的对应关系

控制台输入vi /etc/hosts进行相应配置

[hadoop@csdn ~]$ vi /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.5.133 csdn
~

3.创建用户和用户组

配置Hadoop需要创建一个Hadoop用户，而不是使用root用户
在控制台使用useradd命令创建一个Hadoop用户

[root@csdn ~]# useradd hadoop
[root@csdn ~]# cd /home/
[root@csdn home]# ls
hadoop

创建完Hadoop用户后，授予Hadoop用户root权限

[root@csdn ~]# visudo

在最后一行添加Hadoop用户

## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
#includedir /etc/sudoers.d
hadoop ALL=(ALL) NOPASSWD:ALL

切换到Hadoop用户，查看是否有root权限

[root@csdn ~]# su hadoop
[hadoop@csdn root]$ cd
[hadoop@csdn ~]$ sudo visudo

4.配置SSH免密登录

在控制台使用命令mkdir创建.ssh目录，使用命令ssh-keygen -t rsa生成密钥对

[hadoop@csdn ~]$ mkdir .ssh
[hadoop@csdn ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
c9:ca:d5:72:43:58:8d:6a:8a:a2:ad:78:48:61:da:4a syc@syc
The key's randomart image is:
+--[ RSA 2048]----+
|          .o     |
|         o. .    |
|        ...      |
| o     .o+       |
|o..  . oS +      |
|.Eo ...o o .     |
|o= .  o          |
|= o              |
|oo               |
+-----------------+

将公钥文件id_rsa.pub中的内容复制到相同目录下的authorized_keys文件中

[hadoop@csdn ~]$ cd .ssh/
[hadoop@csdn .ssh]$ ls
id_rsa  id_rsa.pub
[hadoop@csdn .ssh]$ cat id_rsa.pub >> authorized_keys
[hadoop@csdn .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAy3vBbjh5NfGaSX6KyGI5e5VBZxiyrH3tFp94kFc9DYBOXLwdV8PJKzpGyEs1ASWtTlSFVHEP/iBlnPI5bjN2MmSYOibfDLFsiTz0NlajIzAeQvm41Ekr8ya5HxTppZ7pnYBBbLesTvsiHF9VtV11QMtWJ4hiN5cc5PUD9Fc5/GMIQk4AWTBtGbdwfisqYSn0s3h3HP+/2DmIRcR6GTSAbcn2K8XumYBsigUT57NSJqK0VYRqvoTT1QasrxpkqSS2R9PkV54FgMxakHIFVmjQKrSR1CAAJRZ+Vy1/2b1iJvdOXTNEm5oarV/7NabHxbjHXxE7H3Laz0OsfWdQDHsADQ==
0