HDFS 环境搭建

                                           HDFS 环境搭建

1.Window 安装64 bit CentOS(需要额外开启Intel 虚拟化技术)

参考:https://blog.csdn.net/yuzongtao/article/details/44700927 

 

2 .安装JDK配置JAVA_HOME

alt+p 把 jdk-8u171-linux-x64.rpm拖过去

yum一个插件 

yum install lrzsz -y

rz #回车
选择上传的文件

sz 下载文件

[root@CentOS ~]# rpm -ivh jdk-8u171-linux-x64.rpm  安装
[root@CentOS ~]# vi /root/.bashrc # 配置环境变量变量

JAVA_HOME=/usr/java/latest
PATH=$PATH:$JAVA_HOME/bin
CLASSPATH=.
export JAVA_HOME
export PATH
export CLASSPATH

[root@CentOS ~]# ls -l /usr/java/查看安装后产生的文件

[root@CentOS ~]# source /root/.bashrc  //重新加载文件
[root@CentOS ~]# jps
1495 Jps

配置主机名和IP映射关系/etc/hosts

[root@CentOS ~]# vi /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.169.139 CentOS

然后ping CentOS看能ping通

在分布式系统中很多服务都是以主机名标示节点,因此配置IP和主机名的映射关系.用户可以查看以下文件

[root@CentOS ~]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=CentOS

关闭防火墙服务

[root@CentOS ~]# service iptables stop #关闭服务
iptables: Setting chains to policy ACCEPT: filter [  OK  ]
iptables: Flushing firewall rules: [  OK  ]
iptables: Unloading modules: [  OK  ]
[root@CentOS ~]# chkconfig iptables off #关闭开机自起
[root@CentOS ~]# chkconfig --list | grep iptables 
iptables        0:off   1:off   2:off   3:off   4:off   5:off   6:off

因为搭建分布式服务之间可能会产生相互的调度,为了保证正常的通信,一般需要关闭防火墙.

 

配置主机SSH免密码认证(密匙)

SSH 为 Secure Shell 的缩写，SSH 为建立在应用层基础上的安全协议，专为远程登录会话和其他网络服务提供安全性的协议。

==基于口令的安全验证==:基于口令用户名/密码

==基于密匙的安全验证==:

需要依靠密匙，也就是你必须为自己创建一对密匙，并把公用密匙放在需要访问的服务器上。如果你要连接到SSH服务器上，客户端软件就会向服务器发出请求，请求用你的密匙进行安全验证。服务器收到请求之后，先在该服务器上你的主目录下寻找你的公用密匙，然后把它和你发送过来的公用密匙进行比较。如果两个密匙一致，服务器就用公用密匙加密“质询”（challenge）并把它发送给客户端软件。客户端软件收到“质询”之后就可以用你的私人密匙解密再把它发送给服务器。

 

[root@CentOS ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
c3:b7:c4:e3:5e:6f:db:69:48:23:1e:f7:81:9b:d1:8e root@CentOS
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|                 |
|       . .       |
|        S =   o  |
|         = = * o |
|          + * X .|
|         . o E.=.|
|          .  .+o.|
+-----------------+
[root@CentOS ~]# ssh-copy-id CentOS
The authenticity of host 'centos (192.168.169.139)' can't be established.
RSA key fingerprint is f0:63:ed:d6:21:3b:b5:47:ad:e2:7f:98:bd:8f:54:94.
Are you sure you want to continue connecting (yes/no)? `yes`
Warning: Permanently added 'centos,192.168.169.139' (RSA) to the list of known hosts.
root@centos's password:
Now try logging into the machine, with "ssh 'CentOS'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

输入 ssh CentOS 看是否免密

exit 退出

rm -rf 文件名  删除文件

3.HADOOP HDFS安装与配置

  rz   hadoop-2.6.0_x64.tar.gz上传

1. 解压并且配置HADOOP_HOME

[root@CentOS ~]# tar -zxf hadoop-2.6.0_x64.tar.gz -C /usr/ #解压到usr文件架

[root@CentOS ~]# vi /root/.bashrc

HADOOP_HOME=/usr/hadoop-2.6.0
JAVA_HOME=/usr/java/latest
PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
CLASSPATH=.
export JAVA_HOME
export PATH
export CLASSPATH
export HADOOP_HOME



[root@CentOS ~]# source /root/.bashrc #重新加载文件
[root@CentOS ~]# echo $HADOOP_HOME #看配置成功
/usr/hadoop-2.6.0

2.配置etc/hadoop/core-site.xml  

参考  http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html

[root@CentOS ~]# cd /usr/hadoop-2.6.0/
[root@CentOS ~]# ls
[root@CentOS ~]# vi etc/hadoop/core-site.xml


<property>
    <name>fs.defaultFS</name>
    <value>hdfs://CentOS:9000</value>
</property>

<property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/hadoop-2.6.0/hadoop-${user.name}</value>
</property>

配置etc/hadoop/hdfs-site.xml

[root@CentOS ~]# vi etc/hadoop/hdfs-site.xml

<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>

配置etc/hadoop/slaves

vi /usr/hadoop-2.6.0/etc/hadoop/slaves

CentOS

 

启动

如果是第一次启动hdfs,需要格式话namenode

[root@CentOS ~]# cd
[root@CentOS ~]# hdfs namenode -format #格式化

...
19/01/02 20:19:37 INFO common.Storage: Storage directory /usr/hadoop-2.6.0/hadoop-root/dfs/name has been successfully formatted.
...

 

yum一个插件

yum install -y tree

[root@CentOS ~]# tree /usr/hadoop-2.6.0/hadoop-root/
/usr/hadoop-2.6.0/hadoop-root/
└── dfs
    └── name
        └── current
            ├── fsimage_0000000000000000000
            ├── fsimage_0000000000000000000.md5
            ├── seen_txid
            └── VERSION

2.启动HDFS服务

[root@CentOS ~]# start-dfs.sh #启动
Starting namenodes on [CentOS]
CentOS: namenode running as process 1846. Stop it first.
CentOS: starting datanode, logging to /usr/hadoop-2.6.0/logs/hadoop-root-datanode-CentOS.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
RSA key fingerprint is f0:63:ed:d6:21:3b:b5:47:ad:e2:7f:98:bd:8f:54:94.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /usr/hadoop-2.6.0/logs/hadoop-root-secondarynamenode-CentOS.out
[root@CentOS ~]# jps #查看
2370 Jps
2133 DataNode
1846 NameNode
2267 SecondaryNameNode
[root@CentOS ~]# stop-dfs.sh #关闭
Stopping namenodes on [CentOS]
CentOS: stopping namenode
CentOS: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode

或者用户可以访问浏览器:http://192.168.169.139:50070