本文主要是HDFS伪分布式环境搭建
的补充,可以直接查看原文,如果有无法解决的问题,可以回来本文看是否符合自己的情况
下载Hadoop 2.6.0-cdh5.7.0的tar.gz包并解压:
[root@localhost ~]# cd /usr/local/src/
[root@localhost /usr/local/src]# wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
[root@localhost /usr/local/src]# tar -zxvf hadoop-2.6.0-cdh5.7.0.tar.gz -C /usr/local/
把JAVA_HOME配置一下:
[root@localhost /usr/local/hadoop-2.6.0-cdh5.7.0]# cd etc/
[root@localhost /usr/local/hadoop-2.6.0-cdh5.7.0/etc]# cd hadoop
[root@localhost /usr/local/hadoop-2.6.0-cdh5.7.0/etc/hadoop]# vim hadoop-env.sh
export JAVA_HOME=/usr/local/jdk1.8/ # 根据你的环境变量进行修改,如果你是CentOS,直接看下边
注意:比如笔者用的是CentOS,如果安装的时候有选择,是自带Java的,那么这个的配置可以参考
第二个问题是,HDFS支持比较好的java版本实际上是1.7,如果我们使用的1.8版本,你的JAVA_HOME的配置应该多一个
/jre
即刚才的命令应该改为
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre
由于我们要进行的是单节点伪分布式环境的搭建,所以还需要配置两个配置文件,分别是core-site.xml以及hdfs-site.xml,如下:
[root@localhost /usr/local/hadoop-2.6.0-cdh5.7.0/etc/hadoop]# vim core-site.xml # 增加如下内容
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.168.77.130:8020</value> # 指定默认的访问地址以及端口号
</property>
<property>
<name>hadoop.tmp.dir</name> # 指定临时文件所存放的目录
<value>/data/tmp/</value>
</property>
</configuration>
[root@localhost /usr/local/hadoop-2.6.0-cdh5.7.0/etc/hadoop]# mkdir /data/tmp/
[root@localhost /usr/local/hadoop-2.6.0-cdh5.7.0/etc/hadoop]# vim hdfs-site.xml # 增加如下内容
<configuration>
<property>
<name>dfs.replication</name> # 指定只产生一个副本
<value>1</value>
</property>
</configuration>
然后配置一下密钥对,设置本地免密登录,搭建伪分布式的话这一步是必须的:
[root@localhost ~]# ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
Generating public/private dsa key pair.
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
c2:41:89:65:bd:04:9e:3e:3f:f9:a7:51:cd:e9:cf:1e root@localhost
The key's randomart image is:
+--[ DSA 1024]----+
| o=+ |
| .+..o |
| +. . |
| o .. o . |
| = S . + |
| + . . . |
| + . .E |
| o .. o.|
| oo .+|
+-----------------+
[root@localhost ~]# cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[root@localhost ~]# ssh localhost
ssh_exchange_identification: read: Connection reset by peer
[root@localhost ~]#
接下来就可以启动HDFS了,不过在启动之前需要先格式化文件系统:
[root@localhost ~]# /usr/local/hadoop-2.6.0-cdh5.7.0/bin/hdfs namenode -format
注:只有第一次启动才需要格式化
然后如果没有jps命令,可以用yum安装jps
centos 没有 jps
使用服务启动脚本启动服务:
[root@localhost ~]# /usr/local/hadoop-2.6.0-cdh5.7.0/sbin/start-dfs.sh
18/03/24 21:59:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [192.168.77.130]
192.168.77.130: namenode running as process 8928. Stop it first.
localhost: starting datanode, logging to /usr/local/hadoop-2.6.0-cdh5.7.0/logs/hadoop-root-datanode-localhost.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is 63:74:14:e8:15:4c:45:13:9e:16:56:92:6a:8c:1a:84.
Are you sure you want to continue connecting (yes/no)? yes # 第一次启动会询问是否连接节点
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.6.0-cdh5.7.0/logs/hadoop-root-secondarynamenode-localhost.out
18/03/24 21:59:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@localhost ~]# jps # 检查是否有以下几个进程,如果少了一个都是不成功的
8928 NameNode
9875 Jps
9578 DataNode
9757 SecondaryNameNode
[root@localhost ~]# netstat -lntp |grep java # 检查端口
tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 9757/java
tcp 0 0 192.168.77.130:8020 0.0.0.0:* LISTEN 8928/java
tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 8928/java
tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 9578/java
tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 9578/java
tcp 0 0 0.0.0.0:50020 0.0.0.0:* LISTEN 9578/java
tcp 0 0 127.0.0.1:53703 0.0.0.0:* LISTEN 9578/java
[root@localhost ~]#
如果上一步有问题,可以关闭防火墙
centos7怎么永久关闭防火墙
确认服务启动成功后,使用浏览器访问
如果live node结点为0,尝试关闭防火墙后重启
192.168.77.130:50070