HFDS伪分布式集群搭建

HFDS伪分布式，顾名思义就是不是真正的分布式，它是搭建在一台其上的HDFS，以下是搭建HFDS伪分布式的操作和一些相关问题

设置免密登录

先生成公钥和密钥

[root@localhost .ssh]# ssh-keygen -t rsa

将公钥给自己

[root@localhost .ssh]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@node01

配置java环境

解压jdk

[root@localhost software]# tar -zxf jdk-8u181-linux-x64.tar.gz

在/etc目录下的profile配置jdk

[root@localhost jdk1.8.0_181]# vi /etc/profile

在文档最后加上一下字段即可

export JAVA_HOME=/opt/software/jdk1.8.0_181
export PATH=$JAVA_HOME/bin:$PATH

编辑完成之后，source运行profile，配置成功输入java -version可查看当前java版本

[root@localhost jdk1.8.0_181]# source /etc/profile
[root@localhost jdk1.8.0_181]# java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

另需在/root目录下修改.bashrc文件，在最后加上上面profile修改的字段，source运行

[root@localhost jdk1.8.0_181]# vi ~/.bashrc
[root@localhost jdk1.8.0_181]# source ~/.bashrc

安装Hadoop

下载Hadoop安装包，解压 ，并在~/.bashrc文件和/etc/profile文件把HADOOP_HOME环境配置好

[root@node1 software]# tar -zxf hadoop-2.6.5.tar.gz

配置Hadoop

1. 配置hdfs-site.xml

[root@node1 software]# vi hadoop-2.6.5/etc/hadoop/hdfs-site.xml

<configuration> 
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>

    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>node01:50090</value>
    </property>
</configuration>

2. 配置core-site.xml文件

[root@node1 software]# vi hadoop-2.6.5/etc/hadoop/core-site.xml 

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://node01:9000</value>
    </property>
    
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/var/abc/hadoop/local</value>
    </property>
</configuration>

3. 配置slaves文件

[root@node1 software]# vi hadoop-2.6.5/etc/hadoop/slaves

4. 格式化

[root@node1 bin]# hdfs namenode -format

/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node01/192.168.253.128
************************************************************/

出现以上字段说明配置成功，如果抛出java.net.UnknownHostException异常则是hostname主机名配置与Hadoop配置中的不一致，需修改Hadoop的配置文件或者hostname

/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: node1: node1: Name or service not known
************************************************************/

启动HDFS

[root@node1 bin]# start-dfs.sh
Starting namenodes on [node01]
node01: starting namenode, logging to /opt/software/hadoop-2.6.5/logs/hadoop-root-namenode-node01.out
node01: starting datanode, logging to /opt/software/hadoop-2.6.5/logs/hadoop-root-datanode-node01.out
Starting secondary namenodes [node01]
node01: starting secondarynamenode, logging to /opt/software/hadoop-2.6.5/logs/hadoop-root-secondarynamenode-node01.out

使用jps命令查看是否启动成功

[root@node1 bin]# jps
43576 SecondaryNameNode
43353 NameNode
43434 DataNode
43711 Jps

相关问题可以参考搭建HDFS集群应该注意虚拟机的相关配置