hadoop伪分布式环境搭建

新建目录

cd /usr/local/bin/

mkdir cloud

cd cloud

下载

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.1.1/hadoop-3.1.1.tar.gz

解压

tar -zxvf hadoop-3.1.1.tar.gz

mv hadoop-3.1.1 hadoop

cd hadoop

环境变量

export HADOOP_HOME="/usr/local/bin/cloud/hadoop"

source /etc/profile

vi etc/hadoop/hadoop-env.sh

# JAVA_HOME
export JAVA_HOME=/home/root/jdk/jdk1.8.0_171
 
# USERS
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
 
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

测试配置

bin/hadoop version

伪分布式配置

vi etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

vi etc/hadoop/hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

bin/hdfs namenode -format

下面这一步要配置好ssh localhost 免密码登录

sbin/start-dfs.sh

查看后台

jps

2512 Jps
2145 DataNode
2028 NameNode
2334 SecondaryNameNode

能打开网站则配置成功

http:/localhost:9870/

 

ssh-server安装配置

sudo apt-get install openssh-server

gedit /etc/ssh/sshd_config

PermitRootLogin的参数设置为yes

service ssh restart

 

本地免密码登录设置

ssh-keygen -t rsa

ssh-copy-id localhost