Hadoop-02 伪分布集群安装

环境：centos 6.5  Hadoop 2.7.5

1.上传Hadoop 2.7.5的安装包

2.解压Hadoop的安装包

[root@hadoop01 ~]# tar -zxvf hadoop-3.2.0.tar.gz

3.修改Hadoop相关配置文件

# 进入配置文件所在目录 
[root@hadoop01 soft]# cd hadoop-3.2.0/etc/hadoop/ 
# 首先修改 hadoop-env.sh 文件，增加环境变量信息 
[root@hadoop01 hadoop]# vi hadoop-env.sh 
export JAVA_HOME=/home/jdk1.8 #注意自己的安装目录
export HADOOP_LOG_DIR=/data/hadoop_repo/logs/hadoop 

#共需要修改四个配置文件（为避免错误直接复制下列的代码）

# 修改 core-site.xml 文件，注意 fs.defaultFS 属性中的主机名需要和配置的主机名相同
[root@hadoop01 hadoop]# vi core-site.xml 
<configuration> 
    <property> 
        <name>fs.defaultFS</name> 
        <value>hdfs://hadoop01:9000</value> </property> 
    <property> 
    <name>hadoop.tmp.dir</name> 
        <value>/data/hadoop_repo</value> 
    </property> 
</configuration> 

# 修改 hdfs-site.xml 文件，把 hdfs 中文件副本的数量设置为 1，因为现在伪分布集群只有一个节点 
[root@hadoop01 hadoop]# vi hdfs-site.xml 
<configuration> 
    <property> 
        <name>dfs.replication</name> 
        <value>1</value> 
    </property> 
</configuration> 

# 修改 mapred-site.xml，设置 mapreduce 使用的资源调度框架 
[root@hadoop01 hadoop]# vi mapred-site.xml 
<configuration> 
    <property> 
        <name>mapreduce.framework.name</name> 
        <value>yarn</value> 
    </property> 
</configuration> 

# 修改 yarn-site.xml，设置 yarn 上支持运行的服务和环境变量白名单 
[root@hadoop01 hadoop]# vi yarn-site.xml 
<configuration> 
    <property> 
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value> 
    </property> 
    <property> 
        <name>yarn.nodemanager.env-whitelist</name>                         
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CL     
         ASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property> 
</configuration>

4.格式化 namenode

[root@hadoop01 hadoop]# cd /home/hadoop-2.7.5

[root@hadoop01 hadoop-2.7.5]# bin/hdfs namenode -format  #format：格式

 

5.添加HDFS和YARN的用户信息

使用vi 编辑器修改/hadoop-2.7.5/sbin下的四个脚本文件

 

#start-dfs.sh，stop-dfs.sh在文件前面添加如下

HDFS_DATANODE_USER=root

HDFS_DATANODE_SECURE_USER=hdfs

HDFS_NAMENODE_USER=root

HDFS_SECONDARYNAMENODE_USER=root

 

#start-yarn.sh，stop-yarn.sh在文件前面添加如下

YARN_RESOURCEMANAGER_USER=root

HADOOP_SECURE_DN_USER=yarn

YARN_NODEMANAGER_USER=root

6.启动集群

#执行脚本文件
[root@hadoop01 ~]# cd /home/hadoop-2.7.5/sbin
[root@hadoop01 sbin]# ./start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [hadoop01]
hadoop01: starting namenode, logging to /home/hadoop-2.7.5/logs/hadoop-root-namenode-hadoop01.out
192.168.228.100: starting datanode, logging to /home/hadoop-2.7.5/logs/hadoop-root-datanode-hadoop01.out
Starting secondary namenodes [hadoop01]
hadoop01: starting secondarynamenode, logging to /home/hadoop-2.7.5/logs/hadoop-root-secondarynamenode-hadoop01.out
starting yarn daemons
starting resourcemanager, logging to /home/hadoop-2.7.5/logs/yarn-root-resourcemanager-hadoop01.out
192.168.228.100: starting nodemanager, logging to /home/hadoop-2.7.5/logs/yarn-root-nodemanager-hadoop01.out

7.验证集群进程信息，执行jps命令可查看集群进程信息
[root@hadoop01 sbin]# jps
5552 SecondaryNameNode
5265 NameNode
5364 DataNode
5706 ResourceManager
5803 NodeManager
5967 Jps  #除了这项外还需要有5个进程才说明为正常启动

浏览器验证方式：

192.168.228.100:8088
192.168.228.100:9870

出现页面即可

8.停止集群

[root@hadoop01 sbin]# ./stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [hadoop01]
hadoop01: stopping namenode
192.168.228.100: stopping datanode
Stopping secondary namenodes [hadoop01]
hadoop01: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
192.168.228.100: stopping nodemanager
no proxyserver to stop