Hadoop2.8集群搭建

前期准备：
环境搭建完成：bs1、bs2、bs3
SSH免密登录配置完成
JDK已安装

1、上传Hadoop压缩包并解压

hadoop下载地址：https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz

[root@bs1 zipfile]#tar -zxvf hadoop-2.8.5.tar.gz -C /root/soft/app/

2、进入/root/soft/app/hadoop-2.8.5/etc/hadoop目录，修改配置文件

2.1、配置hadoop-env.sh

添加实际Java路径

2.2、配置core-site.xml

<configuration>
   <!--HDFS的地址-->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://bs1:9000</value>
    </property>
    <!--Hadoop临时存储目录-->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/root/soft/data/hadoop/tmp</value>
    </property>
</configuration>

2.3、配置hdfs-site.xml

<configuration>
    <!--hdfs副本数量-->
    <name>dfs.replication</name>
    <value>3</value>
</configuration>

其他设置

 <property>
      <name>dfs.namenode.name.dir</name>
      <value>/root/soft/data/hadoop/data/namenode</value>
</property>
<property>
      <name>dfs.datanode.data.dir</name>
      <value>/root/soft/data/hadoop/data/datanode</value>
</property>

2.4、配置mapred-site.xml

cp mapred-site.xml.template mapred-site.xml

<configuration>
    <!--MR是有YARN-->
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

2.5、配置yarn-site.xml

<configuration>
<!-- Site specific YARN configuration properties -->
	<property>
		<name>yarn.resourcemanager.hostname</name>
		<value>bs1</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-service</name>
        <value>mapreduce_shuffle</value>
    </property>
    <!--以下可以默认不配置。如果子节点向主节点注册出错，则显示配置-->
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>bs1:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>bs1:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>bs1:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>bs1:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>bs1:8088</value>
    </property>
</configuration>     

2.6、配置slaves

bs2
bs3

2.7、将主节点上配置好的hadoop文件分发到从节点上

[root@bs1 app]# scp -r hadoop-2.8.5 bs2:/root/soft/app
[root@bs1 app]# scp -r hadoop-2.8.5 bs3:/root/soft/app

3、启动Hadoop

3.1、格式化Hadoop

[root@bs1 hadoop-2.8.5]# bin/hdfs namenode -format 

3.2、启动HDFS和YAR

[root@bs1 hadoop-2.8.5]# sbin/start-dfs.sh
[root@bs1 hadoop-2.8.5]# sbin/start-yarn.sh

3.3、验证是否成功

主节点

[root@bs1 ~]# hdfs dfsadmin -report

[root@bs1 ~]# yarn node -list -all

4、配置环境变量