hadoop-3.0.0分布式部署与配置

3台机器centos 7

hadoop.master 192.168.0.116

hadoop.slave1 192.168.0.117

hadoop.slave2 192.168.0.118

配置好ssh免密登陆

下载hadoop-3.0.0-alpha4.tar.gz到本机

解压tar xzvf hadoop-3.0.0-alpha4.tar.gz

修改配置文件(目录/home/glsc/hadoop-3.0.0/etc/hadoop/)：

1. core-site.xml 

<configuration>
  <property>  
    <name>fs.defaultFS</name>   
     <value>hdfs://hadoop.master:9000</value>  
  </property> 
  <property>  
      <name>hadoop.tmp.dir</name>  
      <value>file:///home/glsc/hadoop-3.0.0/tmp</value>  
  </property>  
</configuration>

2. hdfs-site.xml

<configuration>
 <property>  
   <name>dfs.replication</name>  
   <value>2</value>  
 </property>  
 <property>  
   <name>dfs.namenode.name.dir</name>  
   <value>file:///home/glsc/hadoop-3.0.0/hdfs/name</value>  
 </property>  
 <property>  
   <name>dfs.datanode.data.dir</name>  
   <value>file:///home/glsc/hadoop-3.0.0/hdfs/data</value>  
 </property>  
 <property>  
   <name>dfs.namenode.secondary.http-address</name>  
   <value>hadoop.slave1:9001</value>  
 </property>  
</configuration>

3.yarn-site.xml

<configuration>
<!-- Site specific YARN configuration properties -->
    <property>  
        <name>yarn.nodemanager.aux-services</name>  
        <value>mapreduce_shuffle</value>  
    </property>  
    <property>  
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>  
        <value>org.apache.hadoop.mapred.ShuffleHandle</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.resource-tracker.address</name>  
        <value>hadoop.master:8025</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.scheduler.address</name>  
        <value>hadoop.master:8030</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.address</name>  
        <value>hadoop.master:8040</value>  
    </property>  
</configuration>

4.mapred-site.xml

<configuration>
    <property>  
        <name>mapreduce.framework.name</name>  
        <value>yarn</value>  
    </property>  
  
<property>  
 <name>mapreduce.application.classpath</name>  
 <value>  
  /home/glsc/hadoop-3.0.0/etc/hadoop,  
  /home/glsc/hadoop-3.0.0/share/hadoop/common/*,  
  /home/glsc/hadoop-3.0.0/share/hadoop/common/lib/*,  
  /home/glsc/hadoop-3.0.0/share/hadoop/hdfs/*,  
  /home/glsc/hadoop-3.0.0/share/hadoop/hdfs/lib/*,  
  /home/glsc/hadoop-3.0.0/share/hadoop/mapreduce/*,  
  /home/glsc/hadoop-3.0.0/share/hadoop/mapreduce/lib/*,  
  /home/glsc/hadoop-3.0.0/share/hadoop/yarn/*,  
  /home/glsc/hadoop-3.0.0/share/hadoop/yarn/lib/*  
 </value>  
</property> 
</configuration>

5. works

hadoop.slave1
hadoop.slave2

6. 添加Java_Home

hadoop-env.sh

hadoop-env.sh

yarn-env.sh

export JAVA_HOME=/home/glsc/web/jdk1.8.0_131

配置完成，将hadoop.master上编辑的hadoop目录，完整scp到hadoop.slave1和hadoop.slave2机器

scp -r hadoop-3.0.0/ glsc@hadoop.slave1:/home/glsc/

scp -r hadoop-3.0.0/ glsc@hadoop.slave2:/home/glsc/

启动hadoop3:

1. 格式化namenode

进入bin目录

./hdfs namenode -format

2. 启动hadoop

进入sbin目录

./start-dfs.sh

./start-yarn.sh

测试查看网页:

NameNode   http://hadoop.master:8088

ResourceManager http://hadoop.master:9870