Hadoop配置

解压Hadoop安装包

[whzy@master software]$ cp hadoop-2.7.3.tar.gz ~/

[whzy@master ~]$ tar -xvf hadoop-2.7.3.tar.gz

检查是否是否成功

配置环境

1. 配置环境变量hadoop-env.sh

whzy@master hadoop]$ vi hadoop-env.sh
将export JAVA_HOME=${JAVA_HOME}改为下图所示，后面的路径为JDK所在的位置

2. 配置环境变量yarn-env.sh

将上面的内容改为下图所示

3. 配置核心组件core-site.xml

[whzy@master hadoop]$ vi core-site.xml

将以下内容添加进去：

<configuration>
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://master:9000</value>
</property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/home/whzy/hadooptmpdata</value>
</property>
</configuration>

4.配置文件系统hdfs-site.xml

[whzy@master hadoop]$ vi hdfs-site.xml

<configuration>
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
<property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/home/whzy/hadooptmpdata/dfs/name</value>
</property>
<property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/home/whzy/hadooptmpdata/dfs/data</value>
</property>
<property>
    <name>dfs.block.size</name>
    <value>134217728</value>
</property>
<property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>master:9001</value>
</property>
<property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>master:9001</value>
</property>
<property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
</property>
<property>
    <name>dfs.permissions.enabled</name>
    <value>true</value>
</property>
<property>
    <name>dfs.namenode.acls.enabled</name>
    <value>true</value>
</property>
</configuration>

5.配置文件系统yarn-site.xml

[whzy@master hadoop]$ vi yarn-site.xml

<configuration>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.resourcemanager.address</name>
    <value>master:18040</value>
</property>
<property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>master:18030</value>
</property>
<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>master:18025</value>
</property>
<property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>master:18141</value>
</property>
<property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>master:18088</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

6.配置计算框架mapred-site.xml
由于本身没有mapred-site.xml文件，只有一个 mapred-site.xml.template 文件，需要利用 mapred-site.xml.template 文件复制一个mapred-site.xml文件。

[whzy@master hadoop]$ cp mapred-site.xml.template ~/hadoop-2.7.3/etc/hadoop/mapred-site.xml
[whzy@master hadoop]$ vi mapred-site.xml

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
</configuration>

7.在master节点配置slave文件

[whzy@master hadoop]$ vi slaves
将slave替换文件中的内容

在master节点上配置hadoop系统环境变量

[whzy@master ~]$ vi ~/.bash_profile

export HADOOP_HOME=/home/whzy/hadoop-2.7.3
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

用source使改变生效，加载环境变量
[whzy@master ~]$ source ~/.bash_profile