Hadoop配置安装

三台机器，配置了免密码登录

172.16.20.57 jintui-test-57
172.16.20.58 jintui-test-58
172.16.20.59 jintui-test-59

1、第一步下载java、hadoop包,解压

java version "1.8.0_73"
hadoop version  2.7.3

2、配置hadoop的环境变量,配置hadoop-env.sh

export HADOOP_IDENT_STRING=$USER
export HADOOP_SSH_OPTS="-p 62222"
export JAVA_HOME=/data/servers/jdk
export HOSTNAME=jintui-test-57

3、配置core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://jintui-test-57:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/data/hadoop/tmp</value>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131702</value>
    </property>
</configuration>

4、配置hdfs-site.xml

<configuration>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/data/hadoop/hdfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/data/hadoop/hdfs/data</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>jintui-test-57:9001</value>
    </property>
    <property>
	<name>dfs.webhdfs.enabled</name>
	<value>true</value>
    </property>
</configuration>

5、yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>jintui-test-57:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>jintui-test-57:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>jintui-test-57:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>jintui-test-57:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>jintui-test-57:8088</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>1024</value>
    </property>
</configuration>

6、配置mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>jintui-test-57:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>jintui-test-57:19888</value>
    </property>
</configuration>

7、配置完成后，执行下面命令，完成初始化。

bin/hdfs namenode -format

8、启动

sbin/start-dfs.sh

sbin/start-yarn.sh

三台机器进程如下

172.16.20.57 jintui-test-57	172.16.20.58 jintui-test-58	172.16.20.59 jintui-test-59
6877 Jps 4880 NameNode 5109 SecondaryNameNode	26733 Jps 3886 DataNode	28863 Jps 2369 DataNode
6830 ResourceManager	5333 NodeManager	3724 NodeManager

9、监控页面

Daemon	Web Interface	Notes
NameNode	http://jintui-test-57:port/	Default HTTP port is 50070
ResourceManager	http://jintui-test-57:port/	Default HTTP port is 8088
MapReduce JobHistory Server	http://jintui-test-57:port/	Default HTTP port is 19888