Hadoop集群搭建

Hadoop集群搭建

准备工作

1. ssh免登陆
2. 防火墙关闭
3. 分别创建hadoop、hdfs、yarn、mapred四个用户，并将hdfs、yarn、mapred三个用户添加到hadoop用户组中

集群规划

三个节点分别为node2、node3、node4；节点角色分配如下，
HDFS
node2：NAMENODE(由hdfs用户启动)
node3：SECONDARY-NAMENODE(由hdfs用户启动)、DATANODE(由hdfs用户启动)
node4：DATANODE(由hdfs用户启动)
YARN
node2：ResourceManager(由yarn用户启动)、JobHistoryServer(由mapred用户启动)
node3：NodeManager(由yarn用户启动)
node4：NodeManager(由yarn用户启动)

集群安装与配置

将hadoop安装包解压到/home/software目录下，并将安装包分发到各个节点中相同的位置，配置core-site.xml、hdfs-site.xml、yarn-site.xml，并将配置后的文件发送到各个节点/home/software/hadoop-2.6.4/etc/hadoop/目录下，每个配置文件的配置信息如下：

core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://node2:8020</value>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>4096</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/software/hadoop-2.6.4/data</value>
    </property>
<configuration>    

hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///home/software/hadoop-2.6.4/data/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///home/software/hadoop-2.6.4/data/dfs/data</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-adress</name>
        <value>node3:50090</value>
    </property>
    <property>
        <name>dfs.namenode.checkpoint.dir</name>
        <value>file:///home/software/hadoop-2.6.4/data/namesecondary</value>
    </property>
    <property>
        <name>dfs.nameservices</name>
        <value>node2</value>
    </property>
</configuration>

yarn-site.xml

<?xml version="1.0"?>
<configuration>

    <property>
        <name>yarn.acl.enable</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

    <!--ResourceManager Config-->
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>node2:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>node2:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>node2:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>node2:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>node2:8088</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
    </property>

    <!-- NodeManager Config -->
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>1900</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>2</value>
    </property>
    <property>
        <name>yarn.nodemanager.log-dirs</name>
        <value>/home/software/hadoop-2.6.4/logs/userlogs</value>
    </property>
</configuration>

mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>node2:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>node2:19888</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.intermediate-done-dir</name>
        <value>/tmp/mr-history/tmp</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.done-dir</name>
        <value>/tmp/mr-history/done</value>
    </property>
    <property>
        <!-- hadoop的日志输出指定目录-->
        <name>mapred.local.dir</name>
        <value>/home/software/hadoop-2.6.4/logs</value>
    </property>
</configuration>

启动hadoop集群

1. 启动HDFS

在node2节点上启动namenode服务

# su hdfs
$ cd /home/software/hadoop-2.6.4/
$ ./bin/hadoop-daemon.sh start namenode

在node3节点上启动secondarynamenode、datanode服务

$ ./bin/hadoop-daemon.sh start secondarynamenode
$ ./bin/hadoop-daemon.sh start datanode

在node4节点上启动datanode服务

$ ./bin/hadoop-daemon.sh start datanode

2. 启动YARN

在node2节点上执行下面命令执行ResourceManager、NodeManager、JobHistoryServer服务

# su yarn
$ ./sbin/start-yarn.sh
$ sudo su mapred
$ ./sbin/mr-jobhistory-daemon.sh start historyserver