Hadoop集群搭建

vi /etc/hosts

三台机器的内容统一增加以下host配置：（此处IP可以自行设置）

192.168.100.10 node1

192.168.100.20 node2

192.168.100.30 node3

配置免密ssh

ssh-keygen(一直回车)

ssh-copy-id (yes,输入虚拟机密码)

逐层创建文件夹

mkdir -p /export/software

mkdir -p /export/servers

mkdir -p /export/data

mkdir -p /export/logs

上传jdk和hadoop的包到/export/software目录

解压到/export/servers目录

关闭防火墙（systemctl stop firewalld;systemctl disable firewalld）

和selinux(setenforce 0{这是临时关闭}，永久关闭执行sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' 并重启虚拟机，可以用 getenforce 命令验证)

用vi /etc/profile 进入配置jdk和hadoop环境变量 ；

#set java environment

export  JAVA_HOME=/export/servers/jdk

export  PATH=$JAVA_HOME/bin:$PATH

export  CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JRE_HOME/lib/tools.jar

#set hadoop environment

export   HADOOP_HOME=/export/servers/hadoop

export  PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

为让hadoop环境变量生效，执行如下命令：source /etc/profile

再使用hadoop version命令，发现可以有hadoop的提示，则表示配置生效了。

3、Hadoop的配置

1）首先配置master节点

进入master的/etc/hadoop，编辑文件hadoop-env.sh，修改全局变量(Hadoop运行时需要的JDK环境变量)

export JAVA_HOME=/export/servers/jdk

export HDFS_NAMENODE_USER=root

export HDFS_DATANODE_USER=root

export HDFS_SECONDARYNAMENODE_USER=root

export YARN_RESOURCEMANAGER_USER=root

export YARN_NODEMANAGER_USER=root

依次修改core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml

a.修改core-site.xml

vi core-site.xml

<configuration>

<property>

                <name>fs.defaultFS</name>

                <value>hdfs://node1:8020</value>

        </property>

        <property>

                <name>hadoop.tmp.dir</name>

                <value>/export/servers/hadoop/</value>

        </property>

        <property>

                <name>hadoop.http.staticuser.user</name>

                <value>/root</value>

        </property>

        <property>

                <name>hadoop.proxyuser.root.hosts</name>

                <value>*</value>

        </property>

        <property>

                <name>hadoop.proxyuser.root.groups</name>

                <value>*</value>

        </property>

</configuration> 

b.修改hdfs-site.xml

vi hdfs-site.xml

<configuration>

        <property>

              <name>dfs.namenode.secondary.http-address</name>

                <value>node2:9868</value>

        </property>

</configuration>

c.修改mapred-site.xml

复制template生成xml，命令如下：

cp  mapred-site.xml.template   mapred-site.xml

vi mapred-site.xml

<configuration>

<property>

                <name>mapreduce.framework.name</name>

                <value>yarn</value>

        </property>

        <property>

                <name>mapreduce.jobhistory.address</name>

                <value>node1:10020</value>

        </property>

        <property>

                <name>mapreduce.jobhistory.webapp.address</name>

                <value>node1:19888</value>

        </property>

        <property>

                <name>yarn.app.mapreduce.am.env</name>

                <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>

        </property>

        <property>

                <name>mapreduce.map.env</name>

                <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>

        </property>

        <property>

                <name>mapreduce.reduce.env</name>

                <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>

        </property>

</configuration>

d.修改 yarn-site.xml

vi yarn-site.xml

<configuration>

 <property>

                <name>yarn.resourcemanager.hostname</name>

                <value>node1</value>

        </property>

        <property>

                <name>yarn.nodemanager.aux-services</name>

                <value>mapreduce_shuffle</value>

        </property>

        <property>

                <name>yarn.nodemanager.pmem-check-enabled</name>

                <value>false</value>

        </property>

        <property>

                <name>yarn.nodemanager.vmem-check-enabled</name>

                <value>false</value>

        </property>

        <property>

                <name>yarn.log-aggregation-enable</name>

                <value>true</value>

        </property>

        <property>

                <name>yarn.log.server.url</name>

                <value>http://node1:19888/jobhistory/logs</value>

        </property>

        <property>

                <name>yarn.log-aggregation.retain-seconds</name>

                <value>604800</value>

        </property>

</configuration>

e.(hadoop2)修改slaves,(hadoop3)修改w开头的

f.将主节点配置分发到另外两个节点

scp  /etc/profile  node2:/etc/profile

scp  /etc/profile  node3:/etc/profile

scp  -r  /export  node2:/

scp  -r  /export  node3:/

h.格式化文件系统

node1节点：hdfs  namenode   -format

如看到successful formatted 表示成功

g.启动集群并测试

主节点master上，start-dfs.sh

                 start-yarn.sh（或者直接 start-all.sh 启动）

mr-jobhistory-daemon.sh start historyserver

执行jps ，测试集群搭建是否成功

node1:

15570 Jps

15273 ResourceManager

13997 DataNode

14349 NodeManager

15149 NameNode

node2:

13748 NodeManager

13606 DataNode

14598 Jps

13678 SecondaryNameNode

node3:

13526 NodeManager

13449 DataNode

13916 Jps

* 如果存在上面状态，说明Hadoop的集群搭建成功*

三、测试平台性能

1、UI界面测试

（1）在windows中，c:\Windows\System32\drives\etc中，重新写文件hosts，内容为

192.168.100.10 ndoe1

192.168.100.20 node2

192.168.100.30 node3

浏览器进入node1:9870

node1:8088

node1:19888

进行验证