（三）Hadoop集群模式搭建

（三）集群模式搭建

传送门：

（一）Hadoop基础搭建（Ubuntu）
（二）Hadoop伪分布式与初步集群模式
（三）Hadoop集群模式搭建
（四）Hive搭建

配置免密ssh

免密登录原理
每台主机authorized_keys文件里面包含的主机（ssh密钥），该主机都能无密码登录，所以只要每台主机的authorized_keys文件里面都放入其他主机（需要无密码登录的主机）的ssh密钥就行了。
1. 每台linux主机上均打开22端口

sudo vim /etc/ssh/ssh_config
port 22

**2. 在每个节点生成ssh密钥 **

[hadoop@master ~/.ssh]# ssh-keygen -t rsa

3. 在主节点上将公钥拷到一个特定文件authorized_keys中

[hadoop@master ~/.ssh]# cp id_rsa.pub authorized_keys

4. 将authorized_keys文件拷到下一个节点，并将该节点的ssh密钥id_rsa.pub加入该文件中
在master上使用scp命令实现远程文件拷贝

[hadoop@master ~/.ssh]# scp /home/hadoop/.ssh/authorized_keys  hadoop@slave1:/home/hadoop/.ssh/authorized_keys
[hadoop@slave1~]# cd ~/.ssh/
 #使用cat追加方式
[hadoop@slave1 .ssh]# cat id_rsa.pub >> authorized_keys

5. 重复第3步的操作，将slave2节点的ssh密钥加入到authorized_keys文件中

[hadoop@slave1 ~/.ssh]# scp /home/murphy/.ssh/authorized_keys  murphy@slave2:/home/murphy/.ssh/authorized_keys
[hadoop@slave2 ~]# cd ~/.ssh/
 #使用cat追加方式
[hadoop@slave2 .ssh]# cat id_rsa.pub >> authorized_keys 

6. 将slav2节点生成的authorized_keys文件拷贝到其他两个个节点（master、slave1、slave2）

[hadoop@slave2 .ssh]# scp /home/murphy/.ssh/authorized_keys murphy@master:/home/murphy/.ssh/authorized_keys
[hadoop@slave2 .ssh]# scp /home/murphy/.ssh/authorized_keys murphy@slave1:/home/murphy/.ssh/authorized_keys

7. 验证各节点间免密ssh正确

# slave2节点ssh免密访问其他节点:
hadoop@slave2:~$ ssh master
hadoop@master:~$ exit
hadoop@slave2:~$ ssh slave1
hadoop@ slave1:~$ exit

# slave1节点ssh免密访问其他节点
hadoop@slave1:/home$ ssh master
hadoop@master:~$ exit
hadoop@slave1:/home$ ssh slave2
hadoop@slave2:~$ exit

# master节点ssh免密访问其他节点：
hadoop@master:~/.ssh$ ssh slave1
hadoop@slave1:~$ exit
hadoop@master:~/.ssh$ ssh slave2
hadoop@slave2:~$ exit

集群环境配置

1. 配置master的xml文件

hadoop@master:~$ vim /usr/local/hadoop-2.7.7/etc/hadoop/slaves

slave1
slave2

hadoop@master:~$ vim /usr/local/hadoop-2.7.7/etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
    </property>
    <property>
         <!--hadoop用户创建tmp目录赋予读写执行权限-->
        <name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-2.7.7/data/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>
</configuration>

hadoop@master:~$ vim /usr/local/hadoop-2.7.7/etc/hadoop/hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
	<!-- 指定Hadoop辅助名称节点主机配置 -->
	<property>
		<name>dfs.namenode.secondary.http-address</name>
		<value>master:50090</value>
	</property>
    <property>
        <name>dfs.namenode.name.dir</name>
	<value>/usr/local/hadoop-2.7.7/data/tmp/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/usr/local/hadoop-2.7.7/data/tmp/data</value>
    </property>
</configuration>

hadoop@master:~$ vim /usr/local/hadoop-2.7.7/etc/hadoop/mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>master:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>master:19888</value>
    </property>
</configuration>

hadoop@master:~$ vim /usr/local/hadoop-2.7.7/etc/hadoop/yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux_services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <!-- 指定YARN的ResourceManager的地址 -->
	<property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
    </property>
        <!--property>
                <name>yarn.resourcemanager.address</name>
                <value>0.0.0.0:8032</value>
         </property>
         <property>
                <name>yarn.resourcemanager.scheduler.address</name>
                <value>0.0.0.0:8030</value>
         </property>
         <property>
                <name>yarn.resourcemanager.resource-tracker.address</name>
                <value>0.0.0.0:8031</value>
         </property>
         <property>
                <name>yarn.resourcemanager.admin.address</name>
                <value>0.0.0.0:8033</value>
         </property>
         <property>
                <name>yarn.resourcemanager.webapp.address</name>
                <value>0.0.0.0:8088</value>
         </property-->
<!-- Site specific YARN configuration properties -->
</configuration>

2. 检查yarn-env.sh mapred-env.sh hadoop-env.sh中有JAVA_HOME配置

hadoop@master:/usr/local/hadoop-2.7.7/etc/hadoop$ sudo vim yarn-env.sh
hadoop@master:/usr/local/hadoop-2.7.7/etc/hadoop$ sudo vim mapred-env.sh
hadoop@master:/usr/local/hadoop-2.7.7/etc/hadoop$ sudo vim hadoop-env.sh

检查看以上文件是否有JAVA_HOME

export JAVA_HOME=/usr/local/jdk1.8.0_121

启动集群

1. 格式化集群

# 删除日志文件
hadoop@slave2:/usr/local/hadoop-2.7.7/data/tmp$ rm -rf dfs
hadoop@slave2:/usr/local/hadoop-2.7.7/data/tmp$ rm -rf nm-local-dir/
hadoop@master:/usr/local/hadoop-2.7.7/logs$ rm -rf *
# 格式化集群
hadoop@master:/usr/local/hadoop-2.7.7/logs$ hdfs namenode –format

2. 开启集群并验证

hadoop@master:/usr/local/hadoop-2.7.7/logs$ start-dfs.sh

hadoop@master:/usr/local/hadoop-2.7.7/logs$ jps

hadoop@master:/usr/local/hadoop-2.7.7/logs$ ssh slave1
hadoop@slave1:~$ jps

hadoop@master:/usr/local/hadoop-2.7.7/logs$ ssh slave2
hadoop@slave2:~$ jps

hadoop@slave1:~$ exit

hadoop@master:/usr/local/hadoop-2.7.7/logs$ start-yarn.sh

hadoop@master:/usr/local/hadoop-2.7.7/logs$ jps

hadoop@master:/usr/local/hadoop-2.7.7/logs$ ssh slave1
hadoop@slave1:~$ jps

hadoop@slave1:~$ exit
hadoop@master:/usr/local/hadoop-2.7.7/logs$ ssh slave2
hadoop@slave2:~$ jps

hadoop@slave2:~$ exit

配套视频