Centos7.8快速搭建分布式Hadoop2.7.7集群

一、软件版本

• centos 7.8
• Hadoop 2.7.7
• jdk 1.8

二、集群规划

• hdfs架构：NameNode，DataNode，SecondaryNameNode

• yarn架构：ResourceManager，NodeManager（Container，ApplicationMaster）

• 主机规划：

  主机名	ip	节点配置
  hadoop-01	192.168.247.131	NameNode, DataNode, NodeManager
  hadoop-02	192.168.247.132	ResourceManager, DataNode, NodeManager
  hadoop-03	192.168.247.133	SecondaryNameNode, DataNode, NodeManager

三、虚拟机安装配置

1. 最小化安装虚拟机hadoop-01，下面对其进行配置

2. 安装net-tools

   yum -y install net-tools
   

3. 安装vim

   yum -y install vim*
   

   （可选配置）
   vim /etc/vimrc
   #添加如下内容
   set nu          " 设置显示行号
   set showmode    " 设置在命令行界面最下面显示当前模式等
   set ruler       " 在右下角显示光标所在的行数等信息
   set autoindent  " 设置每次单击Enter键后，光标移动到下一行时与上一行的起始字符对齐
   syntax on       " 即设置语法检测，当编辑C或者Shell脚本时，关键字会用特殊颜色显示
   

4. 设置主机名

   vim /etc/hostname
   

5. 设置hosts

   vim /etc/hosts
   #修改如下
   127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
   ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
   192.168.247.131 hadoop-01
   192.168.247.132 hadoop-02
   192.168.247.133 hadoop-03
   

6. 设置静态ip

   vim /etc/sysconfig/network-scripts/ifcfg-ens33
   #修改如下
   BOOTPROTO=static
   ONBOOT=yes
   IPADDR=192.168.247.131
   NETMASK=255.255.255.0
   GATEWAY=192.168.247.2
   DNS1=8.8.8.8
   DNS2=8.8.4.4
   

7. 关闭防火墙

   #1.查看防火墙状态
   systemctl status firewalld
   
   #2.关闭防火墙，start开启
   systemctl stop firewalld
   
   #3.设置开机禁用防火墙，enable启用
   systemctl disable firewalld
   

8. 创建用户和组

   #1.新建工作组
   groupadd hadoop
   
   #2.创建用户并添加组
   方式一：
   #新建用户
   useradd hadoop
   #给用户设置密码
   passwd hadoop
   #给用户增加工作组
   usermod -G hadoop hadoop
   方式二：
   #新建用户同时增加工作组
   useradd -g hadoop hadoop
   
   #3.授权
   vim /etc/sudoers
   hadoop  ALL=(ALL)       ALL
   

9. 安装jdk

   #1.创建软件安装目录
   mkdir -p /opt/module
   
   #2.修改所属用户组
   chown -R hadoop:hadoop /opt/module
   
   #3.解压jdk
   tar -zxvf jdk-8u141-linux-x64.tar.gz -C /opt/module
   
   #4.添加环境变量
   vim /etc/profile
   export JAVA_HOME=/opt/module/jdk1.8.0_141
   export PATH=$PATH:$JAVA_HOME/bin
   
   #5.刷新
   source /etc/profile
   
   #6.查看版本
   java -version
   

四、虚拟机克隆

根据hadoop-01克隆得hadoop-02，hadoop-03

1. 修改主机名（同上）
2. 修改ip地址（同上）

五、ntp时间同步

1. 安装ntp

   yum -y install ntp
   

2. 服务器配置修改

   vim /etc/ntp.conf
   
   # Hosts on local network are less restricted.
   restrict 192.168.247.2 mask 255.255.255.0 nomodify notrap
   
   # Use public servers from the pool.ntp.org project.
   # Please consider joining the pool (http://www.pool.ntp.org/join.html).
   #server 0.centos.pool.ntp.org iburst
   #server 1.centos.pool.ntp.org iburst
   #server 2.centos.pool.ntp.org iburst
   #server 3.centos.pool.ntp.org iburst
   
   server 127.127.1.0
   fudge 127.127.1.0 stratum 10
   

   vim /etc/sysconfig/ntpd
   
   SYNC_HWCLOCK=yes
   

   systemctl status ntpd #查看状态
   systemctl start ntpd #启动
   systemctl enable ntpd #开机自启
   

3. 客户端配置修改

   vim /etc/ntp.conf
   
   # Hosts on local network are less restricted.
   restrict 192.168.247.2 mask 255.255.255.0 nomodify notrap
   
   # Use public servers from the pool.ntp.org project.
   # Please consider joining the pool (http://www.pool.ntp.org/join.html).
   #server 0.centos.pool.ntp.org iburst
   #server 1.centos.pool.ntp.org iburst
   #server 2.centos.pool.ntp.org iburst
   #server 3.centos.pool.ntp.org iburst
   
   server hadoop-01
   

   客户端新建定时任务

   crontab -e
   
   * 8 * * * /usr/sbin/ntpdate hadoop-01
   

六、ssh免密登录

1. 所有节点

   ssh-keygen -t rsa
   
   cd ~/.ssh
   
   ssh-copy-id hadoop-01
   ssh-copy-id hadoop-02
   ssh-copy-id hadoop-03
   

七、主节点hadoop安装配置

1. 安装hadoop

   tar -zxvf hadoop-2.7.7.tar.gz -C /opt/module/
   
   vim /etc/profile
   
   export JAVA_HOME=/opt/module/jdk1.8.0_141
   export HADOOP_HOME=/opt/module/hadoop-2.7.7
   export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
   
   source /etc/profile
   

2. 修改etc/hadoop下的配置文件

   • 分别修改hadoop-env.sh、mapred-env.sh、yarn-env.sh文件中的JAVA_HOME

     export JAVA_HOME=/opt/module/jdk1.8.0_141
     

   • core-site.xml

     <!-- hdfs地址 -->
     <property>
         <name>fs.default.name</name>
         <value>hdfs://hadoop-01:9000</value>
     </property>
     
     <!-- 文件存储目录 -->
     <property>
         <name>hadoop.tmp.dir</name>
         <value>/opt/module/hadoop-2.7.7/data/tmp</value>
     </property>
     

   • hdfs-site.xml

     <!-- 副本数 -->
     <property>
         <name>dfs.replication</name>
         <value>2</value>
     </property>
     
     <!-- NN节点 -->
     <property>
         <name>dfs.namenode.http-address</name>
         <value>hadoop-01:50070</value>
     </property>
     
     <!-- 2NN节点 -->
     <property>
         <name>dfs.namenode.secondary.http-address</name>
         <value>hadoop-03:50090</value>
     </property>
     

   • yarn-site.xml

     <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
     </property>
     
     <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop-02</value>
     </property>
     
     <!-- 日志聚集 -->
     <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
     </property>
         
     <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
     </property>
     

   • mapred-site.xml

     <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
     </property>
     
     <!-- 历史服务器 -->
     <property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoop-01:10020</value>
     </property>
     
     <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoop-01:19888</value>
     </property>
     

八、拷贝主节点hadoop目录到从节点

scp -r hadoop-2.7.7/ hadoop@hadoop-01:/opt/module
scp -r hadoop-2.7.7/ hadoop@hadoop-02:/opt/module
scp -r hadoop-2.7.7/ hadoop@hadoop-03:/opt/module

九、启动集群

1. 格式化并启动NameNode节点

   cd /opt/module/hadoop-2.7.7/bin
   
   ./hdfs namenode -format
   
   cd /opt/module/hadoop-2.7.7/sbin
   
   ./start-dfs.sh
   

2. 启动ResourceManager节点

   cd /opt/module/hadoop-2.7.7/sbin
   
   ./start-yarn.sh
   

3. 查看进程

   jps
   

4. 打开浏览器，输入：http://hadoop-01:50070/，http://hadoop-02:8088/

最后：对你有所帮助的话，记得点赞哦！