老王大数据教程(一) centos上安装hadoop集群

1. 准备机器

   主机名 ip 对应的角色 master 192.168.202.5 NameNode 
   node1 192.168.202.6 Datanode1 
   node2 192.168.202.7 Datanode2

2. 修改hosts

   1. ###修改hostname
       192.168.202.5 主机：
       echo "master" > /etc/hostname
   
       192.168.202.6 主机：
       echo "node1" > /etc/hostname
   
       192.168.202.7 主机：
       echo "node2" > /etc/hostname
   
   2. ###master上修改hosts文件
        echo "192.168.202.5 master" >>/etc/hosts
        echo "192.168.202.6 node1" >>/etc/hosts
        echo "192.168.202.7 node2" >>/etc/hosts
   
   3. ###node修改同步到node1，node2上的主机
         scp /etc/hosts node1:/etc/
         scp /etc/hosts node2:/etc/
   
   4. ###各台主机上相互ping，测试是否可以通过host连通
        ping master 
        ping node1 
        ping node2
   
   5. ###master，node1，node2上关闭防火墙
     systemctl stop firewalld 
     systemctl disable firewalld
   

3. 配置免密码登陆

   1. 本机无密钥登录

      master 机器上配置如下 
      ssh-keygen -t rsa
      cd ~/.ssh
      cp id_rsa.pub authorized_keys 
      service sshd restart
      ssh master 
      

   2. master 与node1, node2 无密登录配置

      在node1和node2上面生成ssh 
      ssh-keygen -t rsa
      在maste 机器上
      scp ~/.ssh/id_rsa.pub root@192.168.30.251:~/.ssh
      scp ~/.ssh/id_rsa.pub root@192.168.30.252:~/.ssh 
      

4. hadoop 安装(master)

下载 hadoop-2.7.3.tar.gz 把文件解压到 /usr/local/hadoop 修改环境变量 vi /etc/profile 环境变量(在master,node1,node2上都修改)

    export JAVA_HOME=/usr/java/jdk1.8.0_65  
    export HADOOP_HOME=/usr/local/hadoop   
    export PATH=$PATH:${HADOOP_HOME}/bin  
    关闭防火墙 service iptables stop 

1. 修改hadoop配置

   master 服务器配置

   1. vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh

      export JAVAHOME=/usr/java/jdk1.8.0121

   2. vim /usr/local/hadoop/etc/hadoop/core-site.xml

      <property>  
              <name>hadoop.tmp.dir</name>  
              <value>/hadoop/tmp</value>  
              <description>Abase for other temporary directories.</description>  
      </property>  
      <property>  
              <name>fs.defaultFS</name>  
              <value>hdfs://master:9000</value>  
      </property>  
      <property>  
              <name>io.file.buffer.size</name>  
              <value>4096</value>  
      </property>  
      

   3. vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml

      <property>  
              <name>dfs.replication</name>  
              <value>2</value>  
              <description>nodes total count</description>  
      </property>  
      

   4. cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml

   5. vi /usr/local/hadoop/etc/hadoop/mapred-site.xml

      <property>  
          <name>mapreduce.framework.name</name>  
          <value>yarn</value>  
          <final>true</final>  
      </property>  
      <property>  
          <name>mapreduce.jobtracker.http.address</name>  
          <value>master:50030</value>  
      </property>  
      <property>  
          <name>mapreduce.jobhistory.address</name>  
          <value>master:10020</value>  
      </property>  
      <property>  
          <name>mapreduce.jobhistory.webapp.address</name>  
          <value>master:19888</value>  
      </property>  
      <property>  
           <name>mapred.job.tracker</name>  
           <value>http://master:9001</value>  
      </property>  
      

   6. vi /usr/local/hadoop/etc/hadoop/yarn-site.xml

      <property>  
           <name>yarn.resourcemanager.hostname</name>  
           <value>master</value>  
      </property>  
      <property>  
          <name>yarn.nodemanager.aux-services</name>  
          <value>mapreduce_shuffle</value>  
      </property>  
      <property>  
          <name>yarn.resourcemanager.address</name>  
          <value>master:8032</value>  
      </property>  
      <property>  
          <name>yarn.resourcemanager.scheduler.address</name>  
          <value>master:8030</value>  
      </property>  
      <property>  
          <name>yarn.resourcemanager.resource-tracker.address</name>  
          <value>master:8031</value>  
      </property>  
      <property>  
          <name>yarn.resourcemanager.admin.address</name>  
          <value>master:8033</value>  
      </property>  
      <property>  
          <name>yarn.resourcemanager.webapp.address</name>  
          <value>master:8088</value>  
      </property> 
      

   7. vi /usr/local/hadoop/etc/hadoop/slaves

       node1  
       node2  
      

   8. node1/node2服务器配置和master一样

      scp -r /usr/local/hadoop node1:/usr/local
      scp -r /usr/local/hadoop node1:/usr/local
      

   9. 格式化master

      cd /usr/local/hadoop
      ./hadoop namenode -format
      

   10. 启动hadoop(只在master启动)

       cd /usr/local/hadoop/sbin
        ./start-all.sh 
       输入几个yes          
       

   11. 查看进程 在master 上面 输入jps

       24400 SecondaryNameNode
       24565 ResourceManager
       24678 NodeManager
       24998 Jps
       24200 DataNode
       1391 Bootstrap
       24063 NameNode
       

       在node1和node2上输入jps

       11408 Jps
       11209 DataNode
       1482 Bootstrap
       11307 NodeManager
       

hadoop在web页面查看

http://192.168.202.5:8088/

参考文献

理论文档 
http://www.cnblogs.com/edisonchou/p/3485135.html

安装教程 

http://www.cnblogs.com/laov/p/3421479.html

https://my.oschina.net/lanzp/blog/309078