Redhat上Hadoop2.6.0集群安装



一、安装虚拟机Vmware、redhat

1．Vmware11、redhat6.5下载

安装参考

http://jingyan.baidu.com/article/c910274be62c6bcd361d2d37.html

2.  Linux与Windows文件共享

安装Winscp实现文件共享

 

 

二、SSH无密码登录

1.关闭防火墙

   执行命令 service iptables stop

 永久关闭防火墙  chkconfigiptables off

验证     service iptables status

2.创建hadoop用户

 # adduser hadoop

# passwd hadoop

2.复制三个linux镜像文件

3.修改ip

3.1编辑虚拟机设置--网络配置-仅主机模式

 

3.2启动主机编辑右上角网络配置，修改ip

   192.168.80.100  255.255.255.0 192.168.80.1（是vmwarenet1的ip\ vmwarenet8禁用）

192.168.80.101  255.255.255.0 192.168.80.1 

         192.168.80.102  255.255.255.0 192.168.80.1 

 

3.3修改主机名 

第一步：
#hostname
第二步：
修改/etc/sysconfig/network中的hostname
第三步：
修改/etc/hosts文件

      192.168.80.100  主机名:Master.Hadoop

            192.168.80.101  主机名: Slave1.Hadoop

           192.168.80.102  主机名:Slave2.Hadoop

   重启-验证命令hostname

 

3.4 ip与hostname绑定

执行命令 vi /etc/hosts

       添加 192.168.80.100   Master.Hadoop

                  192.168.80.101  Slave1.Hadoop

                 192.168.80.102   Slave2.Hadoop

验证    ping Master.Hadoop

 

3.5 设置ssh免密码登录

3.5.1在Master.Hadoop下执行以下操作

    (1) su- hadoop

 (2) ll ~/.ssh查看是否存在id_rsa.pub否则执行(2)是则执行（3）

(3) ssh-keygen -t rsa一直回车生成id_rsa.pub

(4) cat id_rsa.pub >> authorized_keys 将id_rsa.pub加到授权的key里面去(等价于命令：cpid_rsa.pub authorized_keys )

3.5.2在Slave1.Hadoop\Slave2.Hadoop下执行以下操作

    执行命令

mkdir /home/hadoop/.ssh

chmod 700 /home/hadoop/.ssh

　 3.5.3 Master.Hadoop把刚刚产生的authorized_keys文件拷一份到每个节点上.本次操作需要输入密码

scp authorized_keys hadoop@192.168.80.101:/home/hadoop/.ssh 

     scp authorized_keys hadoop@192.168.80.102:/home/hadoop/.ssh

如果报错没有该目录则

3.5.4在Slave1.Hadoop\Slave2.Hadoop下执行命令

cd /home/hadoop/.ssh

chmod 600 authorized_keys      

3.5.4登录.100执行命令 ssh192.168.80.101\ ssh 192.168.80.101

若无需输入密码表示ssh无密码登录成功

 

 到此完成了Master.Hadoop对Slave1.Hadoop和Slave2.Hadoop的无密码登录

重复以上操作完成各节点之间的无密法登陆。

 

三、安装jdk、Hadoop

1. Jdk1.7、Hadoop2.6.0下载

2. Jdk1.7压缩文件共享到/usr/newdir/java

   Hadoop2.6.0压缩文件共享到/usr/newdir/hadoop        

3. 解压tar –zxvf jdk1.7.0_75.tar.gz

       tar -zxvf Hadoop-2.6.0.tar.gz

4. 修改权限chown -Rhadoop:hadoop java/

chown -R hadoop:hadoop hadoop/

5. 配置环境变量

 vi /etc/profile

       exportJAVA_HOME=/usr/newdir/java/jdk1.7.0_75

export HADOOP_HOME=/usr/newdir/hadoop/hadoop-2.6.0

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:

exportPATH=.:$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:

 

 6. source /etc/profile

 

验证  java –version

      hadoop version

 

四、hadoop伪分布参数配置

 4.1 Master.Hadoop参数配置

mkdir /usr/hadoop/tmp

mkdir /usr/hadoop/dfs/name

mkdir /usr/hadoop/dfs/data

 

cd /usr/newdir/hadoop/hadoop-2.6.0/etc/hadoop/

 

这里要涉及到的配置文件有7个：

hadoop-env.sh

yarn-env.sh

slaves

core-site.xml

hdfs-site.xml

mapred-site.xml（复制相应的template文件获得）

yarn-site.xml

 

配置文件1：hadoop-env.sh

修改JAVA_HOME值（exportJAVA_HOME=/usr/newdir/java/jdk1.7.0_75）

 

配置文件2：yarn-env.sh

修改JAVA_HOME值（export JAVA_HOME=/usr/newdir/java/jdk1.7.0_75）

 

配置文件3：slaves（这个文件里面保存所有slave节点）

 Slave1.Hadoop

 Slave2.Hadoop

 

配置文件4: core-site.xml

 

<configuration>

      <property>

           <name>fs.defaultFS</name>

          <value>hdfs://Master.Hadoop:9000</value>

       </property>

       <property>

           <name>io.file.buffer.size</name>

           <value>131072</value>

        </property>

         <property>

            <name>hadoop.tmp.dir</name>

            <value>file:/usr/newdir/hadoop/tmp</value>

            <description>Abase for other temporary  directories.</description>

         </property>                                                                              

       <property>                                                                                                                                       

    <name>hadoop.proxyuser.hadoop.hosts</name>                                                                                                                                            <value>*</value>                                                                                                                                                         

          </property>

</configuration>

 

                                                                                                                   

配置文件5：hdfs-site.xml

 

<configuration>

 

      <property>

                 <name>dfs.namenode.secondary.http-address</name>

 

              <value>Master.Hadoop:9001</value>

 

      </property>

 

    <property>

 

            <name>dfs.namenode.name.dir</name>

 

            <value>file:/usr/newdir/hadoop/dfs/name</value>

 

      </property>

 

     <property>

 

             <name>dfs.datanode.data.dir</name>

 

             <value>file:/usr/newdir/hadoop/dfs/data</value>

 

      </property>

 

      <property>

 

              <name>dfs.replication</name>

 

              <value>1</value>

 

       </property>

 

       <property>

 

                <name>dfs.webhdfs.enabled</name>

 

                 <value>true</value>

 

        </property>

  </configuration>

 

配置文件6：mapred-site.xml

 

<configuration>

        <property>

             <name>mapreduce.framework.name</name>

 

               <value>yarn</value>

 

          </property>

 

         <property>

 

                 <name>mapreduce.jobhistory.address</name>

 

                 <value>Master.Hadoop:10020</value>

 

        </property>

         <property>

               <name>mapreduce.jobhistory.webapp.address</name>

 

               <value>Master.Hadoop:19888</value>

 

      </property>

 

</configuration>

 

配置文件7：yarn-site.xml

 

<configuration>

 

<!-- Site specific YARN configuration properties-->

 

        <property>

 

              <name>yarn.nodemanager.aux-services</name>

 

              <value>mapreduce_shuffle</value>

 

       </property>

 

       <property>                                                               

 

              <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

 

              <value>org.apache.hadoop.mapred.ShuffleHandler</value>

 

       </property>

 

       <property>

 

              <name>yarn.resourcemanager.address</name>

 

              <value>Master.Hadoop:8032</value>

 

      </property>

 

      <property>

 

              <name>yarn.resourcemanager.scheduler.address</name>

 

              <value>Master.Hadoop:8030</value>

 

      </property>

 

      <property>

            <name>yarn.resourcemanager.resource-tracker.address</name>

 

            <value>Master.Hadoop:8031</value>

 

     </property>

 

     <property>

              <name>yarn.resourcemanager.admin.address</name>

 

              <value>Master.Hadoop:8033</value>

 

      </property>

      <property>

 

              <name>yarn.resourcemanager.webapp.address</name>

 

              <value>Master.Hadoop:8088</value>

 

      </property>

 

</configuration>

 

4.2复制到其他节点

上面配置完毕，我们基本上完成了90%了剩下就是复制。我们可以把整个hadoop和java文件夹复制过去：使用如下命令：

 scp -r /usr/newdir@slave1。Hadoop:/usr/

scp -r /usr/newdir @slave1。Hadoop:/usr/

五、启动验证

格式化namenode：

cd /usr/newdir/hadoop/hadoop-2.6.0/bin

 

hdfs namenode –format

启动hdfs:

2. start-dfs.sh

此时在master上面运行的进程有：jps

 

namenode

secondarynamenode

slave节点上面运行的进程有：jps

datanode

 

启动yarn:

start-yarn.sh

我们看到如下效果：

Master.Hadoop有如下进程：

namenode

secondarynamenode

ResourceManager

slave1有如下进程
datanode

nodemanager

【注意】：而且所有的配置文件<name>和<value>节点处不要有空格，否则会报错！

http://192.168.80.100:50070/dfshealth.html#tab-snapshot