hadoop2集群搭建

本地装了3个虚拟机，作为hadoop的三个节点（namenode，datanode1，datanode2）

1、首先配置三台虚拟机的静态Ip，相关操作可以查看：http://blog.csdn.net/u013035314/article/details/46662781

2、安装jdk

现在虚拟机一般都自己已经装好了默认jdk，首先将其卸载

（1）先查看默认jdk：rpm -qa | grep jdk

（2）运行命令卸载：rpm -e --nodeps java-1.7...（有多少卸多少）

（3）重新下载jdk并解压: 

wget -c http://download.oracle.com/otn-pub/java/jdk/7/jdk-7-linux-i586.tar.gz

tar zxvf ./jdk-7-...

 (4) 配置文件：

vim /etc/profile

export JAVA_HOME=/home/wxshi/java/jdk1.7.0_76
export JRE_HOME=/home/wxshi/java/jdk1.7.0_76/jre
export CLASSPATH=.:%JAVA_HOME%/lib/dt.jar:%JAVA_HOME%/lib/tools.jar
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH</span>

#使其生效

source /etc/profile

#验证

java -version

jdk安装完毕

3、配置ssh自动登录（namenode到datanode自动登录就好）

[root@nameNode ~]# cd .ssh  
[root@nameNode .ssh]#

[root@nameNode /]# ssh-keygen -t rsa 

一直enter下去，直接使用默认配就好：

[root@nameNode /]# ssh-keygen -t rsa    
Generating public/private rsa key pair.    
Enter file in which to save the key (/root/.ssh/id_rsa):    
Enter passphrase (empty for no passphrase):    
Enter same passphrase again:    
Your identification has been saved in /root/.ssh/id_rsa.    
Your public key has been saved in /root/.ssh/id_rsa.pub.    
The key fingerprint is:    
98:3c:31:5c:23:21:73:a0:a0:1f:c6:d3:c3:dc:58:32 root@gifer    
The key's randomart image is:    
+--[ RSA 2048]----+    
|.   E.=.o        |    
|.o = @ o .       |    
|. * * =          |    
| o o o =         |    
|  .   = S        |    
|       .         |    
|                 |    
|                 |    
|                 |    
+-----------------+

上图密钥生成成功，目录下多出两个文件

私钥文件：id_raa
公钥文件：id_rsa.pub

将公钥文件id_rsa.pub内容放到authorized_keys文件中：

cat id_rsa.pub >> authorized_keys 

私钥文件id_raa分发给各个datanode节点：

scp authorized_keys  root@dataNode:/root/.ssh/ 

验证：

ssh root@datanode1

4、配置安装hadoop

（1）下载hadoop-2.7.1

http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz

（2）解压到指定目录：

tar -zxf hadoop-2.7.1.tar.gz  

（3）配置核心文件：

cd hadoop-2.7.1/etc/hadoop

a. core-site.xml

<configuration>  
    <property>  
        <name>hadoop.tmp.dir</name>  
        <value>/home/hadoop/tmp</value>  
        <description>Abase for other temporary directories.</description>  
    </property>  
    <property>  
        <name>fs.defaultFS</name>  
        <value>hdfs://namenode:9000</value>  
    </property>  
    <property>  
        <name>io.file.buffer.size</name>  
        <value>4096</value>  
    </property>  
</configuration>

b. hdfs-site.xml

<configuration>  
    <property>  
        <name>dfs.nameservices</name>  
        <value>hadoop-cluster1</value>  
    </property>  
    <property>  
        <name>dfs.namenode.secondary.http-address</name>  
        <value>namenode:50090</value>  
    </property>  
    <property>  
        <name>dfs.namenode.name.dir</name>  
        <value>file:///home/hadoop/dfs/name</value>  
    </property>  
    <property>  
        <name>dfs.datanode.data.dir</name>  
        <value>file:///home/hadoop/dfs/data</value>  
    </property>  
    <property>  
        <name>dfs.replication</name>  
        <value>2</value>  
    </property>  
    <property>  
        <name>dfs.webhdfs.enabled</name>  
        <value>true</value>  
    </property>  
</configuration>  

c. mapred-site.xml

<configuration>  
    <property>  
        <name>mapreduce.framework.name</name>  
        <value>yarn</value>  
    </property>  
    <property>  
        <name>mapreduce.jobtracker.http.address</name>  
        <value>namenode:50030</value>  
    </property>  
    <property>  
        <name>mapreduce.jobhistory.address</name>  
        <value>namenode:10020</value>  
    </property>  
    <property>  
        <name>mapreduce.jobhistory.webapp.address</name>  
        <value>namenode:19888</value>  
    </property>  
</configuration> 

d. yarn-site.xml

<configuration>  
  
<!-- Site specific YARN configuration properties -->  
    <property>  
        <name>yarn.nodemanager.aux-services</name>  
        <value>mapreduce_shuffle</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.address</name>  
        <value>namenode:8032</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.scheduler.address</name>  
        <value>namenode:8030</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.resource-tracker.address</name>  
        <value>nameNode:8031</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.admin.address</name>  
        <value>namevode:8033</value>  
    </property>  
    <property>  
        <name>yarn.resourcemanager.webapp.address</name>  
        <value>namenode:8088</value>  
    </property>  
</configuration>  

e. slaves

datanode1  
datanode2  

f. 分别在文件hadoop-env.sh和yarn-env.sh中添加JAVA_HOME配置

#文件1

vi hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_65

#文件2

vi yarn-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_65 

格式化文件系统：

bin/hdfs namenode -format  

h.启动服务：

[root@nameNode sbin]# ./start-dfs.sh  
[root@nameNode sbin]# ./start-yarn.sh 

i.停止服务：

[root@nameNode sbin]# ./stop-dfs.sh 
[root@nameNode sbin]# ./stop-yarn.sh  

j.查看：

http://192.168.60.200:50070/
http://192.168.60.200:8088/

搞定！