Hadoop-2.5.1集群环境搭建

1. 新建目录，在这个目录下安装自己的jdk和hadoop
2. JDK安装
   这个很简单，官网下载jdk到自己的用户目录下,解压配置环境变量，设为系统默认的jdk。
3. Hadoop-2.5.1的安装
   下载官方安装文件，解压到自己的安装目录，先按本地模式安装，在按伪分布模式建立和修改配置文件。全分布配置，需要的配置如下：
   core-site.xml

<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://master:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/tangm/hadoop/hadoop-2.5.1/tmp</value>
    </property>
    <property>
            <name>dfs.namenode.name.dir</name>
            <value>/usr/local/tangm/hadoop/hadoop-2.5.1/hdfs/name</value>
    </property>
    <property>
            <name>dfs.datanode.data.dir</name>
            <value>/usr/local/tangm/hadoop/hadoop-2.5.1/hdfs/data</value>
    </property>
</configuration>

hadoop-env.sh

export JAVA_HOME=/usr/local/jdk1.7.0_45
export HADOOP_HOME=/usr/local/tangm/hadoop/hadoop-2.5.1
export PATH=$PATH:${HADOOP_HOME}/bin

mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>Yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>master:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>master:19888</value>
    </property>
</configuration>

hdfs-site.xml

<configuration>
    <property>
            <name>dfs.replication</name>
            <value>3</value>
    </property>
    <property>
            <name>dfs.namenode.name.dir</name>
            <value>/usr/local/tangm/hadoop/hadoop-2.5.1/hdfs/name</value>
    </property>
    <property>
            <name>dfs.namenode.secondary.http-address</name>
            <value>master:9001</value>
    </property>
    <property>
            <name>dfs.datanode.data.dir</name>
            <value>/usr/local/tangm/hadoop/hadoop-2.5.1/hdfs/data</value>
    </property>
    <property>
            <name>dfs.webhdfs.enabled</name>
            <value>true</value>
    </property>
</configuration>

yarn-env.sh

export JAVA_HOME=/usr/local/jdk1.7.0_45

yarn-site.xml

<configuration>
    <!-- Site specific YARN configuration properties -->
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <description>The address of the applications manager interface in the RM.</description>
        <name>yarn.resourcemanager.address</name>
        <value>master:18040</value>
    </property>
    <property>
        <description>The address of the scheduler interface.</description>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>master:18030</value>
    </property>
    <property>
        <description>The address of the RM web application.</description>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>master:18088</value>
    </property>
    <property>
        <description>The address of the resource tracker interface.</description>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>master:8025</value>  
    </property>
    <property>
        <description>The address of the resource tracker interface.</description>
        <name>yarn.resourcemanager.admin.address</name>
        <value>master:8033</value>  
    </property>
</configuration>

slaves

slave1
slave2
slave3

4.遇到的问题
（1）etc/hadoop/slaves：没有这个文件或目录

[root@master hadoop-2.5.1]# sbin/start-dfs.sh 
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on [2015-06-03 17:13:21,548 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable]
-: usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
-:            [-D [bind_address:]port] [-e escape_char] [-F configfile]
-:            [-i identity_file] [-L [bind_address:]port:host:hostport]
-:            [-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]
-:            [-R [bind_address:]port:host:hostport] [-S ctl_path]
-:            [-W host:port] [-w local_tun[:remote_tun]]
-:            [user@]hostname [command]
util.NativeCodeLoader: ssh: Could not resolve hostname util.NativeCodeLoader: Temporary failure in name resolution
Unable: ssh: Could not resolve hostname Unable: Temporary failure in name resolution
2015-06-03: ssh: Could not resolve hostname 2015-06-03: Temporary failure in name resolution
WARN: ssh: Could not resolve hostname WARN: Temporary failure in name resolution
native-hadoop: ssh: Could not resolve hostname native-hadoop: Temporary failure in name resolution
[main]: ssh: Could not resolve hostname [main]: Temporary failure in name resolution
(NativeCodeLoader.java:<clinit>(62)): ssh: Could not resolve hostname (NativeCodeLoader.java:<clinit>(62)): Temporary failure in name resolution
17:13:21,548: ssh: Could not resolve hostname 17:13:21,548: Temporary failure in name resolution
library: ssh: Could not resolve hostname library: Temporary failure in name resolution
platform...: ssh: Could not resolve hostname platform...: Name or service not known
load: ssh: Could not resolve hostname load: Temporary failure in name resolution
using: ssh: Could not resolve hostname using: Temporary failure in name resolution
for: ssh: Could not resolve hostname for: Temporary failure in name resolution
applicable: ssh: Could not resolve hostname applicable: Temporary failure in name resolution
where: ssh: Could not resolve hostname where: Temporary failure in name resolution
builtin-java: ssh: Could not resolve hostname builtin-java: Temporary failure in name resolution
classes: ssh: Could not resolve hostname classes: Temporary failure in name resolution
your: ssh: Could not resolve hostname your: Temporary failure in name resolution
to: ssh: Could not resolve hostname to: Temporary failure in name resolution
cat: /etc/hadoop/slaves: 没有那个文件或目录

就是这个slaves文件没有，或者不是纯文本文档格式的文件，重写一个就ok。
（2）Exception in thread “main” java.lang.UnsupportedClassVersionError

[root@master hadoop-2.5.1]# bin/hadoop jar test.jar 
Exception in thread "main" java.lang.UnsupportedClassVersionError: com/tangm/CGAMapRedDriver : Unsupported major.minor version 52.0
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:205)
[root@master hadoop-2.5.1]# 

eclipse中编译的jdk版本（Java Compiler）与服务器jdk不一致，选择一个一直的版本作为compiler就ok。
（3）hdfs操作命令执行不成功

[root@master hadoop-2.5.1]# bin/hadoop dfs -mkdir input
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

hadoop版本将hadoop 命令换成了hdfs命令，上面的命令如同下面的命令：

[root@master hadoop]$ hdfs dfs -ls