centos6.5 搭建hadoop平台

1、下载hadoop-2.3.0-cdh5.0.0.tar.gz
地址：http://archive.cloudera.com/cdh5/cdh/5/
2、useradd hadoop
3、passwd hadoop
……
4、cp hadoop-2.3.0-cdh5.0.0.tar.gz /home/hadoop
5、su hadoop
6、各节点间建立ssh(可参考http://blog.csdn.net/qq_30831935/article/details/52311726)
7、解压并修改配置文件

tar -zxvf hadoop-2.3.0-cdh5.0.0.tar.gz
cd hadoop-2.3.0-cdh5.0.0/etc/hadoop

vim core-site.xml

<configuration>

    <property>
      <name>fs.defaultFS</name>
      <value>hdfs://m1:9000</value>
     </property>

     <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
      </property>
      <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/home/hadoop/tmp</value>
        <description>Abasefor other temporary directories.</description>
      </property>
      <property>
         <name>hadoop.proxyuser.spark.hosts</name>
         <value>*</value>
      </property>
      <property>
         <name>hadoop.proxyuser.spark.groups</name>
          <value>*</value>
        </property>

</configuration>

vim slaves(节点的主机名)

m1
m2
m3

vim hdfs-site.xml

<configuration>
<property>
  <name>dfs.namenode.secondary.http-address</name>
    <value>m1:9001</value>
     </property>

  <property>
     <name>dfs.namenode.name.dir</name>
        <value>file:/home/hadoop/tmp/dfs/name</value>
         </property>

 <property>
   <name>dfs.datanode.data.dir</name>
     <value>file:/home/hadoop/tmp/dfs/data</value>
       </property>

 <property>
   <name>dfs.replication</name>
     <value>3</value>
      </property>

 <property>
   <name>dfs.webhdfs.enabled</name>
     <value>true</value>
      </property>
</configuration>

vim mapred-site.xml（没有就 cp mapred-site.xml.template mapred-site.xml）

<configuration>
 <property>
    <name>mapreduce.framework.name</name>
       <value>yarn</value>
   </property>
   <property>
        <name>mapreduce.jobhistory.address</name>
        <value>m1:10020</value>
    </property>
       <property>
          <name>mapreduce.jobhistory.webapp.address</name>
          <value>m1:19888</value>
       </property>
</configuration>

vim yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->

 <property>
    <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
    </property>
    <property>
            <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
            <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>
        <property>
            <name>yarn.resourcemanager.address</name>
            <value>m1:8032</value>
        </property>
       <property>
            <name>yarn.resourcemanager.scheduler.address</name>
            <value>m1:8030</value>
       </property>
       <property>
            <name>yarn.resourcemanager.resource-tracker.address</name>
            <value>m1:8035</value>
       </property>
       <property>
             <name>yarn.resourcemanager.admin.address</name>
             <value>m1:8033</value>
       </property>
       <property>
             <name>yarn.resourcemanager.webapp.address</name>
            <value>m1:8088</value>
       </property>

</configuration>

8、配置环境变量（3台）

export HADOOP_HOME=/home/hadoop/hadoop-2.3.0-cdh5.0.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

文件夹拷贝至节点上
scp -r hadoop-2.3.0-cdh5.0.0 hadoop@m2:~
scp -r hadoop-2.3.0-cdh5.0.0 hadoop@m3:~
9、hadoop namenode -format
10、start-all.sh
如果出现java_home not found
解决办法：vim hadoop-env.sh 将java路径写全

 Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use.
export JAVA_HOME=/usr/java/jdk1.8.0_20

11、stop-all.sh
12、运行自带案例

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.0.jar pi 2 2
第一个参数2指用2个map来计算
第二个参数2指每个map执行2次
设的越大，计算时间越长，结果也更精准

13、hdfs的操作

hadoop fs -ls / 查看hdfs里的文件夹
hadoop fs -put 文件 /文件夹
hadoop fs -mkdir /文件夹…
hadoop fs -text /文件 查看文件

14、浏览器打开8088端口

浏览器打开50070端口

15、hadoop计算
hadoop jar **.jar 完整类名 文件地址 结果存到哪