hadoop单机伪分布式安装(windows及centos)

Hadoop安装

         1.windows

                  安装JDK，配置环境变量

                  安装cygwin （linux虚拟平台）  http://www.cygwin.com

                  安装SSH，实现SSH无密码登陆

                          $ssh-keygen

                                   完成后会在~/.ssh/生成两个文件：id_dsa 和 id_dsa.pub

                          $cat~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

                                   ($ssh-copy-id hadoop@localhost) --推荐使用这个命令

                                   将id_rsa.pub追加到授权key里面

 

                  下载Hadoop（http://hadoop.apache.org）

                  配置Hadoop

                          配置hadoop-env.sh   (hadoop环境变量配置)

                                   指定java_home目录

                                   exportJAVA_HOME=/cygdrive/c/jdk

                                   $shhadoop-env.sh

 

                          配置       conf/core-site.xml (common属性配置)   

                                                     添加：   <property>

                                                                               <name>fs.default.name</name>

                                                                               <value>hdfs://localhost:9000</value>

                                                                      </property>

                                                                      <property>

                                                                               <name>fs.checkpoint.dir</name>

                                                                               <value>/hadoop/dfs/secondnode</value>

                                                                      </property>

                                            conf/hdfs-site.xml(HDFS属性配置) 

                                                     添加：    <property>

                                                                               <name>dfs.replication</name>

                                                                               <value>1</value>

                                                                      </property>

                                                                      <property>

                                                                               <name>dfs.name.dir</name>

                                                                               <value>/hadoop/dfs/namenode</value>

                                                                      </property>

                                                                      <property>

                                                                               <name>dfs.data.dir</name>

                                                                               <value>/hadoop/dfs/datanode</value>

                                                                      </property>

 

                                            conf/mapred-site.xml (MapReduce属性配置)   文件

                                                     添加：   <property>

                                                                               <name>mapred.job.tracker</name>

                                                                               <value>localhost:9001</value>

                                                                      </property>

                  启动

                          1.格式化文件系统

                                   hadoopnamenode -format

                          2.启动hadoop

                                   1.启动所有任务  start-all.sh/stop-all.sh

                                   2.启动关闭HDFS  start-dfs.sh/stop-dfs.sh

                                   3.启动关闭MapReduce   start-mapred.sh/stop-mapred.sh

 

                          3.用jps命令查看进程，确保有

                                            namenode,datanode, jobtracker, tasktracker,secondarynamenode

                                    

 

         2.Linux (CentOS 6.4系统)

                  1.下载安装JDK，Hadoop

                  2.配置环境变量

                          修改/etc/profile文件，添加

                                   exportJAVA_HOME=/usr/jdk

                                   exportHADOOP_INSTALL=/usr/hadoop

                                   exportPATH=/usr/jdk/bin:/usr/hadoop/bin:$PATH

                  3.配置ssh, 实现无密码登陆

                          $ssh-keygen

                                   完成后会在~/.ssh/生成两个文件：id_dsa 和 id_dsa.pub

                          $cat ~/.ssh/id_rsa.pub>> ~/.ssh/authorized_keys

                                   将id_rsa.pub追加到授权key里面

                          $ssh localhost

                  4.配置Hadoop

                          参考windows配置

                  5.启动

                          参考windows配置

 

         3.Hadoop 端口

                          1.Job Tracker 管理界面50030  

                          2.HDFS 管理界面 50070  

                          3.HDFS通信端口 9000 

                          4.MapReduce 通信端口 9001 

 

         4.WordCount 示例

                  1.现在本地磁盘上建立两个文件file1和file2

                           $echo "Hello World bye world">> /input/file1

                           $echo "hello hadoop bye hadoop">> /input/file2

                  2.在HDFS上建立一个input目录

                           $hadoop fs -mkdir /tmp/input

                  3.将file1和file2拷贝到hdfs的input目录下

                           $hadoopfs -put /input/* /tmp/input

                  4.检查hdfs上有没有file1,file2

                           $hadoopfs -ls /tmp/input

                  5.执行wordcount

                           $hadoopjar hadoop-examples-1.0.0.jar wordcount /tmp/input /tmp/output

                  6.运行完成，查看结果

                           $hadoopfs -cat /tmp/output/part-r-00000