用cygwin实现hadoop 全分布配置

最近在研究hadoop

搞了半个月终于实现在window下利用cygwin的全分布配置

环境

1、hadoop 0.20.2

2、vista 机器一台 192.168.0.102   机器名：ken-PC(master)

       winxp 虚拟机   192.168.0.222  机器名：winxp   (slaves)

步骤

1、分别在vista和xp上安装cygwin，分别在两个系统中建立两个名字相同的用户，ken，配置好SSH，确保master机器可以无密码在cygwin上访问slaves机器， 这个网上很多文章有介绍,记得关闭window防火墙，否则无办法互联

2、在vista上安装hadoop,具体配置如下，注意红色字

  hadoop-env.sh

# Set Hadoop-specific environment variables here.


# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.


# The java implementation to use.  Required.
 export JAVA_HOME=/cygdrive/C/soft/Java/jdk1.6.0_12 


# Extra Java CLASSPATH elements.  Optional.
# export HADOOP_CLASSPATH=


# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000


# Extra Java runtime options.  Empty by default.
# export HADOOP_OPTS=-server


# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS


# Extra ssh options.  Empty by default.
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"


# Where log files are stored.  $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs


# File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves


# host:path where hadoop code should be rsync'd from.  Unset by default.
# export HADOOP_MASTER=master:/home/$USER/src/hadoop


# Seconds to sleep between slave commands.  Unset by default.  This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=0.1


# The directory where pid files are stored. /tmp by default.
# export HADOOP_PID_DIR=/var/hadoop/pids


# A string representing this instance of hadoop. $USER by default.
# export HADOOP_IDENT_STRING=$USER


# The scheduling priority for daemon processes.  See 'man nice'.
# export HADOOP_NICENESS=10

core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>

<property><name>fs.default.name</name><value>hdfs://192.168.0.102:9991</value></property>

<property><name>hadoop.tmp.dir</name><value>/root/hadoopfile/202/coretmp/</value></property>

</configuration>

hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>

<property><name>dfs.replication</name><value>1</value></property>

<property><name>dfs.name.dir</name><value>hadoopfile/name/</value></property>

<property><name>dfs.data.dir</name><value>hadoopfile/data/</value></property>

</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property><name>mapred.job.tracker</name><value>192.168.0.102:9992</value></property>

<property><name>mapred.child.tmp</name><value>C:\root\tmp</value></property>

</configuration>

注意：mapred.child.tmp一定要是绝对路径

masters

192.168.0.102

slaves

192.168.0.222

3、在虚拟的winxp上安装hadoop，配置文件内容要与vista上的配置文件内容相同,hadoop  存放的路径也要与vista相同

4、分别在vista和winxp上的host文件添加如下内容

192.168.0.102             ken-PC
192.168.0.222             winxp

大小写敏感

5、在vista上格式化hadoop

6、在vista上启动hadoop

7、在vista上通过http://localhost:50070查看运行情况

 

8、在vista上运行wordcount示例

首先在hadoop当前目录建立一个input文件

然后在input文件中建立两个text文件，文件内容随意

在cygwin里面输入

运行wordcount