关闭

[置顶] 用cygwin实现hadoop 全分布配置

标签: hadoopjavavariablesstringinputssh
1060人阅读 评论(0) 收藏 举报
分类:

最近在研究hadoop

搞了半个月终于实现在window下利用cygwin的全分布配置

环境

1、hadoop 0.20.2

2、vista 机器一台 192.168.0.102   机器名:ken-PC(master)

       winxp 虚拟机   192.168.0.222  机器名:winxp   (slaves)


步骤

1、分别在vista和xp上安装cygwin,分别在两个系统中建立两个名字相同的用户,ken,配置好SSH,确保master机器可以无密码在cygwin上访问slaves机器, 这个网上很多文章有介绍,记得关闭window防火墙,否则无办法互联

2、在vista上安装hadoop,具体配置如下,注意红色字

  hadoop-env.sh

# Set Hadoop-specific environment variables here.


# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.


# The java implementation to use.  Required.
 export JAVA_HOME=/cygdrive/C/soft/Java/jdk1.6.0_12 


# Extra Java CLASSPATH elements.  Optional.
# export HADOOP_CLASSPATH=


# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000


# Extra Java runtime options.  Empty by default.
# export HADOOP_OPTS=-server


# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS


# Extra ssh options.  Empty by default.
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"


# Where log files are stored.  $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs


# File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves


# host:path where hadoop code should be rsync'd from.  Unset by default.
# export HADOOP_MASTER=master:/home/$USER/src/hadoop


# Seconds to sleep between slave commands.  Unset by default.  This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=0.1


# The directory where pid files are stored. /tmp by default.
# export HADOOP_PID_DIR=/var/hadoop/pids


# A string representing this instance of hadoop. $USER by default.
# export HADOOP_IDENT_STRING=$USER


# The scheduling priority for daemon processes.  See 'man nice'.
# export HADOOP_NICENESS=10



core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>

<property><name>fs.default.name</name><value>hdfs://192.168.0.102:9991</value></property>

<property><name>hadoop.tmp.dir</name><value>/root/hadoopfile/202/coretmp/</value></property>

</configuration>



hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>

<property><name>dfs.replication</name><value>1</value></property>

<property><name>dfs.name.dir</name><value>hadoopfile/name/</value></property>

<property><name>dfs.data.dir</name><value>hadoopfile/data/</value></property>

</configuration>


mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property><name>mapred.job.tracker</name><value>192.168.0.102:9992</value></property>

<property><name>mapred.child.tmp</name><value>C:\root\tmp</value></property>

</configuration>

注意:mapred.child.tmp一定要是绝对路径



masters

192.168.0.102


slaves

192.168.0.222


3、在虚拟的winxp上安装hadoop,配置文件内容要与vista上的配置文件内容相同,hadoop  存放的路径也要与vista相同


4、分别在vista和winxp上的host文件添加如下内容

192.168.0.102             ken-PC
192.168.0.222             winxp

大小写敏感


5、在vista上格式化hadoop


6、在vista上启动hadoop


7、在vista上通过http://localhost:50070查看运行情况

 

8、在vista上运行wordcount示例

首先在hadoop当前目录建立一个input文件

然后在input文件中建立两个text文件,文件内容随意


在cygwin里面输入


运行wordcount



 


0
0

查看评论
* 以上用户言论只代表其个人观点,不代表CSDN网站的观点或立场
    个人资料
    • 访问:1367次
    • 积分:20
    • 等级:
    • 排名:千里之外
    • 原创:1篇
    • 转载:0篇
    • 译文:0篇
    • 评论:0条
    文章分类
    文章存档