Hadoop

Hadoop

下载：http://labs.renren.com/apache-mirror/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.tar.gz

Windows下安装：

http://www.cygwin.com/setup.exe

 

Cygwin  DLL：

 

http://src.chromium.org/viewvc/chrome/trunk/src/third_party/cygwin/bin/?pathrev=490

 

安装出错的时候，重装就OK

 

下载

gzip hadoop-0.20.2.tar.gz

tar xvf hadoop-0.20.2.tar

cd /home/bmb/hadoop-0.20.2

chmod 777  /home/bmb/hadoop-0.20.2/bin

 

JDK

/home/bmb/jdk1.6.0_16/

vi /etc/profile

export JAVA_HOME=/home/bmb/jdk1.6.0_16/

export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH:$HOMR/bin

 

touch /etc/profile

 

设置conf/hadoop-env.sh里面的Java_Home

vi conf/hadoop-env.sh

export JAVA_HOME=/home/bmb/jdk1.6.0_16

SSH 处理

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

SSH 信任 无密码 无口令 登录 ssh-keygen

http://www.leakon.com/archives/281

http://plog.longwin.com.tw/post/1/293

修改配置文件（hadoop-0.20.2\docs\quickstart.html ）

所有地址，要么用IP，要么用域名。放弃localhost

/home/bmb/hadoop-0.20.2/conf

conf/core-site.xml:

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://192.168.2.79:9000</value>
  </property>
</configuration>

conf/hdfs-site.xml:

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>

  <property>
    <name>dfs.name.dir</name>
    <value>/home/bmb/hadoop-0.20.2/hdfs-name</value>
  </property>

  <property>
    <name>dfs.data.dir</name>
    <value>/home/bmb/hadoop-0.20.2/hdfs-data</value>
  </property>
</configuration>

conf/mapred-site.xml:

<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>192.168.2.79:9001</value>
  </property>
</configuration>

conf/masters

192.168.2.79

 

 

conf/slaves

192.168.2.79

 

HDFS操作

格式化文件系统

bin/hadoop namenode -format

启动Hadoop

bin/start-all.sh

停止

bin/stop-all.sh

用jps命令可以查看启动的Java进程

 

修改windows文件：C:\WINDOWS\system32\drivers\etc\hosts,添加对IP和机器名的映射

192.168.2.79    webdev

NameNode - http://192.168.2.79:50070/  

JobTracker - http://192.168.2.79:50030/

 

 

 

Pseudo-Distributed (可以用来检测按照是否正确)

cd /home/bmb/hadoop-0.20.2

 

mkdir input 
cp conf/*.xml input 

bin/hadoop fs -put conf input

bin/hadoop fs -ls /user/root/input
bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+' 
查看结果

bin/hadoop fs -get output output

cat output/*

或者

bin/hadoop fs -cat output/* 

 

 

批量删除进程

kill -9 `ps -ef |grep hadoop |awk '{print $2}' `