今天下载安装、运行了下hadoop,跟着doc走,但有时也会出现一些问题,所以在这里记录一下:
1. 下载hadoop
我这里下载的是hadoop-0.21.0,之前在公司使用的是0.18,现在新版本的hadoop已经升级到了1.0.0,对API做了新的处理,下载地址:https://www.apache.org/dyn/closer.cgi/hadoop/core/
2. 解压、配置环境变量
tar -zxvf hadoop-0.21.0.tar.gz
ln -s hadoop-0.21.0 hadoop
cd hadoop
cd conf
修改hadoop-env.sh
配置JAVA_HOME
配置core-site.xml
<configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000/</value> </property> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>
mapred-site.xml
<configuration> <property> <name>mapred.job.tracker</name> <value>localhost:9001</value> </property> </configuration>
3. 格式化
./bin/hadoop namenode -format
启动hadoop
./bin/hadoop start-all.sh
验证启动是否成功
jps
4. hadoop分为单机模式、伪分布式模式、集群模式,这里 采用伪分布式模式,例子可参考hadoop快速入门文档:http://hadoop.apache.org/common/docs/r0.19.2/cn/quickstart.html
以下便是对wordcount的操作记录:
1> 编译WordCount.java
javac -classpath $classpath -d wordcount_classes WordCount.java
2>打jar包
jar -cvf wordcount.jar -C wordcount_classes
./bin/hadoop fs -put wordcount.jar .
3>增加file01 file02到input
./bin/hadoop fs -mkdir input
./bin/hadoop fs -put file01 input
./bin/hadoop fs -put file02 input
查看input目录文件
./bin/hadoop fs -ls input
4>执行wordcount统计
./bin/hadoop jar wordcount.jar WordCount input output
5>查看统计结果
./bin/hadoop fs -cat /user/gavin/output/part-00000
Bye 1
Goodbye 1
Hadoop 2
Hello 2
World 2
期间遇到的问题是output目录已存在:
gavin@ubuntu:~/programs/hadoop$ ./bin/hadoop jar wordcount.jar WordCount input output
11/12/24 23:01:18 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
11/12/24 23:01:18 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
11/12/24 23:01:19 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
11/12/24 23:01:19 INFO mapreduce.JobSubmitter: Cleaning up the staging area hdfs://localhost:9000/tmp/hadoop-gavin/mapred/staging/gavin/.staging/job_201112242122_0007
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost:9000/user/gavin/output already exists
at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:125)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:375)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:334)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:960)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:534)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:779)
at WordCount.main(WordCount.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
这是只要删除output目录即可
./bin/hadoop fs -rmr output
再重新执行jar命令就可以看到输出结果
参考:
Hadoop Map/Reduce教程
http://hadoop.apache.org/common/docs/r0.19.2/cn/mapred_tutorial.html#%E7%9B%AE%E7%9A%84
hadoop 快速入门
http://hadoop.apache.org/common/docs/r0.19.2/cn/quickstart.html
hadoop 命令手册
http://hadoop.apache.org/common/docs/r0.19.2/cn/commands_manual.html
Hadoop Shell命令
http://hadoop.apache.org/common/docs/r0.19.2/cn/hdfs_shell.html
Hadoop分布式文件系统:架构和设计
http://hadoop.apache.org/common/docs/r0.19.2/cn/hdfs_design.html
hadoop下载
https://www.apache.org/dyn/closer.cgi/hadoop/core/