ubuntu下hadoop运行wordcount程序

本机环境

ubuntu 12

hadoop 1.1.2

首先保证hadoop配置成功

1、在Hadoop的解压目录的如下位置可以找到WordCount.java的源文件 src/examples/org/apache/hadoop/examples/WordCount.java

新建一个wordcount的文件夹，将WordCount.java拷贝至dev/wordcount文件夹下

2.编译wordcount.java

3.将生成的class文件打包

4.在wordcount下建立file01 file02两个文件

5.启动hadoop,在hdfs上创建input文件夹，并将两个输入文件上传至input文件夹

root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop dfs -ls
ls: Cannot access .: No such file or directory.
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop dfs -mkdir input
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop dfs -ls
Found 1 items
drwxr-xr-x   - root supergroup          0 2014-03-04 17:48 /user/root/input
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -put /home/zcf/ 桌面/file01 input
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -put /home/zcf/ 桌面/file02 input
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -ls input
Found 2 items
-rw-r--r--   1 root supergroup         22 2014-03-04 17:50 /user/root/input/file01
-rw-r--r--   1 root supergroup         28 2014-03-04 17:50 /user/root/input/file02

6.运行wordcount.jar

root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop jar wordcount/wordcount.jar org.apache.hadoop.examples.WordCount input output
14/03/04 17:58:14 INFO input.FileInputFormat: Total input paths to process : 2
14/03/04 17:58:14 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/03/04 17:58:14 WARN snappy.LoadSnappy: Snappy native library not loaded
14/03/04 17:58:15 INFO mapred.JobClient: Running job: job_201403041744_0001
14/03/04 17:58:16 INFO mapred.JobClient:  map 0% reduce 0%
14/03/04 17:58:21 INFO mapred.JobClient:  map 50% reduce 0%
14/03/04 17:58:22 INFO mapred.JobClient:  map 100% reduce 0%
14/03/04 17:58:29 INFO mapred.JobClient:  map 100% reduce 33%
14/03/04 17:58:31 INFO mapred.JobClient:  map 100% reduce 100%
14/03/04 17:58:32 INFO mapred.JobClient: Job complete: job_201403041744_0001
14/03/04 17:58:32 INFO mapred.JobClient: Counters: 29
14/03/04 17:58:32 INFO mapred.JobClient:   Job Counters 
14/03/04 17:58:32 INFO mapred.JobClient:     Launched reduce tasks=1
14/03/04 17:58:32 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=8421
14/03/04 17:58:32 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/03/04 17:58:32 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/03/04 17:58:32 INFO mapred.JobClient:     Launched map tasks=2
14/03/04 17:58:32 INFO mapred.JobClient:     Data-local map tasks=2
14/03/04 17:58:32 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=9155
14/03/04 17:58:32 INFO mapred.JobClient:   File Output Format Counters 
14/03/04 17:58:32 INFO mapred.JobClient:     Bytes Written=41
14/03/04 17:58:32 INFO mapred.JobClient:   FileSystemCounters
14/03/04 17:58:32 INFO mapred.JobClient:     FILE_BYTES_READ=79
14/03/04 17:58:32 INFO mapred.JobClient:     HDFS_BYTES_READ=268
14/03/04 17:58:32 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=152857
14/03/04 17:58:32 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=41
14/03/04 17:58:32 INFO mapred.JobClient:   File Input Format Counters 
14/03/04 17:58:32 INFO mapred.JobClient:     Bytes Read=50
14/03/04 17:58:32 INFO mapred.JobClient:   Map-Reduce Framework
14/03/04 17:58:32 INFO mapred.JobClient:     Map output materialized bytes=85
14/03/04 17:58:32 INFO mapred.JobClient:     Map input records=2
14/03/04 17:58:32 INFO mapred.JobClient:     Reduce shuffle bytes=85
14/03/04 17:58:32 INFO mapred.JobClient:     Spilled Records=12
14/03/04 17:58:32 INFO mapred.JobClient:     Map output bytes=82
14/03/04 17:58:32 INFO mapred.JobClient:     CPU time spent (ms)=2840
14/03/04 17:58:32 INFO mapred.JobClient:     Total committed heap usage (bytes)=306511872
14/03/04 17:58:32 INFO mapred.JobClient:     Combine input records=8
14/03/04 17:58:32 INFO mapred.JobClient:     SPLIT_RAW_BYTES=218
14/03/04 17:58:32 INFO mapred.JobClient:     Reduce input records=6
14/03/04 17:58:32 INFO mapred.JobClient:     Reduce input groups=5
14/03/04 17:58:32 INFO mapred.JobClient:     Combine output records=6
14/03/04 17:58:32 INFO mapred.JobClient:     Physical memory (bytes) snapshot=382898176
14/03/04 17:58:32 INFO mapred.JobClient:     Reduce output records=5
14/03/04 17:58:32 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=1164251136
14/03/04 17:58:32 INFO mapred.JobClient:     Map output records=8

7.查看运行结果

root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -ls
Found 2 items
drwxr-xr-x   - root supergroup          0 2014-03-04 17:50 /user/root/input
drwxr-xr-x   - root supergroup          0 2014-03-04 17:58 /user/root/output
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -ls output
Found 3 items
-rw-r--r--   1 root supergroup          0 2014-03-04 17:58 /user/root/output/_SUCCESS
drwxr-xr-x   - root supergroup          0 2014-03-04 17:58 /user/root/output/_logs
-rw-r--r--   1 root supergroup         41 2014-03-04 17:58 /user/root/output/part-r-00000
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -cat /output/part-r-00000
cat: File does not exist: /output/part-r-00000
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -cat output/part-r-00000
Bye	1
Goodbye	1
Hadoop	2
Hello	2
World	2

至此，hadoop下的WordCount实例运行结束，如果还想重新运行一遍，这需把hadoop下的output文件夹删除，因为hadoop为了保证结果的正确性，存在输出的文件夹的话，就会报异常，异常如下

 ERROR security.UserGroupInformation: PriviledgedActionException as:
root cause:org.apache.hadoop.mapred.FileAlreadyExistsException:
Output directory output already exists

删除hdfs上的output文件

root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -rmr output
Deleted hdfs://localhost:9000/user/root/output