Hadoop2.5.1 运行wordcount

转载自http://blog.csdn.net/zhangzhaokun/article/details/17427835

[hadoop@lrts216 bin]$ hdfs dfs -mkdir -p /lrts/zhangzk

13/12/19 22:20:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@lrts216 bin]$ hdfs dfs -ls /
13/12/19 22:20:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2013-12-19 22:20 /lrts
[hadoop@lrts216 bin]$ hdfs dfs -put hdfs.cmd /lrts/zhangzk/
13/12/19 22:21:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@lrts216 bin]$ ll
total 252
-rwxr--r--. 1 hadoop hadoop 78809 Oct  7 14:38 container-executor
-rwxr--r--. 1 hadoop hadoop  4857 Oct  7 14:38 hadoop
-rwxr--r--. 1 hadoop hadoop  7530 Oct  7 14:38 hadoop.cmd
-rwxr--r--. 1 hadoop hadoop  7954 Oct  7 14:38 hdfs
-rwxr--r--. 1 hadoop hadoop  5138 Oct  7 14:38 hdfs.cmd
-rwxr--r--. 1 hadoop hadoop  4989 Oct  7 14:38 mapred
-rwxr--r--. 1 hadoop hadoop  5560 Oct  7 14:38 mapred.cmd
-rwxr--r--. 1 hadoop hadoop  1776 Oct  7 14:38 rcc
-rwxr--r--. 1 hadoop hadoop 95102 Oct  7 14:38 test-container-executor
-rwxr--r--. 1 hadoop hadoop  8548 Oct  7 14:38 yarn
-rwxr--r--. 1 hadoop hadoop  8322 Oct  7 14:38 yarn.cmd
[hadoop@lrts216 bin]$ hadoop jar /home/hadoop/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /lrts/zhangzk /lrts/out
13/12/19 22:24:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/12/19 22:24:56 INFO client.RMProxy: Connecting to ResourceManager at lrts216/192.168.1.216:8032
13/12/19 22:24:57 INFO input.FileInputFormat: Total input paths to process : 1
13/12/19 22:24:57 INFO mapreduce.JobSubmitter: number of splits:1
13/12/19 22:24:57 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
13/12/19 22:24:57 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
13/12/19 22:24:57 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
13/12/19 22:24:57 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
13/12/19 22:24:57 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
13/12/19 22:24:57 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
13/12/19 22:24:57 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
13/12/19 22:24:57 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
13/12/19 22:24:57 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
13/12/19 22:24:57 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
13/12/19 22:24:57 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
13/12/19 22:24:57 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
13/12/19 22:24:57 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1387462237922_0001
13/12/19 22:24:58 INFO impl.YarnClientImpl: Submitted application application_1387462237922_0001 to ResourceManager at lrts216/192.168.1.216:8032
13/12/19 22:24:58 INFO mapreduce.Job: The url to track the job: http://lrts216:8088/proxy/application_1387462237922_0001/
13/12/19 22:24:58 INFO mapreduce.Job: Running job: job_1387462237922_0001
13/12/19 22:25:06 INFO mapreduce.Job: Job job_1387462237922_0001 running in uber mode : false
13/12/19 22:25:06 INFO mapreduce.Job:  map 0% reduce 0%
13/12/19 22:25:11 INFO mapreduce.Job:  map 100% reduce 0%
13/12/19 22:25:17 INFO mapreduce.Job:  map 100% reduce 100%
13/12/19 22:25:17 INFO mapreduce.Job: Job job_1387462237922_0001 completed successfully
13/12/19 22:25:18 INFO mapreduce.Job: Counters: 43
        File System Counters
                FILE: Number of bytes read=4715
                FILE: Number of bytes written=167555
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=5244
                HDFS: Number of bytes written=3660
                HDFS: Number of read operations=6
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=1
                Launched reduce tasks=1
                Rack-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=2410
                Total time spent by all reduces in occupied slots (ms)=2941
        Map-Reduce Framework
                Map input records=171
                Map output records=523
                Map output bytes=6735
                Map output materialized bytes=4715
                Input split bytes=106
                Combine input records=523
                Combine output records=264
                Reduce input groups=264
                Reduce shuffle bytes=4715
                Reduce input records=264
                Reduce output records=264
                Spilled Records=528
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=35
                CPU time spent (ms)=750
                Physical memory (bytes) snapshot=398831616
                Virtual memory (bytes) snapshot=1765736448
                Total committed heap usage (bytes)=354942976
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=5138
        File Output Format Counters 
                Bytes Written=3660
[hadoop@lrts216 bin]$ hdfs dfs -ls /lrts/out
13/12/19 22:26:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   2 hadoop supergroup          0 2013-12-19 22:25 /lrts/out/_SUCCESS
-rw-r--r--   2 hadoop supergroup       3660 2013-12-19 22:25 /lrts/out/part-r-00000
[hadoop@lrts216 bin]$ hdfs dfs -cat /lrts/out/part-r-00000
13/12/19 22:27:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
"%1"    2
"%HADOOP_BIN_PATH:~-1%" 1
"--config"      2
"AS     1
"License");     1
"\"     1
%*      2
%1      1
%1,     1

%2      1

安装参考：

http://www.yongbok.net/blog/how-to-install-hadoop-2-2-0-pseudo-distributed-mode/

http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ClusterSetup.html

由于我使用的OS是centos 64位的，所以其中出了些问题，需要自行编译hadoop。

参考资料如下：

http://blog.csdn.net/w13770269691/article/details/16883663

http://blog.csdn.net/bamuta/article/details/13506843

http://blog.csdn.net/lalaguozhe/article/details/10580727

http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/NativeLibraries.html