Mapreduce 测试自带实例 wordcount

2.7.3版本的hadoop：

jar程序所在目录：$HADOOP_HOME/shar/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar

 

1.本地创建测试文本：

mkdir /home/hadoop/data  //创建data文件夹
touch testinput.txt  //创建测试文本
vim testinput.txt  //修改文本

//文本添加一行
this is a test log

cat testinput.txt  //检查txt文本

 

2.hdfs

hadoop fs -ls /  //查看hdfs上的目录
hadoop fs -mkdir /input  //创建input目录
hadoop fs -rm -r /output  //如果有output目录，删除
hadoop fs -put /home/hadoop/data/testiinput.txt /input  //把测试文本上传到input目录上

hadoop jar /home/hadoop/softwares/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /input /output  
//执行程序,wordcount为程序的主类名， /input  输入目录  /output 输出目录（输出目录不能存在）


hadoop fs -ls /output  //完成后查看输出目录
hadoop fs -cat /output/part-r-00000   //查看输出结果

 

运行过程：

18/04/16 19:43:15 INFO client.RMProxy: Connecting to ResourceManager at hadoop/192.168.30.129:8032
18/04/16 19:43:18 INFO input.FileInputFormat: Total input paths to process : 1
18/04/16 19:43:18 INFO mapreduce.JobSubmitter: number of splits:1
18/04/16 19:43:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1523884458275_0001
18/04/16 19:43:19 INFO impl.YarnClientImpl: Submitted application application_1523884458275_0001
18/04/16 19:43:19 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1523884458275_0001/
18/04/16 19:43:19 INFO mapreduce.Job: Running job: job_1523884458275_0001
18/04/16 19:43:41 INFO mapreduce.Job: Job job_1523884458275_0001 running in uber mode : false
18/04/16 19:43:41 INFO mapreduce.Job:  map 0% reduce 0%
18/04/16 19:43:51 INFO mapreduce.Job:  map 100% reduce 0%
18/04/16 19:44:03 INFO mapreduce.Job:  map 100% reduce 100%
18/04/16 19:44:05 INFO mapreduce.Job: Job job_1523884458275_0001 completed successfully
18/04/16 19:44:05 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=6
        FILE: Number of bytes written=237375
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=100
        HDFS: Number of bytes written=0
        HDFS: Number of read operations=6
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Launched reduce tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=8459
        Total time spent by all reduces in occupied slots (ms)=8037
        Total time spent by all map tasks (ms)=8459
        Total time spent by all reduce tasks (ms)=8037
        Total vcore-milliseconds taken by all map tasks=8459
        Total vcore-milliseconds taken by all reduce tasks=8037
        Total megabyte-milliseconds taken by all map tasks=8662016
        Total megabyte-milliseconds taken by all reduce tasks=8229888
    Map-Reduce Framework
        Map input records=0
        Map output records=0
        Map output bytes=0
        Map output materialized bytes=6
        Input split bytes=100
        Combine input records=0
        Combine output records=0
        Reduce input groups=0
        Reduce shuffle bytes=6
        Reduce input records=0
        Reduce output records=0
        Spilled Records=0
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=264
        CPU time spent (ms)=1460
        Physical memory (bytes) snapshot=287879168
        Virtual memory (bytes) snapshot=3887841280
        Total committed heap usage (bytes)=139845632
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=0

View Code

 

结果：

a    1
is    1
log    1
test    1
this    1

 

转载于:https://www.cnblogs.com/xym4869/p/8862417.html