15/04/22 15:01:51 INFO mapred.JobClient: map 0% reduce 0%
15/04/22 15:02:06 INFO mapred.JobClient: Task Id : attempt_201504211850_0019_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: myback.Testmapreduce$testMap
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: myback.Testmapreduce$testMap
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
... 8 more
通过阅读以上错误,我们发现主要是自己定义的testmap类寻找不到,这是为什么?
首先,我们是在windows下通过Eclipse连接Linux下的Hadoop,当选择将程序运行在Hadoop的时候,虽然本工程下有testmap类,但是run on Hadoop的时候,由于配置环境中没有testmap这个类,故而寻找不到。
解决办法是将工程达成jar包,加入到配置环境中。运行就不会再出现以上错误了。
15/04/22 15:11:10 WARN snappy.LoadSnappy: Snappy native library not loaded
15/04/22 15:11:10 INFO mapred.JobClient: Running job: job_201504211850_0020
15/04/22 15:11:11 INFO mapred.JobClient: map 0% reduce 0%
15/04/22 15:11:35 INFO mapred.JobClient: map 100% reduce 0%
15/04/22 15:11:51 INFO mapred.JobClient: map 100% reduce 100%
15/04/22 15:11:56 INFO mapred.JobClient: Job complete: job_201504211850_0020
15/04/22 15:11:56 INFO mapred.JobClient: Counters: 29
15/04/22 15:11:56 INFO mapred.JobClient: Job Counters
15/04/22 15:11:56 INFO mapred.JobClient: Launched reduce tasks=1
15/04/22 15:11:56 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=21618
15/04/22 15:11:56 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
15/04/22 15:11:56 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
15/04/22 15:11:56 INFO mapred.JobClient: Launched map tasks=1
15/04/22 15:11:56 INFO mapred.JobClient: Data-local map tasks=1
15/04/22 15:11:56 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=15858
15/04/22 15:11:56 INFO mapred.JobClient: File Output Format Counters
15/04/22 15:11:56 INFO mapred.JobClient: Bytes Written=71
15/04/22 15:11:56 INFO mapred.JobClient: FileSystemCounters
15/04/22 15:11:56 INFO mapred.JobClient: FILE_BYTES_READ=116
15/04/22 15:11:56 INFO mapred.JobClient: HDFS_BYTES_READ=184
15/04/22 15:11:56 INFO mapred.JobClient: FILE_BYTES_WRITTEN=43503
15/04/22 15:11:56 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=71
15/04/22 15:11:56 INFO mapred.JobClient: File Input Format Counters
15/04/22 15:11:56 INFO mapred.JobClient: Bytes Read=78
15/04/22 15:11:56 INFO mapred.JobClient: Map-Reduce Framework
15/04/22 15:11:56 INFO mapred.JobClient: Map output materialized bytes=116
15/04/22 15:11:56 INFO mapred.JobClient: Map input records=5
15/04/22 15:11:56 INFO mapred.JobClient: Reduce shuffle bytes=116
15/04/22 15:11:56 INFO mapred.JobClient: Spilled Records=10
15/04/22 15:11:56 INFO mapred.JobClient: Map output bytes=100
15/04/22 15:11:56 INFO mapred.JobClient: CPU time spent (ms)=3730
15/04/22 15:11:56 INFO mapred.JobClient: Total committed heap usage (bytes)=131796992
15/04/22 15:11:56 INFO mapred.JobClient: Combine input records=0
15/04/22 15:11:56 INFO mapred.JobClient: SPLIT_RAW_BYTES=106
15/04/22 15:11:56 INFO mapred.JobClient: Reduce input records=5
15/04/22 15:11:56 INFO mapred.JobClient: Reduce input groups=1
15/04/22 15:11:56 INFO mapred.JobClient: Combine output records=0
15/04/22 15:11:56 INFO mapred.JobClient: Physical memory (bytes) snapshot=182493184
15/04/22 15:11:56 INFO mapred.JobClient: Reduce output records=5
15/04/22 15:11:56 INFO mapred.JobClient: Virtual memory (bytes) snapshot=747270144
15/04/22 15:11:56 INFO mapred.JobClient: Map output records=5