学习hadoop时遇到的问题以及解决方案
- ClassCastException: class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$Text
- .NoClassDefFoundError: org/apache/hadoop/yarn/util/Apps
- expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable
- .UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
- FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
- Not a host:port pair: PBUF
- Unrecognized VM option 'UseCompressedOops'
版本:hadoop2.4.1
平台 Java
windows 1.8
linux 1.7
ClassCastException: class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$Text
平台:linux
bug
20/08/24 03:14:12 INFO mapreduce.Job: Task Id : attempt_1598263751336_0001_m_000000_0, Status : FAILED
Error: java.lang.ClassCastException: class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider T e x t a t j a v a . l a n g . C l a s s . a s S u b c l a s s ( C l a s s . j a v a : 3165 ) a t o r g . a p a c h e . h a d o o p . m a p r e d . J o b C o n f . g e t O u t p u t K e y C o m p a r a t o r ( J o b C o n f . j a v a : 884 ) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p T a s k Text at java.lang.Class.asSubclass(Class.java:3165) at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:884) at org.apache.hadoop.mapred.MapTask Textatjava.lang.Class.asSubclass(Class.java:3165)atorg.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:884)atorg.apache.hadoop.mapred.MapTaskMapOutputBuffer.init(MapTask.java:981)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.access 100 ( M a p T a s k . j a v a : 80 ) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p T a s k 100(MapTask.java:80) at org.apache.hadoop.mapred.MapTask 100(MapTask.java:80)atorg.apache.hadoop.mapred.MapTaskNewOutputCollector.(MapTask.java:675)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
20/08/24 03:14:13 INFO mapreduce.Job: map 0% reduce 0%
20/08/24 03:14:15 INFO mapreduce.Job: Task Id : attempt_1598263751336_0001_m_000000_1, Status : FAILED
Error: java.lang.ClassCastException: class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider T e x t a t j a v a . l a n g . C l a s s . a s S u b c l a s s ( C l a s s . j a v a : 3165 ) a t o r g . a p a c h e . h a d o o p . m a p r e d . J o b C o n f . g e t O u t p u t K e y C o m p a r a t o r ( J o b C o n f . j a v a : 884 ) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p T a s k Text at java.lang.Class.asSubclass(Class.java:3165) at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:884) at org.apache.hadoop.mapred.MapTask Textatjava.lang.Class.asSubclass(Class.java:3165)atorg.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:884)atorg.apache.hadoop.mapred.MapTaskMapOutputBuffer.init(MapTask.java:981)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.access 100 ( M a p T a s k . j a v a : 80 ) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p T a s k 100(MapTask.java:80) at org.apache.hadoop.mapred.MapTask 100(MapTask.java:80)atorg.apache.hadoop.mapred.MapTaskNewOutputCollector.(MapTask.java:675)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
20/08/24 03:14:18 INFO mapreduce.Job: Task Id : attempt_1598263751336_0001_m_000000_2, Status : FAILED
Error: java.lang.ClassCastException: class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider T e x t a t j a v a . l a n g . C l a s s . a s S u b c l a s s ( C l a s s . j a v a : 3165 ) a t o r g . a p a c h e . h a d o o p . m a p r e d . J o b C o n f . g e t O u t p u t K e y C o m p a r a t o r ( J o b C o n f . j a v a : 884 ) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p T a s k Text at java.lang.Class.asSubclass(Class.java:3165) at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:884) at org.apache.hadoop.mapred.MapTask Textatjava.lang.Class.asSubclass(Class.java:3165)atorg.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:884)atorg.apache.hadoop.mapred.MapTaskMapOutputBuffer.init(MapTask.java:981)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.access 100 ( M a p T a s k . j a v a : 80 ) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p T a s k 100(MapTask.java:80) at org.apache.hadoop.mapred.MapTask 100(MapTask.java:80)atorg.apache.hadoop.mapred.MapTaskNewOutputCollector.(MapTask.java:675)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
20/08/24 03:14:24 INFO mapreduce.Job: map 100% reduce 100%
20/08/24 03:14:24 INFO mapreduce.Job: Job job_1598263751336_0001 failed with state FAILED due to: Task failed task_1598263751336_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
20/08/24 03:14:24 INFO mapreduce.Job: Counters: 9
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=10038
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=10038
Total vcore-seconds taken by all map tasks=10038
Total megabyte-seconds taken by all map tasks=10278912
解决方法
引用了错误的Text,
应当是
import org.apache.hadoop.io.Text;
.NoClassDefFoundError: org/apache/hadoop/yarn/util/Apps
Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/util/Apps
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader 1. r u n ( U R L C l a s s L o a d e r . j a v a : 362 ) a t j a v a . s e c u r i t y . A c c e s s C o n t r o l l e r . d o P r i v i l e g e d (