学习hadoop时遇到的问题以及解决方案

最新推荐文章于 2024-05-14 20:16:40 发布

BinaryJack

最新推荐文章于 2024-05-14 20:16:40 发布

阅读量479

点赞数

分类专栏： hadoop整理总结文章标签： hadoop mapreduce hdfs

本文链接：https://blog.csdn.net/qq_40473204/article/details/108218416

版权

学习hadoop时遇到的问题以及解决方案

ClassCastException: class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$Text
- bug
- 解决方法
.NoClassDefFoundError: org/apache/hadoop/yarn/util/Apps
expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable
- bug
- 解决方法
.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
- bug
- 解决方法
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
- bug
- 解决方法
Not a host:port pair: PBUF
- bug
- 解决方法
Unrecognized VM option 'UseCompressedOops'
- bug
- 解决方法

版本：hadoop2.4.1

平台 Java
windows 1.8
linux 1.7

ClassCastException: class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$Text

平台：linux

bug

20/08/24 03:14:12 INFO mapreduce.Job: Task Id : attempt_1598263751336_0001_m_000000_0, Status : FAILED
Error: java.lang.ClassCastException: class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider $T e x t a t j a v a . l a n g . C l a s s . a s S u b c l a s s (C l a s s . j a v a : 3165) a t o r g . a p a c h e . h a d o o p . m a p r e d . J o b C o n f . g e t O u t p u t K e y C o m p a r a t o r (J o b C o n f . j a v a : 884) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p T a s k$ MapOutputBuffer.init(MapTask.java:981)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.access $100 (M a p T a s k . j a v a : 80) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p T a s k$ NewOutputCollector.(MapTask.java:675)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

20/08/24 03:14:13 INFO mapreduce.Job: map 0% reduce 0%
20/08/24 03:14:15 INFO mapreduce.Job: Task Id : attempt_1598263751336_0001_m_000000_1, Status : FAILED
Error: java.lang.ClassCastException: class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider $T e x t a t j a v a . l a n g . C l a s s . a s S u b c l a s s (C l a s s . j a v a : 3165) a t o r g . a p a c h e . h a d o o p . m a p r e d . J o b C o n f . g e t O u t p u t K e y C o m p a r a t o r (J o b C o n f . j a v a : 884) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p T a s k$ MapOutputBuffer.init(MapTask.java:981)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.access $100 (M a p T a s k . j a v a : 80) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p T a s k$ NewOutputCollector.(MapTask.java:675)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

20/08/24 03:14:18 INFO mapreduce.Job: Task Id : attempt_1598263751336_0001_m_000000_2, Status : FAILED
Error: java.lang.ClassCastException: class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider $T e x t a t j a v a . l a n g . C l a s s . a s S u b c l a s s (C l a s s . j a v a : 3165) a t o r g . a p a c h e . h a d o o p . m a p r e d . J o b C o n f . g e t O u t p u t K e y C o m p a r a t o r (J o b C o n f . j a v a : 884) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p T a s k$ MapOutputBuffer.init(MapTask.java:981)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.access $100 (M a p T a s k . j a v a : 80) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p T a s k$ NewOutputCollector.(MapTask.java:675)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

20/08/24 03:14:24 INFO mapreduce.Job: map 100% reduce 100%
20/08/24 03:14:24 INFO mapreduce.Job: Job job_1598263751336_0001 failed with state FAILED due to: Task failed task_1598263751336_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

20/08/24 03:14:24 INFO mapreduce.Job: Counters: 9
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=10038
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=10038
Total vcore-seconds taken by all map tasks=10038
Total megabyte-seconds taken by all map tasks=10278912

解决方法

引用了错误的Text，
应当是
import org.apache.hadoop.io.Text;

.NoClassDefFoundError: org/apache/hadoop/yarn/util/Apps

Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/util/Apps
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader

最低0.47元/天解锁文章

BinaryJack

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
学习hadoop时遇到的问题以及解决方案

学习hadoop时遇到的问题以及解决方案ClassCastException: class com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$Textbug解决方法.NoClassDefFoundError: org/apache/hadoop/yarn/util/Appsexpected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritablebu
复制链接

扫一扫