hadoop2.3异常问题

最新推荐文章于 2024-03-28 17:00:11 发布

Xlucas

最新推荐文章于 2024-03-28 17:00:11 发布

阅读量1.5k

点赞数 1

分类专栏： hadoop 文章标签： hadoop jar

本文链接：https://blog.csdn.net/paicmis/article/details/72739015

版权

hadoop 专栏收录该内容

42 篇文章 3 订阅

订阅专栏

为了验证druid的在hadoop上的overlord，决定安装hadoop2.3的版本，
在安装hadoop2.3的过程中发现一些环境问题，在运行hadoop自身带的例子就遇到了如下的错误。从错误信息来看
这个就是没有配置classpath，本想直接在yarn-site.xml里面配置yarn.application.classpath配置jar包的路径，

<name>yarn.application.classpath</name>
        <value>$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,
               $HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
               $HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
               $YARN_HOME/share/hadoop/yarn/*,$YARN_HOME/share/hadoop/yarn/lib/*
        </value>

运行结果发现了一个新的问题

[hadoop@SZB-L0038787 2.4.0]$ hadoop jar hadoop-mapreduce-examples-2.3.0.jar wordcount /input/passwd /output
WARNING: Use "yarn jar" to launch YARN applications.
java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/partition/InputSampler$Sampler
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2615)
        at java.lang.Class.getMethod0(Class.java:2856)
        at java.lang.Class.getMethod(Class.java:1668)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.<init>(ProgramDriver.java:60)
        at org.apache.hadoop.util.ProgramDriver.addClass(ProgramDriver.java:103)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:57)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.lib.partition.InputSampler$Sampler
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 12 more

以为这么配置就可以解决这个问题，但是在实际运行过程中又出现了这么一个错误。
查找这个问题的错误竟然是hadoop本身有这么一个问题，但是问题还是需要解决，要么重新编译要么其他方法
官网issues
https://issues.apache.org/jira/browse/YARN-1298
https://issues.apache.org/jira/browse/MAPREDUCE-5655

[hadoop@SZB-L0038787 2.4.0]$ hadoop jar hadoop-mapreduce-examples-2.3.0.jar wordcount /input/passwd /output1
WARNING: Use "yarn jar" to launch YARN applications.
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
17/05/25 08:20:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/25 08:20:49 INFO client.RMProxy: Connecting to ResourceManager at /10.20.23.82:8032
17/05/25 08:20:50 INFO input.FileInputFormat: Total input paths to process : 1
17/05/25 08:20:50 INFO mapreduce.JobSubmitter: number of splits:1
17/05/25 08:20:50 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1495670009175_0002
17/05/25 08:20:51 INFO impl.YarnClientImpl: Submitted application application_1495670009175_0002
17/05/25 08:20:51 INFO mapreduce.Job: The url to track the job: http://SZB-L0038787:8088/proxy/application_1495670009175_0002/
17/05/25 08:20:51 INFO mapreduce.Job: Running job: job_1495670009175_0002
17/05/25 08:20:56 INFO mapreduce.Job: Job job_1495670009175_0002 running in uber mode : false
17/05/25 08:20:56 INFO mapreduce.Job:  map 0% reduce 0%
17/05/25 08:20:56 INFO mapreduce.Job: Job job_1495670009175_0002 failed with state FAILED due to: Application application_1495670009175_0002 failed 2 times due to AM Container for appattempt_1495670009175_0002_000002 exited with  exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: 
org.apache.hadoop.util.Shell$ExitCodeException: 
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
        at org.apache.hadoop.util.Shell.run(Shell.java:418)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
17/05/25 08:20:56 INFO mapreduce.Job: Counters: 0

最终的解决方法是
在hadoop-env.sh增加下面一段，后面的路径是你hadoop依赖jar包的路径，后面的*必须填写

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/hadoop/druid-0.9.2/hadoop-dependencies/hadoop-client/hadoop/*

再次运行成功了

[hadoop@SZB-L0038787 2.4.0]$ hadoop jar hadoop-mapreduce-examples-2.3.0.jar wordcount /input/passwd /output1
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/hadoop-2.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/druid-0.9.2/hadoop-dependencies/hadoop-client/hadoop/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /data/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
17/05/25 08:45:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/25 08:45:25 INFO client.RMProxy: Connecting to ResourceManager at /10.20.23.82:8032
17/05/25 08:45:27 INFO input.FileInputFormat: Total input paths to process : 1
17/05/25 08:45:27 INFO mapreduce.JobSubmitter: number of splits:1
17/05/25 08:45:27 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1495673058088_0001
17/05/25 08:45:28 INFO impl.YarnClientImpl: Submitted application application_1495673058088_0001
17/05/25 08:45:28 INFO mapreduce.Job: The url to track the job: http://SZB-L0038787:8088/proxy/application_1495673058088_0001/
17/05/25 08:45:28 INFO mapreduce.Job: Running job: job_1495673058088_0001
17/05/25 08:45:36 INFO mapreduce.Job: Job job_1495673058088_0001 running in uber mode : false
17/05/25 08:45:36 INFO mapreduce.Job:  map 0% reduce 0%
17/05/25 08:45:43 INFO mapreduce.Job:  map 100% reduce 0%
17/05/25 08:45:49 INFO mapreduce.Job:  map 100% reduce 100%
17/05/25 08:45:49 INFO mapreduce.Job: Job job_1495673058088_0001 completed successfully
17/05/25 08:45:50 INFO mapreduce.Job: Counters: 49