oozie配置sharelib,java.lang.NoClassDefFoundError: org/apache/pig/Main

最新推荐文章于 2021-07-30 16:50:14 发布

tucailing

最新推荐文章于 2021-07-30 16:50:14 发布

阅读量2.6k

点赞数

分类专栏： hadoop oozie 文章标签： hadoop oozie

本文链接：https://blog.csdn.net/tucailing/article/details/9245791

版权

hadoop oozie 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

作为笔记：

sharelib的配置

在用oozie调用pig时，用到sharelib，当时是直接运行的，然后就报错了.

2013-07-03 20:44:48,504 WARN PigActionExecutor:542 - USER[root] GROUP[-] TOKEN[] APP[pig-wf] JOB[0000004-130703183035408-oozie-root-W] ACTION[0000004-130703183035408-oozie-root-W@pig-node] Launcher exception: org/apache/pig/Main
java.lang.NoClassDefFoundError: org/apache/pig/Main
at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:323)
at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:218)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:467)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: org.apache.pig.Main
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 17 more

最后在网上各种扒问题，发现是需要sharelib，action执行时，会对sharelib进行依赖，而sharelib要放到hdfs上。
oozie提供一个sharelib：oozie-sharelib-**-incubating.tar.gz。解压即可。

然后需要把sharelib放到hdfs上，hdfs的sharelib路径发现也是个问题。

这个需要在oozie安装目录下conf/oozie-default.xml中查找：

<property>
<name>oozie.service.WorkflowAppService.system.libpath</name>
<value>/user/${user.name}/share/lib</value>
<description>
System library path to use for workflow applications.
This path is added to workflow application if their job properties sets
the property 'oozie.use.system.libpath' to true.
</description>
</property>

“/user/${user.name}/share/lib”就是我们需要放sharelib的路径。