hadoop开发setjar方法

 

屏蔽

//job.setJar("/Users/sumeng/IdeaProjects/cloudHadoop/out/artifacts/cloudHadoop_jar/cloudHadoop.jar");
job.setJarByClass(WordCount.class);

报错：

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class WordCount$TokenizerMapper not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class WordCount$TokenizerMapper not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)
    ... 8 more

屏蔽

job.setJar("/Users/sumeng/IdeaProjects/cloudHadoop/out/artifacts/cloudHadoop_jar/cloudHadoop.jar");
//job.setJarByClass(WordCount.class);

不会报错

 

//通过Configuration对象获取job对象，该job对象会组织所有的该mapreduce的所有各种组件
Job job = Job.getInstance(conf);
//指定jar包所在路径，本地模式需要这样指定，如果不是本地，则使用setJarByClass指定所在class文件即可
//job.setJarByClass("wordcountJar/wordcount.jar")
job.setJar("wordcountJar/wordcount.jar");
--------------------- 
作者：夜下探戈 
来源：CSDN 
原文：https://blog.csdn.net/dudefu011/article/details/79586191 
版权声明：本文为博主原创文章，转载请附上博文链接！

 

可以确定的是，

情况一：如果采用打成jar包，在hadoop集群中通过hadoop jar命令运行，则只需要写job.setJarByClass

情况二：本地idea开发mapreduce程序提交到hadoop集群执行，参考https://blog.csdn.net/dream_an/article/details/84342770

https://www.jianshu.com/p/2c9c22130225也是把jar包设置到了本机

https://blog.csdn.net/shirukai/article/details/81021872讲解了打成jar包的过程

 

还是不是很明白setjar 与 setJarbyClass 的区别和意义

https://www.codetd.com/article/664330

https://my.oschina.net/zhzhenqin/blog/163158

转载于:https://www.cnblogs.com/SuMeng/p/10259781.html