在eclipse将mapreduce程序运行在hadoop集群中

最新推荐文章于 2021-12-01 09:02:42 发布

sunggff

最新推荐文章于 2021-12-01 09:02:42 发布

阅读量735

点赞数 1

分类专栏： hadoop集群搭建问题集锦文章标签： Hadoop mapreduce

本文链接：https://blog.csdn.net/sunggff/article/details/73784904

版权

hadoop集群搭建问题集锦专栏收录该内容

2 篇文章 0 订阅

订阅专栏

1.将集群配置文件：core-site.xml，hdfs-site.xml，mapred-site.xml，yarn-site,xml拷贝至项目的src目录下

2.将本项目打成JAR包：xxx.jar

3.在mapreduce程序main方法中添加代码：job.setJar("/xxx.jar")//jar包所在文件路径

4.其他套路性代码编写好即可运行。

5.运行时所遇错误

一不小心忘加hadoop-hdfs-2.7.3.jar，报错：

Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:357)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:500)
at com.hadoop.hive.mr.ClickStreamVisit.main(ClickStreamVisit.java:127)

若使用的是job.setJarByClass(xxx.class)，运行后报错：

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.hadoop.hive.mr.ClickStreamVisit$ClickStreamVisitMapper not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.hive.mr.ClickStreamVisit$ClickStreamVisitMapper not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
... 8 more