java多个mapreduce_java – 在hadoop中运行多个MapReduce作业

我想运行一系列map reduce工作,所以最简单的解决方案似乎是jobcontroller.说我有两个工作,job1和job2.我想在job1之后运行job2.嗯,它遇到了一些问题.经过几个小时的调试后,我将代码缩小到以下几行：

JobConf jobConf1 = new JobConf();

JobConf jobConf2 = new JobConf();

System.out.println("*** Point 1");

Job job1 = new Job(jobConf1);

System.out.println("*** Point 2");

Job job2 = new Job(jobConf2);

System.out.println("*** Point 3");

我在运行代码时不断获得此输出：

*** Point 1

10/12/06 17:19:30 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=

*** Point 2

10/12/06 17:19:30 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

*** Point 3

我想我的问题是以某种方式与“无法初始化JMV ……”相关联.那是什么？以及如何实例化多个作业,以便将它们传递给JobController.

当我在初始化第二个作业之前添加job1.waitForTheCompletion(true)时,它给了我这个错误：

10/12/07 11:28:21 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/home/workspace/WikipediaSearch/__TEMP1

at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)

at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)

at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)

at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)

at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)

at ch.ethz.nis.query.HadoopQuery.run(HadoopQuery.java:353)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

at ch.ethz.nis.query.HadoopQuery.main(HadoopQuery.java:308)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

__Temp1是第一个作业的输出文件夹,我希望它是第二个作业的输入.即使我的代码中有这个waitForCompletion行,它仍然抱怨这条路径不存在.