java调用mapreduce,从一个简单的java程序调用mapreduce工作

哦，请不要用runJar，Java API非常好。

看看如何从正常的代码开始工作：

// create a configuration

Configuration conf = new Configuration();

// create a new job based on the configuration

Job job = new Job(conf);

// here you have to put your mapper class

job.setMapperClass(Mapper.class);

// here you have to put your reducer class

job.setReducerClass(Reducer.class);

// here you have to set the jar which is containing your

// map/reduce class, so you can use the mapper class

job.setJarByClass(Mapper.class);

// key/value of your reducer output

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(Text.class);

// this is setting the format of your input, can be TextInputFormat

job.setInputFormatClass(SequenceFileInputFormat.class);

// same with output

job.setOutputFormatClass(TextOutputFormat.class);

// here you can set the path of your input

SequenceFileInputFormat.addInputPath(job, new Path("files/toMap/"));

// this deletes possible output paths to prevent job failures

FileSystem fs = FileSystem.get(conf);

Path out = new Path("files/out/processed/");

fs.delete(out, true);

// finally set the empty out path

TextOutputFormat.setOutputPath(job, out);

// this waits until the job completes and prints debug out to STDOUT or whatever

// has been configured in your log4j properties.

job.waitForCompletion(true);

如果您正在使用外部群集，则必须通过以下方式对您的配置进行以下信息：

// this should be like defined in your mapred-site.xml

conf.set("mapred.job.tracker", "jobtracker.com:50001");

// like defined in hdfs-site.xml

conf.set("fs.default.name", "hdfs://namenode.com:9000");

当您的应用程序容器类路径中有hadoop-core.jar时，这应该是没有问题的。

但是我认为你应该把一些进度指标放在你的网页上，因为完成一个hadoop工作可能需要几分钟到几个小时);

对于YARN(> Hadoop 2)

对于YARN，需要设置以下配置。

// this should be like defined in your yarn-site.xml

conf.set("yarn.resourcemanager.address", "yarn-manager.com:50001");

// framework is now "yarn", should be defined like this in mapred-site.xm

conf.set("mapreduce.framework.name", "yarn");

// like defined in hdfs-site.xml

conf.set("fs.default.name", "hdfs://namenode.com:9000");