最近在搞一个大数据的项目,用的是mapreduce,但是想把mapreduce集成到项目中,通过点击按钮的方式来触发mapreduce。
具体方法如下:
1 先把mapreduce类写好,由于项目的原因,只用了map,没用到reduce过程
//mapreduce类
public static class _map extends Mapper<LongWritable, Text, NullWritable, NullWritable> {
@Override
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
//map过程
}
}
public static boolean main(String[] args) throws IOException, InterruptedException, ClassNotFoundException
{
Configuration conf = new Configuration();
conf.set("参数1", args[1]);
conf.set("参数2", args[2]);
conf.set("参数3", args[3]);
//把需要的参数放到这里即可
conf.set("mapreduce.job.jar","/home/files/AISA.jar");
conf.set("fs.defaultFS", "hdfs://h1:9000/");
conf.set("mapreduce.framework.name", "yarn");
conf.set("yarn.resourcemanager.hostname", "h1");
Job job = new Job(conf, "_mr");
job.setJarByClass(_mr.class);
job.setMapperClass(_mr._map.class);
job.setNumReduceTasks(0);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(NullOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
boolean iscomplete = job.waitForCompletion(true);
return iscomplete;
}
2 现在要在项目中调用 这个类
_mr mf = new _mr(); //上面的mapreduce类
String[] arg = {"参数1","参数2","参数3","..."}; //参数按照类里定义的顺序排列好
mf.main(arg); //执行
这样就可以调用mapreduce了,是不是很简单