使用intellij搭建运行MapReduce环境
说明:因为本人是在Windows中的idea中编写的MapReduce程序,每次编写玩程序后都需要打成jar包发布到集群中去检查程序的正确性,个人感觉比较麻烦。
一、实验环境
物理机:Windows10
idea:intellij 2017
二、具体需求
在idea中搭建一个用于运行MapReduce程序的环境,这样就可以在idea中运行或调试MapReduce程序的正确性,不再需要每次都打包到集群中去运行测试程序的正确性,方便了开发和调试。
三、环境搭建及测试
1、新建一个maven项目(这里不详述)
2、添加hadoop依赖包:一般需要导入common包下的hadoop-common.jar hadoop-nfs.jar、common小的lib包、mapreduce包、yarn包、hdfs包。(如果直接将所有包一起导入,运行MapReduce程序的时候可能会报错:Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper(我测试中是这样的))
3、编写wordcount程序进行测试
mapper:
package com.wp.demo.mapper;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import java.io.IOException;
public class AnasislyTypeMapper extends Mapper<LongWritable,Text,Text,IntWritable> {
private Text k = new Text();
private IntWritable v = new IntWritable();
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] fields = value.toString().split(" ");
for (String field : fields) {
k.set(field);
v.set(1);
context.write(k,v);
}
}
}
reducer:
package reducer;
import org.apache.hadoop.hdfs.TestExtendedAcls;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
import java.io.IOException;
public class AnasislyTypeReducer extends Reducer<Text,IntWritable,Text,IntWritable> {
@Override
protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int count = 0;
IntWritable v = new IntWritable();
for (IntWritable value : values) {
count = count + value.get();
}
v.set(count);
context.write(key,v);
}
}
job:(需要在指定的位置创建一个用于wordcount的文件)
package tool;
import com.wp.demo.mapper.AnasislyTypeMapper;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import reducer.AnasislyTypeReducer;
public class AnasislyTypeTool {
public static void main(String[] args) throws Exception {
args = new String[]{"E:\\input\\demoinput\\hello1.txt","E:\\input\\demooutput\\out1\\"};
Configuration conf = new Configuration();
//获取job对象
Job job = Job.getInstance(conf);
//设置jar包
job.setJarByClass(AnasislyTypeTool.class);
//关联mapper和reducer
job.setMapperClass(AnasislyTypeMapper.class);
job.setReducerClass(AnasislyTypeReducer.class);
//设置map输出数据类型
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
//设置最终输出数据类型kv
job.setOutputKeyClass(Text .class);
job.setOutputValueClass(IntWritable.class);
//设置输入输出文件路径
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
//提交到yarn集群
job.waitForCompletion(true);
}
}
**maven依赖如下:**虽然有些依赖不用导入,这里不进行去除
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.9.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.9.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-common</artifactId>
<version>2.9.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-jobclient</artifactId>
<version>2.9.1</version>
</dependency>
</dependencies>
运行以上wordcount程序**
四、运行结果
控制台打印结果:
将程序打包以便在hadoop集群中运行
1.
2.配置打包信息
点击OK以后,选择build artifact进行打包!
打包完毕之后,在对应的目录中就可以看到打好的jar包(默认是在out目录中)