mapReduce求文本偏移量和每行数据的和
一.文本
二.建三个java类
Map Count Reduce
三.写Map
***1、实例一个实体类,继承Mapper<输入放入key的类型,输入的value的类型,输出放入key的类型,输出的value的类型>***
public class WordCountMap extends Mapper<LongWritable,Text,LongWritable,LongWritable> {
***2、重写map(LongWritablekey,Textvalue,Contextcontext)
key每行行首的偏移量
value每一行的数据
context上下文对象***
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
***3、实现自己业务逻辑的代码
将数据进行拆分,并进行逐一输出。***
//转化value
String datas = value.toString();
//截取
String[] split = datas.split(" ");
//遍历
for (String data : split) {
if ("".equals(data.trim())){
continue;
}
context.write(key,new LongWritable(Long.parseLong(data.trim())));
} }}
四.写Reduce
***1、实例一个实体类,继承Reducer<输入放入key的类型,输入的value的类型,输出放入key的类型,输出的value的类型>***
public class WordCountReduce extends Reducer<LongWritable,LongWritable,Text,LongWritable> {
***2、重写reduce(Textkey,Iterablevalues,Contextcontext)
key:每一个唯一的数据(游戏中的图形)
values:每一个图形,1的标记(1的list)
context:上下文对象***
@Override
protected void reduce(LongWritable key, Iterable<LongWritable> values, Context context) throws IOException, InterruptedException {
long sum=0;
for (LongWritable value : values) {
sum+=value.get();
}
context.write(new Text("偏移量"+key),new LongWritable(sum));
}
}
五.编写drive
public class WordCountDrive extends Configured implements Tool {
@Override
public int run(String[] strings) throws Exception {
Configuration conf=new Configuration();
***1、实例一个job***
Job job = Job.getInstance(conf, "Sam02");
FileSystem fileSystem = FileSystem.get(conf);
***2、使用job设置读物数据(包括数据的路径)***
job.setInputFormatClass(TextInputFormat.class);
TextInputFormat.addInputPath(job,new Path("C:\\SumData.txt"));
***3.使用job设置数据的输出路径***
job.setOutputFormatClass(TextOutputFormat.class);
TextOutputFormat.setOutputPath(job,new Path("D:\\cs\\ddd01"));
***4、使用job设置MAP类(map输入和输出的类型)***
job.setMapperClass(WordCountMap.class);
job.setMapOutputValueClass(LongWritable.class);
job.setMapOutputKeyClass(LongWritable.class);
***5 、使用job设置Reduce类(Reduce输入和输出的类型)***
job.setReducerClass(WordCountReduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(LongWritable.class);
***6.返回执行状态编号***
return job.waitForCompletion(true)? 0 : 1 ;
}
public static void main(String[] args) throws Exception {
ToolRunner.run(new WordCountDrive(),args);
}
}
六.打开
1.见到如下画面说明成功
七.注意
1.输入和输出路径请自行更改
2.输出路径不能覆盖,如果覆盖就会报错