一、定义好友文件qq
hadoop hello
hdfs world
tom cat
cat dog
hello world
hello hdfs
hadoop好友hello,hdfs好友world...依次类推。那么hadoop和world有共同的好友hello,所以hadoop和world可能具有好友关系,world就是hadoop的推荐好友。计算出qq文件内符合上述条件的推荐好友!
实现思路:
在Mapper中,先把qq文件每行好友,互相转换(互为好友),结果如下:
hadoop hello
hello hadoop
hdfs world
world hdfs
tom cat
cat tom
cat dog
dog cat
hello world
world hello
hello hdfs
hdfs hello
然后,Shuffling进行洗牌,把相同key的整理在一起,结果如下:
洗牌结果:
hadoop hello
hello hadoop
hello world
hello hdfs
hdfs world
hdfs hello
tom cat
cat tom
cat dog
dog cat
最后,Reducing进行笛卡尔乘积计算,完毕
下面是相关代码
Mapper代码,QqMapper.java
package com.all58.qq;
import java.io.IOException;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
public class QqMapper extends Mapper<LongWritable, Text, Text, Text> {
/*
* Mapping结果:
* hadoop hello
* hello hadoop
hdfs world
world hdfs
tom cat
cat tom
cat dog
dog cat
hello world
world hello
hello hdfs
hdfs hello
*/
@Override
protected void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
String[] sp = line.split("\\s+");
context.write(new Text(sp[0]), new Text(sp[1]));
context.write(new Text(sp[1]), new Text(sp[0]));
}
}
Reducing代码,QqReducer.java
package com.all58.qq;
import java.io.IOException;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
public class QqReducer extends Reducer<Text, Text, Text, Text> {
/*
洗牌结果:
hadoop hello
hello hadoop
hello world
hello hdfs
hdfs world
hdfs hello
tom cat
cat tom
cat dog
dog cat
*/
@Override
protected void reduce(Text key, Iterable<Text> iter, Context context)
throws IOException, InterruptedException {
Set<String> set = new HashSet<>();
for (Text text : iter) {
set.add(text.toString());
}
if (set.size() > 1) {
for (Iterator i = set.iterator(); i.hasNext();) {
String name = (String) i.next();
for (Iterator j = set.iterator(); j.hasNext();) {
String other = (String) j.next();
if (!name.equals(other)) {
context.write(new Text(name), new Text(other));
}
}
}
}
}
}
最后计算结果:
tom dog
dog tom
hello world
world hello
hdfs world
hdfs hadoop
world hdfs
world hadoop
hadoop hdfs
hadoop world
hello hdfs
hdfs hello