[hadoop]MapReduce例項之好友推薦(六)
阿新 • • 發佈:2019-01-28
一、定義好友檔案qq
hadoop hello
hdfs world
tom cat
cat dog
hello world
hello hdfs
hadoop好友hello,hdfs好友world...依次類推。那麼hadoop和world有共同的好友hello,所以hadoop和world可能具有好友關係,world就是hadoop的推薦好友。計算出qq檔案內符合上述條件的推薦好友!
實現思路:
在Mapper中,先把qq檔案每行好友,互相轉換(互為好友),結果如下:
然後,Shuffling進行洗牌,把相同key的整理在一起,結果如下:hadoop hello hello hadoop hdfs world world hdfs tom cat cat tom cat dog dog cat hello world world hello hello hdfs hdfs hello
洗牌結果:
hadoop hello
hello hadoop
hello world
hello hdfs
hdfs world
hdfs hello
tom cat
cat tom
cat dog
dog cat
最後,Reducing進行笛卡爾乘積計算,完畢
下面是相關程式碼
Mapper程式碼,QqMapper.java
package com.all58.qq; import java.io.IOException; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class QqMapper extends Mapper<LongWritable, Text, Text, Text> { /* * Mapping結果: * hadoop hello * hello hadoop hdfs world world hdfs tom cat cat tom cat dog dog cat hello world world hello hello hdfs hdfs hello */ @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); String[] sp = line.split("\\s+"); context.write(new Text(sp[0]), new Text(sp[1])); context.write(new Text(sp[1]), new Text(sp[0])); } }
Reducing程式碼,QqReducer.java
最後計算結果:package com.all58.qq; import java.io.IOException; import java.util.HashMap; import java.util.HashSet; import java.util.Iterator; import java.util.Map; import java.util.Set; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer; public class QqReducer extends Reducer<Text, Text, Text, Text> { /* 洗牌結果: hadoop hello hello hadoop hello world hello hdfs hdfs world hdfs hello tom cat cat tom cat dog dog cat */ @Override protected void reduce(Text key, Iterable<Text> iter, Context context) throws IOException, InterruptedException { Set<String> set = new HashSet<>(); for (Text text : iter) { set.add(text.toString()); } if (set.size() > 1) { for (Iterator i = set.iterator(); i.hasNext();) { String name = (String) i.next(); for (Iterator j = set.iterator(); j.hasNext();) { String other = (String) j.next(); if (!name.equals(other)) { context.write(new Text(name), new Text(other)); } } } } } }
tom dog
dog tom
hello world
world hello
hdfs world
hdfs hadoop
world hdfs
world hadoop
hadoop hdfs
hadoop world
hello hdfs
hdfs hello