1. 程式人生 > >[hadoop]MapReduce例項之好友推薦(六)

[hadoop]MapReduce例項之好友推薦(六)

一、定義好友檔案qq

hadoop hello
hdfs world
tom cat
cat dog
hello world
hello hdfs

hadoop好友hello,hdfs好友world...依次類推。那麼hadoop和world有共同的好友hello,所以hadoop和world可能具有好友關係,world就是hadoop的推薦好友。計算出qq檔案內符合上述條件的推薦好友!

實現思路:

在Mapper中,先把qq檔案每行好友,互相轉換(互為好友),結果如下:

hadoop hello
hello hadoop
hdfs world
world hdfs
tom cat
cat tom
cat dog
dog cat
hello world
world hello
hello hdfs
hdfs hello
然後,Shuffling進行洗牌,把相同key的整理在一起,結果如下:
洗牌結果:
hadoop hello

hello hadoop
hello world
hello hdfs

hdfs world
hdfs hello

tom cat

cat tom
cat dog

dog cat
最後,Reducing進行笛卡爾乘積計算,完畢

下面是相關程式碼

Mapper程式碼,QqMapper.java

package com.all58.qq;

import java.io.IOException;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class QqMapper extends Mapper<LongWritable, Text, Text, Text> {
	
	/*
	 *  Mapping結果:
	 *  hadoop hello
	 *  hello hadoop
		hdfs world
		world hdfs
		tom cat
		cat tom
		cat dog
		dog cat
		hello world
		world hello
		hello hdfs
		hdfs hello
	 */
	@Override
	protected void map(LongWritable key, Text value, Context context)
			throws IOException, InterruptedException {
		String line = value.toString();
		String[] sp = line.split("\\s+");
		context.write(new Text(sp[0]), new Text(sp[1]));
		context.write(new Text(sp[1]), new Text(sp[0]));
	}
}

Reducing程式碼,QqReducer.java
package com.all58.qq;

import java.io.IOException;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;

import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;


public class QqReducer extends Reducer<Text, Text, Text, Text> {
	
	/*
		洗牌結果:
		hadoop hello
		
		hello hadoop
		hello world
		hello hdfs
		
		hdfs world
		hdfs hello
		
		tom cat
		
		cat tom
		cat dog
		
		dog cat
	 */
	@Override
	protected void reduce(Text key, Iterable<Text> iter, Context context)
			throws IOException, InterruptedException {
		Set<String> set = new HashSet<>();
		for (Text text : iter) {
			set.add(text.toString());
		}
		if (set.size() > 1) {
			for (Iterator i = set.iterator(); i.hasNext();) {
				String name = (String) i.next();
				for (Iterator j = set.iterator(); j.hasNext();) {
					String other = (String) j.next();
					if (!name.equals(other)) {
						context.write(new Text(name), new Text(other));
					}
				}
			}
		}
	}
}
最後計算結果:
tom	dog
dog	tom
hello	world
world	hello
hdfs	world
hdfs	hadoop
world	hdfs
world	hadoop
hadoop	hdfs
hadoop	world
hello	hdfs
hdfs	hello