MapReduce找共同好友
阿新 • • 發佈：2018-12-01
用到的資料建立一個txt檔案放進去就可以啦
A:B,D,E,H,I,O
B:A,C,E,K
C:F,A,D,I
D:A,E,F,L
E:B,C,D,M,L
F:B,C,D,E,O,M
G:Q,W,A,C,E,O
H:A,C,E,D,O
I:A,O
J:B,P
K:A,C,D
L:D,E,F
M:E,F,G
O:A,H,I,J
我是用了兩個mapreduce寫的
第一個mapreduce
map端： 先根據":" 切分 然後在根據"," 切分 String person= split[0]; // 當做 人 String[] friends= split[1].split(","); // 當做好友 static class FriendMap01 extends Mapper<LongWritable, Text, Text, Text>{ @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String[] split = value.toString().split(":"); //好友作為key 人 作為vlaue String person= split[0]; String[] friends= split[1].split(","); for (String str : friends) { context.write(new Text(str), new Text(person)); } } }
reduce端： static class FriendReducer01 extends Reducer<Text, Text, Text, Text>{ @Override protected void reduce(Text friend, Iterable<Text> persons, Context context) throws IOException, InterruptedException { //定義一個Buffer資料做快取 StringBuffer sb= new StringBuffer(); for (Text person : persons) { sb.append(person).append(","); } context.write(friend, new Text(sb.toString())); } }
Driver端： public static void main(String[] args) throws Exception, Exception { //1·建立Configuration Configuration conf = new Configuration(); Job job = Job.getInstance(conf); //2·設定載入路徑 job.setJarByClass(Friends01.class); //3·設定map和reduce job.setMapperClass(FriendMap01.class); job.setReducerClass(FriendReducer01.class); //4·設定map輸出 job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(Text.class); //5·設定reduce輸出 job.setOutputKeyClass(Text.class); job.setOutputValueClass(Text.class); //6·輸入和輸出路徑 FileInputFormat.setInputPaths(job, new Path("C:\\Users\\slm\\Desktop\\datas\\friends.txt")); FileOutputFormat.setOutputPath(job, new Path("C:\\Users\\slm\\Desktop\\datas\\friend01")); //提交任務， job.waitForCompletion(true); }
第一步顯示結果：
A D,O,K,I,H,G,B,C,
B F,E,J,A,
C H,K,E,G,B,F,
D H,E,F,L,K,C,A,
E L,H,G,F,D,B,A,M,
F D,C,M,L,
G M,
H O,A,
I A,C,O,
J O,
K B,
L D,E,
M F,E,
O F,A,G,H,I,
P J,
Q G,
W G,
第二個mapreduce：
就是把上一個輸出當做輸入 通過切分 獲取好友的人
人與人之間進行拼接 用- 做成人人對 做key 共同好友做value傳到reduce端
map端：
static class FriendMap02 extends Mapper<LongWritable, Text, Text, Text>{
@Override
protected void map(LongWritable friend, Text value, Context context)
throws IOException, InterruptedException {
String[] friend_persons = value.toString().split("\t");
String friends = friend_persons[0];
String[] person = friend_persons[1].split(",");
//對集合進行排序
Arrays.sort(person);
//人與人之間進行拼接 用-
for(int i=0;i<person.length-1;i++) {
for(int j=i+1;j<person.length;j++) {
context.write(new Text(person[i]+"-"+person[j]), new Text(friends));
}
}
}
}
reduce端：
static class FriendReducer02 extends Reducer<Text, Text, Text, Text>{
@Override
protected void reduce(Text person_person, Iterable<Text> friends, Context context)
throws IOException, InterruptedException {
for (Text text : friends) {
context.write(new Text(person_person), new Text(text));
}
}
}
Driver端：
public static void main(String[] args) throws Exception {
//1·建立Configuration
Configuration conf = new Configuration();
Job job = Job.getInstance(conf);
//2·設定載入路徑
job.setJarByClass(Friends02.class);
//3·設定map和reduce
job.setMapperClass(FriendMap02.class);
job.setReducerClass(FriendReducer02.class);
//4·設定map輸出
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
//5·設定reduce輸出
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
//6·輸入和輸出路徑
FileInputFormat.setInputPaths(job, new Path("C:\\Users\\slm\\Desktop\\datas\\friend01\\part-r-00000"));
FileOutputFormat.setOutputPath(job, new Path("C:\\Users\\slm\\Desktop\\datas\\friend02"));
//提交任務，
job.waitForCompletion(true);
}
謝謝，留下你來過的足跡！！！！！！！！！