MapReduce--求哪些人兩兩之間是互粉好友
阿新 • • 發佈:2018-10-12
for hot on() config args void person ava dex
數據:
A:B,C,D,F,E,O B:A,C,E,K C:F,A,D,I D:A,E,F,L E:B,C,D,M,L F:A,B,C,D,E,O,M G:A,C,D,E,F H:A,C,D,E,O I:A,O J:B,O K:A,C,D L:D,E,F M:E,F,G O:A,H,I,J,K
求哪些人兩兩之間是互粉好友,形如:A的好友有B,B的好友有A 。 那麽A和B就是互粉好友。
思路:
對每一行數據進行組合輸出 (person-person,1),
然後再Reducer階段進行統計,等於2的就是互粉好友對;
問題:將B-A轉換成A-B這種形式;
方案:比較兩個字符之間的大小,小的在前;
import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.NullWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;public class EachOtherFriend { public static class MyMapper extends Mapper<LongWritable, Text, Text, IntWritable>{ @Override protected void map(LongWritable key, Text value,Context context) throws IOException, InterruptedException { String[] lines = value.toString().split(":"); char person = lines[0].charAt(0); for (String str : lines[1].split(",")) { char friend = str.charAt(0); String per_per = ""; if(person > friend){ per_per += friend+"-"+person; }else{ per_per+= person+"-"+friend; } context.write(new Text(per_per), new IntWritable(1)); } } } public static class MyReducer extends Reducer<Text, IntWritable, Text, NullWritable>{ @Override protected void reduce(Text key, Iterable<IntWritable> values,Context context) throws IOException, InterruptedException { int num = 0; for (IntWritable it : values) { num++; } if(num > 1){ context.write(key, null); } } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf); job.setJarByClass(EachOtherFriend.class); job.setMapperClass(MyMapper.class); job.setReducerClass(MyReducer.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(NullWritable.class); FileInputFormat.setInputPaths(job, new Path("G:/files/input")); FileOutputFormat.setOutputPath(job, new Path("G:/files/output")); boolean isDone = job.waitForCompletion(true); System.exit(isDone ? 0:1); } }
MapReduce--求哪些人兩兩之間是互粉好友