Mapreduce例項——求平均值
阿新 • • 發佈:2021-11-19
現有某電商關於商品點選情況的資料檔案,表名為goods_click,包含兩個欄位(商品分類,商品點選次數),分隔符“\t”,內容如下:
商品分類 商品點選次數
52127 5
52120 93
52092 93
52132 38
52006 462
52109 28
52109 43
52132 0
52132 34
52132 9
52132 30
52132 45
52132 24
52009 2615
52132 25
52090 13
52132 6
52136 0
52090 10
52024 347
使用mapreduce統計出每類商品的平均點選次數
package mapreduce4; import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.NullWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;import org.omg.CORBA.PUBLIC_MEMBER; //02.Mapreduce例項——求平均值 public class MyAverage { public static class Map extends Mapper<Object,Text,Text,IntWritable>{ private static Text newKey = new Text(); public void map(Object key,Text value,Context context) throws IOException,InterruptedException{ String line = value.toString(); System.out.println(line); String arr[] = line.split("\t"); newKey.set(arr[0]); int click=Integer.parseInt(arr[1]); context.write(newKey, new IntWritable(click)); } } public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable>{ public void reduce(Text key,Iterable<IntWritable> values,Context context) throws IOException, InterruptedException{ int num=0; int count=0; for(IntWritable val:values){ num+=val.get(); count++; } int avg=num/count; context.write(key,new IntWritable(avg)); } } public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException{ Configuration conf=new Configuration(); System.out.println("start"); Job job =new Job(conf,"MyAverage"); job.setJarByClass(MyAverage.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); Path in=new Path("hdfs://192.168.51.100:8020/mymapreduce4/in/goods_click"); Path out=new Path("hdfs://192.168.51.100:8020/mymapreduce4/out"); FileInputFormat.addInputPath(job,in); FileOutputFormat.setOutputPath(job,out); System.exit(job.waitForCompletion(true) ? 0 : 1); } }
結果:
原理:
求平均數是MapReduce比較常見的演算法,求平均數的演算法也比較簡單,一種思路是Map端讀取資料,在資料輸入到Reduce之前先經過shuffle,將map函式輸出的key值相同的所有的value值形成一個集合value-list,然後將輸入到Reduce端,Reduce端彙總並且統計記錄數,然後作商即可