wordcount 程式碼解析

阿新 • • 發佈：2018-11-01

MapRuduce

map表示對映 reduce表示化簡。它的思想就是‘分而治之’，具體思想就不用說了這裡主要解析wordcount原始碼。程式碼裡的思想是一直是K,V對(鍵值對)傳輸的重要的是map （）、reduce（）兩個函式。 main方法裡主要job作業的配置、啟動

main

Configuration configuration = new Configuration();
        Job job = new Job(configuration, WordCount.class.getSimpleName());
        job.setJarByClass(WordCount.class);
        // 打jar包 


        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);
        // 通過job設定輸入/輸出格式
        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        // 設定輸入/輸出路徑
        job.setMapperClass(WordMap.class);
        job.setReducerClass(WordReduce.class);
        // 設定處理Map/Reduce階段的類 

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        // 設定最終輸出key/value的型別m
        job.waitForCompletion(true);
        // 提交作業

資料是：兩行文字

    i am Malik Cheng
    i am hadoop

完整的WordCount.java


import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import 
 org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class WordCount {

    public static void main(String[] args) throws IOException,
            ClassNotFoundException, InterruptedException {
        if (args.length != 2 || args == null) {
            System.out.println("please input Path!");
            System.exit(0);
        }
        Configuration configuration = new Configuration();
        Job job = new Job(configuration, WordCount.class.getSimpleName());
        job.setJarByClass(WordCount.class);
        // 打jar包

        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);
        // 通過job設定輸入/輸出格式
        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        // 設定輸入/輸出路徑
        job.setMapperClass(WordMap.class);
        job.setReducerClass(WordReduce.class);
        // 設定處理Map/Reduce階段的類
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        // 設定最終輸出key/value的型別m
        job.waitForCompletion(true);
        // 提交作業
    }

    /*
     * keyin:輸入每行文字的偏移量，型別為LongerWritable(ObjectWritable) value:每行文字的內容，型別為Text
     * keyout:輸出中間結果的key，型別根據實際情況設定 valueout:輸出中間結果的value，型別根據實際情況設定
     */

    static class WordMap extends Mapper<LongWritable, Text, Text, IntWritable> {
        protected void map(
                LongWritable key,
                Text value,
                org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable>.Context context)
                throws java.io.IOException, InterruptedException {
            System.out.println("WordMap.map...");
            System.out.println("Map key:" + key.toString() + " ,Map value:"
                    + value.toString());
            String[] lines = value.toString().split(" ");
            for (String word : lines) {

                context.write(new Text(word), new IntWritable(1));
                // 每個單詞出現１次，作為中間結果輸出
                System.out.println("word:" + word + ",one:"
                        + new IntWritable(1).toString());
            }
            System.out.println("context:" + context.toString());
        };
    }

    /*
     * keyin:輸入的key值，型別與map中的keyout一致 valuein:輸入中間結果的value值，型別與map中的valueout一致
     * keyout:最終結果的key值 valueout:最終結果的value值
     */

    static class WordReduce extends
            Reducer<Text, IntWritable, Text, IntWritable> {
        protected void reduce(
                Text key,
                java.lang.Iterable<IntWritable> values,
                org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable>.Context context)
                throws java.io.IOException, InterruptedException {
            System.out.println("WordReduce rudece...");
            int sum = 0;
            System.out.println("---------------------values:");
            for (IntWritable count : values) {
                sum = sum + count.get();

                System.out.println("count:"+count+", sum:"+sum);
            }
            context.write(key, new IntWritable(sum));// 輸出最終結果
            System.out.println("Rudece key:"+key.toString()+", sum :"+new IntWritable(sum).toString());
            System.out.println("Rudece context:" + context.toString()+", sum :"+new IntWritable(sum).toString());
        };
    }
}

對列印的結果解析：

map階段

WordMap.map...
Map key:0 ,Map value:i am Malik Cheng
word:i,one:1
word:am,one:1
word:Malik,one:1
word:Cheng,one:1
context:[email protected]2ceec589
WordMap.map...
Map key:17 ,Map value:i am hadoop
word:i,one:1
word:am,one:1
word:hadoop,one:1
context:[email protected]2ceec589

map函式引數 :map(LongWritable key,Text value,Context context) key 和 value是我們傳入的資料，value其實是真是資料（i am Malik….）,key 是用來幫助換行的偏移量而context上下文物件，context作為了map和reduce執行中各個函式的一個橋樑，這個設計和java web中的session物件、application物件很相似。
從上面的輸出可以看出有2次 WordMap.map…說明map函式被呼叫了2次，問什麼會有2次呼叫尼？原來因為TextInputFormat型別的，都是按行處理。每一行的內容會在value引數中傳進來，也就是說每一行的內容都對應了一個key，這個key為此行的開頭位置在本檔案中的所在位置（所以第1行的key是0，第2行的key是17）。這樣每個單詞像這樣word:Malik,one:1記錄在context中，用來傳給reduce

reduce階段

 WordReduce rudece...
---------------------values:
count:1, sum:1
Rudece key:Cheng, sum :1
Rudece context:[email protected]5e76ee18, sum :1
WordReduce rudece...
---------------------values:
count:1, sum:1
Rudece key:Malik, sum :1
Rudece context:[email protected]5e76ee18, sum :1
WordReduce rudece...
---------------------values:
count:1, sum:1
count:1, sum:2
Rudece key:am, sum :2
Rudece context:[email protected]5e76ee18, sum :2
WordReduce rudece...
---------------------values:
count:1, sum:1
Rudece key:hadoop, sum :1
Rudece context:[email protected]5e76ee18, sum :1
WordReduce rudece...
---------------------values:
count:1, sum:1
count:1, sum:2
Rudece key:i, sum :2
Rudece context:[email protected]5e76ee18, sum :2

分析：reduce 化簡的意思，第一個問題：為什麼是化簡？在Wordcount裡，化簡就是有對map傳來的相同key（每個key對應的value是1）進行遍歷求和。第二個問題為什麼會有WordReduce rudece…輸出，也就是為什麼有5次呼叫reduce尼？其實知道第一個問題的答案，這個也就有答案了。因為資料裡面有5個不同的單詞（5個不同key），所以也就有5次化簡咯。總結 map（）是按行呼叫、map（）按key呼叫。

控制檯輸出

```
2017-07-20 18:07:02,175 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-07-20 18:07:02,820 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1019)) - session.id is deprecated. Instead, use dfs.metrics.session-id
2017-07-20 18:07:02,821 INFO  [main] jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId=
2017-07-20 18:07:03,090 WARN  [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(150)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2017-07-20 18:07:03,093 WARN  [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(259)) - No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2017-07-20 18:07:03,175 INFO  [main] input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 1
2017-07-20 18:07:03,213 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(396)) - number of splits:1
2017-07-20 18:07:03,394 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(479)) - Submitting tokens for job: job_local130204698_0001
2017-07-20 18:07:03,479 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2368)) - file:/tmp/hadoop-zkpk/mapred/staging/zkpk130204698/.staging/job_local130204698_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2017-07-20 18:07:03,489 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2368)) - file:/tmp/hadoop-zkpk/mapred/staging/zkpk130204698/.staging/job_local130204698_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2017-07-20 18:07:03,713 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2368)) - file:/tmp/hadoop-zkpk/mapred/local/localRunner/zkpk/job_local130204698_0001/job_local130204698_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2017-07-20 18:07:03,733 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2368)) - file:/tmp/hadoop-zkpk/mapred/local/localRunner/zkpk/job_local130204698_0001/job_local130204698_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2017-07-20 18:07:03,761 INFO  [main] mapreduce.Job (Job.java:submit(1289)) - The url to track the job: http://localhost:8080/
2017-07-20 18:07:03,762 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1334)) - Running job: job_local130204698_0001
2017-07-20 18:07:03,763 INFO  [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2017-07-20 18:07:03,779 INFO  [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2017-07-20 18:07:03,936 INFO  [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2017-07-20 18:07:03,937 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local130204698_0001_m_000000_0
2017-07-20 18:07:03,995 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : [ ]
2017-07-20 18:07:04,001 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(733)) - Processing split: hdfs://master:9000/user/wordcount/input1/h:0+29
2017-07-20 18:07:04,017 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:createSortingCollector(388)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2017-07-20 18:07:04,083 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1182)) - (EQUATOR) 0 kvi 26214396(104857584)
2017-07-20 18:07:04,083 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(975)) - mapreduce.task.io.sort.mb: 100
2017-07-20 18:07:04,083 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(976)) - soft limit at 83886080
2017-07-20 18:07:04,084 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(977)) - bufstart = 0; bufvoid = 104857600
2017-07-20 18:07:04,084 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(978)) - kvstart = 26214396; length = 6553600
WordMap.map...
Map key:0 ,Map value:i am Malik Cheng
word:i,one:1
word:am,one:1
word:Malik,one:1
word:Cheng,one:1
context:[email protected]2ceec589
WordMap.map...
Map key:17 ,Map value:i am hadoop
word:i,one:1
word:am,one:1
word:hadoop,one:1
context:[email protected]2ceec589
2017-07-20 18:07:04,420 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2017-07-20 18:07:04,423 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1437)) - Starting flush of map output
2017-07-20 18:07:04,424 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1455)) - Spilling map output
2017-07-20 18:07:04,424 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1456)) - bufstart = 0; bufend = 57; bufvoid = 104857600
2017-07-20 18:07:04,424 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1458)) - kvstart = 26214396(104857584); kvend = 26214372(104857488); length = 25/6553600
2017-07-20 18:07:04,437 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1641)) - Finished spill 0
2017-07-20 18:07:04,441 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(1001)) - Task:attempt_local130204698_0001_m_000000_0 is done. And is in the process of committing
2017-07-20 18:07:04,453 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - map
2017-07-20 18:07:04,453 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local130204698_0001_m_000000_0' done.
2017-07-20 18:07:04,453 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local130204698_0001_m_000000_0
2017-07-20 18:07:04,453 INFO  [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2017-07-20 18:07:04,456 INFO  [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2017-07-20 18:07:04,457 INFO  [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local130204698_0001_r_000000_0
2017-07-20 18:07:04,463 INFO  [pool-6-thread-1] mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : [ ]
2017-07-20 18:07:04,467 INFO  [pool-6-thread-1] mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: [email protected]5031c1e1
2017-07-20 18:07:04,479 INFO  [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(193)) - MergerManager: memoryLimit=304244320, maxSingleShuffleLimit=76061080, mergeThreshold=200801264, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2017-07-20 18:07:04,484 INFO  [EventFetcher for fetching Map Completion Events] reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local130204698_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2017-07-20 18:07:04,517 INFO  [localfetcher#1] reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(140)) - localfetcher#1 about to shuffle output of map attempt_local130204698_0001_m_000000_0 decomp: 73 len: 77 to MEMORY
2017-07-20 18:07:04,521 INFO  [localfetcher#1] reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 73 bytes from map-output for attempt_local130204698_0001_m_000000_0
2017-07-20 18:07:04,523 INFO  [localfetcher#1] reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(307)) - closeInMemoryFile -> map-output of size: 73, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->73
2017-07-20 18:07:04,524 INFO  [EventFetcher for fetching Map Completion Events] reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2017-07-20 18:07:04,525 INFO  [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2017-07-20 18:07:04,525 INFO  [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(667)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2017-07-20 18:07:04,534 INFO  [pool-6-thread-1] mapred.Merger (Merger.java:merge(591)) - Merging 1 sorted segments
2017-07-20 18:07:04,534 INFO  [pool-6-thread-1] mapred.Merger (Merger.java:merge(690)) - Down to the last merge-pass, with 1 segments left of total size: 65 bytes
2017-07-20 18:07:04,536 INFO  [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(742)) - Merged 1 segments, 73 bytes to disk to satisfy reduce memory limit
2017-07-20 18:07:04,536 INFO  [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(772)) - Merging 1 files, 77 bytes from disk
2017-07-20 18:07:04,537 INFO  [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(787)) - Merging 0 segments, 0 bytes from memory into reduce
2017-07-20 18:07:04,537 INFO  [pool-6-thread-1] mapred.Merger (Merger.java:merge(591)) - Merging 1 sorted segments
2017-07-20 18:07:04,537 INFO  [pool-6-thread-1] mapred.Merger (Merger.java:merge(690)) - Down to the last merge-pass, with 1 segments left of total size: 65 bytes
2017-07-20 18:07:04,538 INFO  [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2017-07-20 18:07:04,572 INFO  [pool-6-thread-1] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1019)) - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
WordReduce rudece...
---------------------values:
count:1, sum:1
Rudece key:Cheng, sum :1
Rudece context:[email protected]5e76ee18, sum :1
WordReduce rudece...
---------------------values:
count:1, sum:1
Rudece key:Malik, sum :1
Rudece context:[email protected]5e76ee18, sum :1
WordReduce rudece...
---------------------values:
count:1, sum:1
count:1, sum:2
Rudece key:am, sum :2
Rudece context:[email protected]5e76ee18, sum :2
WordReduce rudece...
---------------------values:
count:1, sum:1
Rudece key:hadoop, sum :1
Rudece context:[email protected]5e76ee18, sum :1
WordReduce rudece...
---------------------values:
count:1, sum:1
count:1, sum:2
Rudece key:i, sum :2
Rudece context:[email protected]5e76ee18, sum :2
2017-07-20 18:07:04,700 INFO  [pool-6-thread-1] mapred.Task (Task.java:done(1001)) - Task:attempt_local130204698_0001_r_000000_0 is done. And is in the process of committing
2017-07-20 18:07:04,703 INFO  [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2017-07-20 18:07:04,703 INFO  [pool-6-thread-1] mapred.Task (Task.java:commit(1162)) - Task attempt_local130204698_0001_r_000000_0 is allowed to commit now
2017-07-20 18:07:04,713 INFO  [pool-6-thread-1] output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local130204698_0001_r_000000_0' to hdfs://master:9000/user/wordcount/output1/_temporary/0/task_local130204698_0001_r_000000
2017-07-20 18:07:04,715 INFO  [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2017-07-20 18:07:04,715 INFO  [pool-6-thread-1] mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local130204698_0001_r_000000_0' done.
2017-07-20 18:07:04,716 INFO  [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local130204698_0001_r_000000_0
2017-07-20 18:07:04,716 INFO  [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2017-07-20 18:07:04,765 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1355)) - Job job_local130204698_0001 running in uber mode : false
2017-07-20 18:07:04,766 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1362)) -  map 100% reduce 100%
2017-07-20 18:07:04,767 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1373)) - Job job_local130204698_0001 completed successfully
2017-07-20 18:07:04,793 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380)) - Counters: 38
    File System Counters
        FILE: Number of bytes read=502
        FILE: Number of bytes written=457295
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=58
        HDFS: Number of bytes written=34
        HDFS: Number of read operations=15
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=4
    Map-Reduce Framework
        Map input records=2
        Map output records=7
        Map output bytes=57
        Map output materialized bytes=77
        Input split bytes=107
        Combine input records=0
        Combine output records=0
        Reduce input groups=5
        Reduce shuffle bytes=77
        Reduce input records=7
        Reduce output records=5
        Spilled Records=14
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=0
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
        Total committed heap usage (bytes)=396361728
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=29
    File Output Format Counters 
        Bytes Written=34

wordcount 程式碼解析

MapRuduce map表示對映 reduce表示化簡。它的思想就是‘分而治之’，具體思想就不用說了這裡主要解析wordcount原始碼。程式碼裡的思想是一直是K,V對(鍵值對)傳輸的重要的是map （）、reduce（）兩個函式。 main方法裡主要job作業的配置、啟動

WordCount程式碼實現及測試

1.專案地址：開發者：201631062515 201631062415 碼雲地址：https://gitee.com/heshuxiang/WordCount/tree/master 2.專案需求對程式設計語言原始檔統計字元數、單詞數、行數，統計結果以指定格式輸出到預

GraphSAGE 程式碼解析 - minibatch.py

class EdgeMinibatchIterator """ This minibatch iterator iterates over batches of sampled edges or random pairs of co-occuring edges.

Faster RCNN演算法訓練程式碼解析（2）

接著上篇的部落格，我們獲取imdb和roidb的資料後，就可以搭建網路進行訓練了。我們回到trian_rpn()函式裡面，此時執行完了roidb, imdb = get_roidb(imdb_name)，取得了imdb和roidb資料。先進入第一階段的訓練： print

Faster RCNN演算法訓練程式碼解析（3）

四個層的forward函式分析： RoIDataLayer：讀資料，隨機打亂等 AnchorTargetLayer：輸出所有anchors（這裡分析這個） ProposalLayer：用產生的anchors平移整圖，裁剪出界、移除低於閾值的的anchors，排序後使用nms，返回頂部排名的anchors

Matlab程式設計之——卷積神經網路CNN程式碼解析

卷積神經網路CNN程式碼解析 deepLearnToolbox-master是一個深度學習matlab包，裡面含有很多機器學習演算法，如卷積神經網路CNN，深度信念網路DBN，自動編碼AutoE ncoder（堆疊SAE，卷積CAE）的作者是 RasmusBerg Palm 今天給介紹d

SparkStream WordCount程式碼詳解

下面我要引用的程式碼來自於sparkStream的官網案例，案例內容為從kafka讀取資料進行wordcount計算 package org.apache.spark.examples.streaming; import java.util.HashMap; import java.util.

GraphSAGE 程式碼解析(四) - models.py

1. 類及其繼承關係 Model / \ / \ MLP GeneralizedModel / \ / \ Node2VecModel SampleAndAggregate 首先看Model, Ge

GraphSAGE 程式碼解析(三) - aggregators.py

1. class MeanAggregator(Layer): 該類主要用於實現 1. __init__() __init_() 用於獲取並初始化成員變數 dropout, bias(False), act(ReLu), concat(False), input_dim, output_

opencv實現螢幕錄製程式碼解析

# -*- coding: utf-8 -*- """ Created on Thu Nov 15 22:37:02 2018 #QQ群：476842922（歡迎加群討論學習） @author: Administrator """ """python + opencv 實現螢幕錄製""" f

C語言程式設計狼追兔子問題程式碼解析

問題描述一隻兔子躲進了10個環形分佈的洞中的一個。狼在第一個洞中沒有找到兔子，就隔一個洞，到第3個洞去找；也沒有找到，就隔2個洞，到第6個洞去找；以後每次多一個洞去找兔子……這樣下去，如果一直找不到兔子，請問兔子可能在哪個洞中？問題分析首先定義一個數組a[11]，其陣列元素為a[1]，a[2]，a

C語言解決常勝將軍問題程式碼解析

問題描述有21根火柴，兩人依次取，每次每人只可取走1〜4根，不能多取，也不能不取，誰取到最後一根火柴誰輸。請編寫一個人機對弈程式，要求人先取，計算機後取；計算機為“常勝將軍”。問題分析可以這樣思考這個問題：要想讓計算機是“常勝將軍”，也就是要讓人取到最後一根火柴。這樣只有一種可能，那就是讓計算機只

筆記-爬蟲-js程式碼解析

筆記-爬蟲-js程式碼解析 1. js程式碼解析 1.1. 前言在爬取網站時經常會有js生成關鍵資訊，而且js程式碼是混淆過的。以瓜子二手車為例，直接請求https://www.guaz

keras/examples/mnist_acgan.py ACGAN程式碼解析

# -*- coding: utf-8 -*- """ Train an Auxiliary Classifier Generative Adversarial Network (ACGAN) on the MNIST dataset. See https://arxiv.org/

經典裝飾器程式碼解析

程式碼： def A(funC): def B(funE): def C(*args, **kwargs): out = funC(funE)(*args,**kwargs) return out + "...111

#######haohaohao#######對抗思想與強化學習的碰撞-SeqGAN模型原理和程式碼解析

1、背景 GAN作為生成模型的一種新型訓練方法，通過discriminative model來指導generative model的訓練，並在真實資料中取得了很好的效果。儘管如此，當目標是一個待生成的非連續性序列時，該方法就會表現出其侷限性。非連續性序列生成，比如說文字生成，為什麼單純的使用GA

python2和python3的區別（附帶程式碼解析）

1. input()函式 python2中的input()函式：獲取當前輸入的內容，並將其作為指令來處理； python3中的input()函式：獲取當前輸入的內容，並將其作為字串來處理；在pytohn2環境中：在輸入中文‘小花’的時候，會顯示語法

ThreadPoolExecutor程式碼解析

派生體系 java.util.concurrent 　　ThreadPoolExecutor 　　　　AbstractExecutorService 　　　　　　ExecutorService 　　　　　　　　Executor 這個類是

JavaScript Base64 作為檔案上傳的例項程式碼解析

這篇文章主要介紹了JavaScript Base64 作為檔案上傳的例項程式碼解析,非常不錯，具有參考借鑑價值，需要的朋友可以參考下。例如我們用某些裁剪外掛得到的圖片是： <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEU

【TensorFlow】多GPU訓練：示例程式碼解析

使用多GPU有助於提升訓練速度和調參效率。本文主要對tensorflow的示例程式碼進行註釋解析：cifar10_multi_gpu_train.py 1080Ti下加速效果如下（batch=128）單卡：兩個GPU比單個GPU加速了近一倍：

wordcount 程式碼解析

MapRuduce

main

資料是：兩行文字

完整的WordCount.java

對列印的結果解析：

map階段

reduce階段

控制檯輸出

相關推薦