MapReduce的本地執行模式(debug除錯)
(1)mapreduce程式是被提交給LocalJobRunner在本地以單程序的形式執行。在本地執行mapreduce程式可以更快地執行,並且可以使用debug進行跟蹤程式碼,方便查錯,在本地執行主要是看mapreduce的業務邏輯是不是正確,如果在本地執行錯誤的話,那麼在叢集上肯定也是錯的
(2)處理的資料及輸出結果可以在本地檔案系統,也可以在hdfs上
(3)本地模式非常便於進行業務邏輯的debug,只要在eclipse中打斷點即可
(4)怎樣實現本地執行?寫一個程式,不要帶叢集的配置檔案(本質是你的mr程式的conf中是否有mapreduce.framework.name=local以及yarn.resourcemanager.hostname引數)
如果在windows下想執行本地模式來測試程式邏輯,需要在windows中配置環境變數:
%HADOOP_HOME% = d:/hadoop-2.6.1
%PATH% = %HADOOP_HOME%\bin
並且要將d:/hadoop-2.6.1的lib和bin目錄替換成windows平臺編譯的版本
無jar版windows平臺hadoop-2.6.1.zip
https://pan.baidu.com/s/1uu8tKq3FvoemccoUiBob0Q
在linux下編譯的hadoop:
https://pan.baidu.com/s/1qtXrWnKjk_klRdPj8DaSYA
只需要替換一下就可以正常使用了
Configuration conf = new Configuration();
//是否執行為本地模式,就是看這個引數值是否為local,預設就是local
conf.set("mapreduce.framework.name", "local");
//本地模式執行mr程式時,輸入輸出的資料可以在本地,也可以在hdfs上
//到底在哪裡,就看以下兩行配置你用哪行,預設就是file:///
/*conf.set("fs.defaultFS", "hdfs://mini1:9000/");*/
conf.set("fs.defaultFS", "file:///");
conf什麼都不寫,就是使用預設的配置,預設的配置就是在本地。
可以看一下預設的配置
預設的配置檔案:
https://pan.baidu.com/s/1xKT7_SPv8qsSQmbze8O6wQ
可以自己在原始碼中分別找到這些配置檔案:
開啟mapred-default.xml這個配置檔案:
開啟core-default.xml這個配置檔案:
所以可以直接在本地執行
在本地執行的效果:
在本地跟叢集執行的效果是一樣的。
input: 裡面的檔案都是隨便輸入的
output:
最終輸出的結果:
程式碼地址:
https://gitee.com/tanghongping/hadoopMapReduce/tree/master/src/com/thp/bigdata/wcdemo
我們可以打幾個斷電除錯一下:
看這map方法的引數:
Name | Value |
---|---|
key | 0 |
value | we are family |
context | [email protected]fa |
然後就是接下的一行的資料
map裡面的引數:
Name | Value |
---|---|
key | 15 |
value | hello sa |
context | [email protected]fa |
之後的input檔案的每一行的資料都會經歷這個過程
看debug模式下下面控制面板的變化:
現在還是處在map階段
此時output會生成這麼一個臨時性的檔案
當input資料夾裡面的第一個檔案處理完畢,接下來就會處理第二個檔案:
我們的input資料夾裡面總共是有7個檔案的,這7個檔案都是同一個檔案:
2018-11-14 09:12:17,123 INFO [localfetcher#1] reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#1 about to shuffle output of map attempt_local1550344450_0001_m_000005_0 decomp: 211 len: 215 to MEMORY
2018-11-14 09:12:17,128 INFO [localfetcher#1] reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 211 bytes from map-output for attempt_local1550344450_0001_m_000005_0
2018-11-14 09:12:17,131 INFO [localfetcher#1] reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(315)) - closeInMemoryFile -> map-output of size: 211, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->211
2018-11-14 09:12:17,138 INFO [localfetcher#1] reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#1 about to shuffle output of map attempt_local1550344450_0001_m_000002_0 decomp: 211 len: 215 to MEMORY
2018-11-14 09:12:17,140 INFO [localfetcher#1] reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 211 bytes from map-output for attempt_local1550344450_0001_m_000002_0
2018-11-14 09:12:17,141 INFO [localfetcher#1] reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(315)) - closeInMemoryFile -> map-output of size: 211, inMemoryMapOutputs.size() -> 2, commitMemory -> 211, usedMemory ->422
2018-11-14 09:12:17,159 INFO [localfetcher#1] reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#1 about to shuffle output of map attempt_local1550344450_0001_m_000006_0 decomp: 211 len: 215 to MEMORY
2018-11-14 09:12:17,160 INFO [localfetcher#1] reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 211 bytes from map-output for attempt_local1550344450_0001_m_000006_0
2018-11-14 09:12:17,161 INFO [localfetcher#1] reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(315)) - closeInMemoryFile -> map-output of size: 211, inMemoryMapOutputs.size() -> 3, commitMemory -> 422, usedMemory ->633
2018-11-14 09:12:17,170 INFO [localfetcher#1] reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#1 about to shuffle output of map attempt_local1550344450_0001_m_000003_0 decomp: 211 len: 215 to MEMORY
2018-11-14 09:12:17,172 INFO [localfetcher#1] reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 211 bytes from map-output for attempt_local1550344450_0001_m_000003_0
2018-11-14 09:12:17,173 INFO [localfetcher#1] reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(315)) - closeInMemoryFile -> map-output of size: 211, inMemoryMapOutputs.size() -> 4, commitMemory -> 633, usedMemory ->844
2018-11-14 09:12:17,180 INFO [localfetcher#1] reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#1 about to shuffle output of map attempt_local1550344450_0001_m_000000_0 decomp: 211 len: 215 to MEMORY
2018-11-14 09:12:17,181 INFO [localfetcher#1] reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 211 bytes from map-output for attempt_local1550344450_0001_m_000000_0
2018-11-14 09:12:17,182 INFO [localfetcher#1] reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(315)) - closeInMemoryFile -> map-output of size: 211, inMemoryMapOutputs.size() -> 5, commitMemory -> 844, usedMemory ->1055
2018-11-14 09:12:17,189 INFO [localfetcher#1] reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#1 about to shuffle output of map attempt_local1550344450_0001_m_000001_0 decomp: 211 len: 215 to MEMORY
2018-11-14 09:12:17,190 INFO [localfetcher#1] reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 211 bytes from map-output for attempt_local1550344450_0001_m_000001_0
2018-11-14 09:12:17,190 INFO [localfetcher#1] reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(315)) - closeInMemoryFile -> map-output of size: 211, inMemoryMapOutputs.size() -> 6, commitMemory -> 1055, usedMemory ->1266
2018-11-14 09:12:17,196 INFO [localfetcher#1] reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#1 about to shuffle output of map attempt_local1550344450_0001_m_000004_0 decomp: 211 len: 215 to MEMORY
2018-11-14 09:12:17,197 INFO [localfetcher#1] reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 211 bytes from map-output for attempt_local1550344450_0001_m_000004_0
2018-11-14 09:12:17,197 INFO [localfetcher#1] reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(315)) - closeInMemoryFile -> map-output of size: 211, inMemoryMapOutputs.size() -> 7, commitMemory -> 1266, usedMemory ->1477
2018-11-14 09:12:17,198 INFO [EventFetcher for fetching Map Completion Events] reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
當所有的map都執行完成,那麼就要走到reduce階段,
protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int count = 0;
for(IntWritable value : values) {
count += value.get(); // 這個count 最後就是某一個單詞的彙總的值
}
context.write(key, new IntWritable(count));
}
reduce的引數:
Name | Value |
---|---|
key | are |
values | o[email protected]6e4a6525 |
context | [email protected]cda784 |
count | 0 |
接下來的階段就是不斷地重複這個過程,但是需要注意的一點是這個reduce的過程是已經排序了的過程,按照字典排序好的。