MRunit 測試MapReduce 程式
阿新 • • 發佈:2018-11-29
使用MRunit 測試 MapReduce ,就比如我們寫java 程式是的junit單元測試,其實,mrunit 和標準的junit框架一起使用,可以將mapreduce作業的測試作為正常開發環境的一部分執行。
準備測試的包:mrunit-1.1.0-hadoop2.jar ,下載地址;https://repository.apache.org/content/repositories/releases/org/apache/mrunit/mrunit/1.1.0/
單單一個 還不夠,因為我測試的時候報了很多錯,發現是缺少
apache-mrunit-1.1.0-hadoop2-bin.tar.gz lib 檔案目錄下相關的包
下載地址: http://archive.apache.org/dist/mrunit/mrunit-1.0.0/
將上面相關的jar檔案匯入eclipse 專案中,同時匯入junit4.
以wordcount 為例:
對WordCountMap 右鍵,新建 junit test case 工程。
我這裡用 MapDriver 測試驅動。
import java.io.IOException; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mrunit.mapreduce.MapDriver; import org.junit.Test; public class WordCountMapTest { @Test public void testMapLongWritableTextContext() throws IOException { Text value=new Text("a good man "); new MapDriver<LongWritable, Text, Text, IntWritable>() .withMapper(new WordCountMap()) .withInput(new LongWritable(1),value) .withOutput(new Text("a"),new IntWritable(1)) .withOutput(new Text("good"),new IntWritable(1)) .withOutput(new Text("man"),new IntWritable(1)) .runTest(); } }
右鍵run as---->junit
WordCountReduce的測試,使用ReduceDriver 驅動。程式碼如下:
import java.io.IOException; import java.util.ArrayList; import java.util.List; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mrunit.mapreduce.ReduceDriver; import org.junit.Before; import org.junit.Test; public class WordCountReduceTest { @Before public void setUp() throws Exception { } @Test public void testReduceTextIterableOfIntWritableContext() throws IOException { List<IntWritable> values = new ArrayList<IntWritable>(); values.add(new IntWritable(1)); values.add(new IntWritable(1)); new ReduceDriver<Text, IntWritable, Text, IntWritable>() .withReducer(new WordCountReduce()) .withInput(new Text("6"), values) .withOutput(new Text("6"), new IntWritable(2)) .runTest(); } }
測試方法,如同map。