叢集基準效能測試

問題引入：輸入端有2T的資料，問多長時間能把資料上傳到叢集？

假如說1個小時，那麼雙11時瞬間產生100T資料，伺服器能頂住嗎？

因此，我們需要對伺服器進行壓力測試，對伺服器整體效能有精準把控

1）測試HDFS寫效能

測試內容，向叢集上傳10個128M的檔案

[root@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.3-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 128MB
#-write 寫測試   -nrFiles 10 寫10個檔案  -fileSize 128M  每個檔案大小128M
2020-04-16 13:41:24,724 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
2020-04-16 13:41:24,724 INFO fs.TestDFSIO: Date & time: Thu Apr 16 13:41:24 CST 2020
2020-04-16 13:41:24,724 INFO fs.TestDFSIO:         Number of files: 10  #上傳了10個檔案
2020-04-16 13:41:24,725 INFO fs.TestDFSIO:  Total MBytes processed: 1280  #每個檔案128M
												#吞吐量（寫效能）8M/s
									#100個G需要的時間 (100*1024)/8=3.5小時
2020-04-16 13:41:24,725 INFO fs.TestDFSIO:       Throughput mb/sec: 8.88
2020-04-16 13:41:24,725 INFO fs.TestDFSIO:  Average IO rate mb/sec: 8.96
2020-04-16 13:41:24,725 INFO fs.TestDFSIO:   IO rate std deviation: 0.87
2020-04-16 13:41:24,725 INFO fs.TestDFSIO:      Test exec time sec: 67.61
 

2)測試讀效能

測試內容，讀取HDFS叢集10個128M的檔案

[root@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.3-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 128MB

2020-04-16 13:43:38,857 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
2020-04-16 13:43:38,858 INFO fs.TestDFSIO:   Date & time: Thu Apr 16 13:43:38 CST 2020
2020-04-16 13:43:38,859 INFO fs.TestDFSIO:         Number of files: 10	#讀取10個檔案
2020-04-16 13:43:38,859 INFO fs.TestDFSIO:  Total MBytes processed: 1280  #每個檔案128M 
2020-04-16 13:43:38,859 INFO fs.TestDFSIO:       Throughput mb/sec: 85.54 #吞吐量85M/s
2020-04-16 13:43:38,860 INFO fs.TestDFSIO:  Average IO rate mb/sec: 100.21  
2020-04-16 13:43:38,860 INFO fs.TestDFSIO:   IO rate std deviation: 44.37
2020-04-16 13:43:38,860 INFO fs.TestDFSIO:      Test exec time sec: 53.61
 

3)刪除測試生成的資料

hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.3-tests.jar TestDFSIO -clean

4)測試MR計算能力

機器128G記憶體以上才可以測試本命令

（1）使用RandomWriter來產生隨機數，每個節點執行10個Map任務，每個Map產生大約1G大小的二進位制隨機數
[root@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar randomwriter random-data
（2）執行Sort程式
[root@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar sort random-data sorted-data
（3）驗證資料是否真正排好序了
[root@hadoop102 mapreduce]$ 
hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.3-tests.jar testmapredsort -sortInput random-data -sortOutput sorted-data