大資料壓力測試工具HiBench
需求描述:需要測試cdh叢集的元件的效能和穩定性。
壓力測試工具選型:HiBench
HiBench測試工具說明:
HiBench是一套基準測試套件,用於幫助我們評估不同的大資料框架效能指標(包括處理速度、吞吐等)的負載指標,可以評估Hadoop、Spark和流式負載等
原始碼下載:wget https://github.com/intel-hadoop/HiBench/archive/HiBench-7.0.zip
原始碼編譯:命令:mvn -Dspark=2.1 -Dscala=2.11 clean package
官方文件:https://github.com/intel-hadoop/HiBench/blob/master/docs/build-hibench.md
前提條件:
1)、需要python2.x(>=2.6)
2)、需要linux bc命令,用於生成Hibench 報告,若沒有,則安裝:yum install bc
修改配置:
vim conf/hadoop.conf
hibench.hadoop.home:hadoop安裝目錄
hibench.hadoop.executable:你的bin/hadoop所在目錄,一般是{HADOOP_HOME}/bin/hadoop
hibench.hadoop.configure.dir:hadoop配置檔案所在目錄,一般位於HADOOP_HOME}/etc/hadoop
hibench.hdfs.master:hdfs上儲存Hibench資料的目錄,如:hdfs://localhost:8020/user/hibench
hibench.hadoop.release:hadoop發行版提供者,支援value:apache, cdh5, hdp
/root/apps/HiBench-master/conf
-rw-r--r-- 1 root root 245 Dec 12 21:30 benchmarks.lst
-rw-r--r-- 1 root root 332 Aug 13 02:34 flink.conf.template
-rw-r--r-- 1 root root 13 Dec 12 21:30 frameworks.lst
-rw-r--r-- 1 root root 246 Aug 13 02:34 gearpump.conf.template
-rw-r--r-- 1 root root 471 Dec 13 20:30 hadoop.conf
-rw-r--r-- 1 root root 448 Aug 13 02:34 hadoop.conf.template
-rw-r--r-- 1 root root 6600 Dec 18 04:57 hibench.conf
-rw-r--r-- 1 root root 1640 Dec 18 22:42 spark.conf
-rw-r--r-- 1 root root 1655 Aug 13 02:34 spark.conf.template
-rw-r--r-- 1 root root 942 Aug 13 02:34 storm.conf.template
drwxr-xr-x 8 root root 109 Dec 14 02:49 workloads #裡面有對應配置產生的資料量配置
舉例:micro
[[email protected] conf]# ll /root/apps/HiBench-master/conf/workloads/micro
total 20
-rw-r--r-- 1 root root 1920 Aug 13 02:34 dfsioe.conf
-rwxr-xr-x 1 root root 805 Aug 13 02:34 sleep.conf
-rw-r--r-- 1 root root 657 Aug 13 02:34 sort.conf
-rw-r--r-- 1 root root 571 Dec 18 04:40 terasort.conf
-rwxr-xr-x 1 root root 658 Dec 14 02:38 wordcount.conf
目錄結構:
/root/apps/HiBench-master/bin/workloads下面有:
graph
micro#選這個!!測試
ml
sql
streaming
websearch
測試演算法包:
/root/apps/HiBench-master/bin/workloads/micro/dfsioe# hdfsio測試
/root/apps/HiBench-master/bin/workloads/micro/sleep
/root/apps/HiBench-master/bin/workloads/micro/sort
/root/apps/HiBench-master/bin/workloads/micro/wordcount
cd /root/apps/HiBench-master/bin/workloads/micro/terasort#排序
準備資料啟動:
/root/apps/HiBench-master/bin/workloads/micro/terasort/prepare/prepare.sh
啟動mr任務:
/root/apps/HiBench-master/bin/workloads/micro/terasort/hadoop/run.sh
啟動spark任務:
/root/apps/HiBench-master/bin/workloads/micro/terasort/spark/run.sh
測試報告:
/root/apps/HiBench-master/report
drwxr-xr-x 4 root root 44 Dec 14 03:58 bayes
-rw-r--r-- 1 root root 6651 Dec 19 01:39 hibench.report
drwxr-xr-x 3 root root 28 Dec 13 23:56 sort
drwxr-xr-x 4 root root 44 Dec 13 03:45 terasort
drwxr-xr-x 5 root root 61 Dec 12 04:11 wordcount