elasticsearch效能測試工具rally深入詳解
題記
elasticsearch效能測試研究了很久，自己想過通過批量匯入資料，然後記錄時間，統計CPU、記憶體等變化，計算得出某個效能指標。但顯然，資料量起伏不定，非常不準確。
研究發現，github上提供了rally作為elasticsearch的效能測試工具，較好的解決了es效能測試問題。
1、esrally功能：es的效能測試工具。
esrally不支援windows版本，目前只支援Linux和Mac OS。
esrally的執行基於以下兩點假設：
1）所有執行在同一臺機器完成。(未來改假設條件可能會去掉）
2）你需要向Elasticsearch索引中新增特定的資料集，然後在其上進行基準查詢（benchmarking queries）.
2、esrally安裝：
前提:驗證是否安裝了以下內容（注意版本）
Python 3.4+ available as python3 on the path (verify with: python3 –version which should print Python 3.4.0 or higher)
pip3 available on the path (verify with pip3 –version)
JDK 8
git
如下所示：
[root@laoyang git-2.4.0]# python
Python 3.5.1 (default, Aug 2 2016 步驟1：安裝 Rally
pip3 install esrally
步驟2：配置 Rally
呼叫： esrally configure 會提示讓你輸入 java8.home（也就是JAVA_HOME設定的環境變數路徑）。
僅java8.home使用自己本機的JAVA_HOME環境變數路徑，其他採用預設值即可。
[[email protected] .rally]# cat /root/.rally/rally.ini
[meta]
config.version = 5
[system]
root.dir = /root/.rally/benchmarks
log.root.dir = logs
env.name = local
[source]
local.src.dir=/root/.rally/src #注意，此路徑並非elaticsearch安裝路徑。
remote.repo.url=https://github.com/elastic/elasticsearch.git
[provisioning]
local.install.dir = install
[runtime]
java8.home = /opt/jdk1.8.0_91 #注意，JAVA_HOME 路徑為：/opt/jdk1.8.0_91
[benchmarks]
local.dataset.cache = ${system:root.dir}/data
[reporting]
datastore.type = elaticsearch
datastore.host = 10.0.1.30
datastore.port = 9200
datastore.secure = False
datastore.user =
datastore.password =
[tracks]
default.url = https://github.com/elastic/rally-tracks步驟3：執行esrally
由於elasticsearch的執行必須非root賬戶。esrally建議用非root賬戶執行。
3、esrally執行常見問題及解決方案
問題1：版本用哪個版本？
目前用最新版本 Rally 0.3.2（2016-8-7發行）。
問題2：注意gradle需要2.13版本，高版本會不相容。
[[email protected] caches]# ll /usr/bin/gradle
lrwxrwxrwx 1 root root 27 Aug 5 17:13 /usr/bin/gradle -> /opt/gradle-2.13/bin/gradle
問題3：安裝時候提示 documents.json.bz2 無法下載成功。
2）移動到 /root/.rally/benchmarks/data/geonames/（對於root使用者）
對於非root使用者,elaticsearch使用者，移動路徑為：/home/elasticsearch/benchmarks/data/geonames/
問題4: esrally單獨執行和加引數執行區別？
下載elaticsearch2.3.4版本執行。
問題5： Could not resolve all dependencies for configuration ‘:benchmarks:classpath’.
37 > Could not download shadow.jar (com.github.jengelman.gradle.plugins:shadow:1.2.3)如何解決？
root賬戶下：
cd /home/elasticsearch/.rally/benchmarks/src
gradle :distribution:tar:assemble
正確提示資訊為：
deprecated API.
Note: Recompile with -Xlint:deprecation for details.
:modules:transport-netty4:processResources UP-TO-DATE
:modules:transport-netty4:classes
:modules:transport-netty4:jar
:modules:transport-netty4:copyPluginPropertiesTemplate
:modules:transport-netty4:pluginProperties
:modules:transport-netty4:bundlePlugin
:distribution:buildModules
:distribution:tar:buildTar
:distribution:tar:generatePomFileForNebulaPublication
:distribution:tar:signArchives SKIPPED
:distribution:tar:assemble
BUILD SUCCESSFUL4、esrally最終執行結果：
Preparing for race (might take a few moments) ...
Building from sources ...
/home/elasticsearch/.rally/benchmarks/races/2016-08-08-07-29-59/local/logs/rally_out.logRacing on track [geonames] and challenge [append-no-conflicts] with car [defaults]
Benchmarking indexing at 12183.1 docs/s [100% done]
Benchmarking stats (warmup iteration 100/100) [100% done]
Benchmarking stats (iteration 100/100) [100% done]
Benchmarking search (warmup iteration 100/100) [100% done]
Benchmarking search (iteration 100/100) [100% done]
------------------------------------------------------
_______ __ _____
/ ____(_)___ ____ _/ / / ___/_________ ________
/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \
/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------
| Metric | Value |
|----------------------------------------------------------:|----------:|
| Min Indexing Throughput [docs/s] | 11949 |
| Median Indexing Throughput [docs/s] | 12307 |
| Max Indexing Throughput [docs/s] | 13349 |
| Indexing time [min] | 164.045 |
| Merge time [min] | 32.3815 |
| Refresh time [min] | 8.82333 |
| Flush time [min] | 1.63852 |
| Merge throttle time [min] | 1.45482 |
| Query latency default (90.0 percentile) [ms] | 68.8676 |
| Query latency default (99.0 percentile) [ms] | 77.6009 |
| Query latency default (100 percentile) [ms] | 78.8328 |
| Query latency term (90.0 percentile) [ms] | 4.63227 |
| Query latency term (99.0 percentile) [ms] | 10.9506 |
| Query latency term (100 percentile) [ms] | 13.0573 |
| Query latency phrase (90.0 percentile) [ms] | 6.35018 |
| Query latency phrase (99.0 percentile) [ms] | 13.1745 |
| Query latency phrase (100 percentile) [ms] | 14.2399 |
| Query latency country_agg_uncached (90.0 percentile) [ms] | 361.913 |
| Query latency country_agg_uncached (99.0 percentile) [ms] | 384.75 |
| Query latency country_agg_uncached (100 percentile) [ms] | 392.645 |
| Query latency country_agg_cached (90.0 percentile) [ms] | 7.71451 |
| Query latency country_agg_cached (99.0 percentile) [ms] | 17.4704 |
| Query latency country_agg_cached (100 percentile) [ms] | 18.4851 |
| Query latency scroll (90.0 percentile) [ms] | 50.4234 |
| Query latency scroll (99.0 percentile) [ms] | 50.9866 |
| Query latency scroll (100 percentile) [ms] | 51.2115 |
| Query latency expression (90.0 percentile) [ms] | 482.02 |
| Query latency expression (99.0 percentile) [ms] | 485.951 |
| Query latency expression (100 percentile) [ms] | 491.999 |
| Query latency painless_static (90.0 percentile) [ms] | 890.382 |
| Query latency painless_static (99.0 percentile) [ms] | 918.354 |
| Query latency painless_static (100 percentile) [ms] | 927.282 |
| Query latency painless_dynamic (90.0 percentile) [ms] | 968.908 |
| Query latency painless_dynamic (99.0 percentile) [ms] | 1019.14 |
| Query latency painless_dynamic (100 percentile) [ms] | 1021.97 |
| Median CPU usage (index) [%] | 887.7 |
| Median CPU usage (stats) [%] | 94.9 |
| Median CPU usage (search) [%] | 445.05 |
| Total Young Gen GC [s] | 89.121 |
| Total Old Gen GC [s] | 12.274 |
| Index size [GB] | 3.30111 |
| Totally written [GB] | 20.2123 |
| Heap used for segments [MB] | 21.6794 |
| Heap used for doc values [MB] | 0.113129 |
| Heap used for terms [MB] | 20.0574 |
| Heap used for norms [MB] | 0.0761719 |
| Heap used for points [MB] | 0.770404 |
| Heap used for stored fields [MB] | 0.6623 |
| Segment count | 96 |
| Indices Stats(90.0 percentile) [ms] | 11.5116 |
| Indices Stats(99.0 percentile) [ms] | 14.1724 |
| Indices Stats(100 percentile) [ms] | 36.8348 |
| Nodes Stats(90.0 percentile) [ms] | 11.342 |
| Nodes Stats(99.0 percentile) [ms] | 13.435 |
| Nodes Stats(100 percentile) [ms] | 16.4768 |Logs for this race are archived in /home/elasticsearch/.rally/benchmarks/races/2016-08-08-07-29-59/local/logs-geonames-append-no-conflicts-defaults.zip5、問題大討論
原始碼 作者danielmitterdorfer一一回復， 最終作者回復：
I am very happy too and I am glad you had so much persistence. You also uncovered a few usability problems that I need to tackle.
幫作者提了一些建議，促使他升級了一個版本0.3.2。
小結
前面的確費勁周折，花費了接近3整天實踐，期間甚至告訴原作者工具太難用，給作者反覆溝通，最終問題解決。看到輸出結果的那一刻，整個人很興奮。堅持到底，相信付出的力量！不斷思考、反思中前行！
——————————————————————————————————
更多ES相關實戰乾貨經驗分享，請掃描下方【銘毅天下】微信公眾號二維碼關注。
（每週至少更新一篇！）
和你一起，死磕Elasticsearch！
——————————————————————————————————