(六)使用ab(Apache-BenchMark)壓測 Dubbo
1、準備
使用ab(Apache-BenchMark)壓測consumer的Controller,觸發呼叫。
dubbo的consumer發起呼叫,呼叫provider暴露的介面。
provider做1w次迴圈,生成隨機數做累加。
2、ab壓測情況
2.1、1k 資料
情況一:
三臺同機房的機器,利用兩臺consumer請求同一臺provider
機器:
consumer1 2h4g -server -Xmx4g -Xms4g -XX:+UseG1GC consumer2 4h8g -server -Xmx4g -Xms4g -XX:+UseG1GC provider 2h4g -server -Xmx2g -Xms2g -XX:+UseG1GC
因為單機無法讓provider機器cpu壓滿
取10併發,consumer2 進行100w次請求,consumer1 進行 80w次(為了和consumer2 在同一時間完成)請求 。
ab -n 1000000 -c 10 http://127.0.0.1:8091/consumer/stressTest/string1k
provider情況:
top - 09:51:00 up 1137 days, 18:04, 4 users, load average: 7.07, 3.55, 2.35 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu0 : 80.4%us, 11.0%sy, 0.0%ni, 1.0%id, 0.0%wa, 0.0%hi, 7.6%si, 0.0%st Cpu1 : 88.6%us, 9.4%sy, 0.0%ni, 2.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 3924680k total, 3124704k used, 799976k free, 102508k buffers Swap: 2097144k total, 6156k used, 2090988k free, 1074068k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1089 root 20 0 4905m 1.5g 13m S 196.8 39.3 49:05.17 java -Dfile.encoding=utf-8 -server -Xmx2g -Xms2g -XX:+UseG1GC -Xloggc:/data/dubboStress/logs/dubbo_gc.log -XX
可以看到CPU負載過高。
consumer1、consumer2 幾乎同時執行結束。
consumer1 2h4g 結果:
Concurrency Level: 10 Time taken for tests: 197.835 seconds Complete requests: 800000 Failed requests: 0 Write errors: 0 Total transferred: 87200000 bytes HTML transferred: 3200000 bytes Requests per second: 4043.77 [#/sec] (mean) Time per request: 2.473 [ms] (mean) Time per request: 0.247 [ms] (mean, across all concurrent requests) Transfer rate: 430.44 [Kbytes/sec] received Percentage of the requests served within a certain time (ms) 50% 2 66% 3 75% 3 80% 3 90% 4 95% 4 98% 5 99% 6 100% 224 (longest request)
consumer2 4h8g:
Concurrency Level: 10
Time taken for tests: 220.944 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 109000000 bytes
HTML transferred: 4000000 bytes
Requests per second: 4526.03 [#/sec] (mean)
Time per request: 2.209 [ms] (mean)
Time per request: 0.221 [ms] (mean, across all concurrent requests)
Transfer rate: 481.77 [Kbytes/sec] received
Percentage of the requests served within a certain time (ms)
50% 2
66% 2
75% 3
80% 3
90% 3
95% 4
98% 4
99% 5
100% 261 (longest request)
可以看到 兩臺機tps加起來有 8500多,95%響應時間都在 5ms以內。
情況二:
我再加一臺跨機房的機器測試一下。
consumer1 2h4g -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer2 4h8g -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer3 2h4g -server -Xmx2g -Xms2g -XX:+UseG1GC (跨網段)
provider 2h4g -server -Xmx2g -Xms2g -XX:+UseG1GC
consumer1:
Concurrency Level: 10
Time taken for tests: 269.711 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 109000000 bytes
HTML transferred: 4000000 bytes
Requests per second: 3707.68 [#/sec] (mean)
Time per request: 2.697 [ms] (mean)
Time per request: 0.270 [ms] (mean, across all concurrent requests)
Transfer rate: 394.66 [Kbytes/sec] received
Percentage of the requests served within a certain time (ms)
50% 2
66% 3
75% 3
80% 3
90% 4
95% 5
98% 5
99% 6
100% 233 (longest request)
consumer2:
Concurrency Level: 10
Time taken for tests: 251.411 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 109000000 bytes
HTML transferred: 4000000 bytes
Requests per second: 3977.55 [#/sec] (mean)
Time per request: 2.514 [ms] (mean)
Time per request: 0.251 [ms] (mean, across all concurrent requests)
Transfer rate: 423.39 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 2
Processing: 1 2 2.8 2 201
Waiting: 0 2 2.8 2 201
Total: 1 2 2.8 2 201
Percentage of the requests served within a certain time (ms)
50% 2
66% 3
75% 3
80% 3
90% 4
95% 4
98% 5
99% 6
100% 201 (longest request)
consumer3:
(跨機房,RTT較慢)
Concurrency Level: 10
Time taken for tests: 253.211 seconds
Complete requests: 220941
Failed requests: 0
Write errors: 0
Total transferred: 24082569 bytes
HTML transferred: 883764 bytes
Requests per second: 872.56 [#/sec] (mean)
Time per request: 11.461 [ms] (mean)
Time per request: 1.146 [ms] (mean, across all concurrent requests)
Transfer rate: 92.88 [Kbytes/sec] received
Percentage of the requests served within a certain time (ms)
50% 11
66% 11
75% 12
80% 12
90% 13
95% 14
98% 15
99% 17
100% 281 (longest request)
可以看到tps累加也差不多是 8500多。
情況三:
換成都是同機房的機器。
consumer1 2h4g -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer2 4h8g -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer3 8h8g -server -Xmx4g -Xms4g -XX:+UseG1GC
provider 2h4g -server -Xmx2g -Xms2g -XX:+UseG1GC
結果差不多,TPS有增加,誤差來自於 consumer1執行完成,consumer2、consumer3還沒執行完成,但是時間差在5秒以內。
consumer1:
Concurrency Level: 10
Time taken for tests: 299.224 seconds
Complete requests: 1000000
Requests per second: 3341.98 [#/sec] (mean)
Transfer rate: 355.74 [Kbytes/sec] received
Percentage of the requests served within a certain time (ms)
50% 3
66% 3
75% 3
80% 4
90% 4
95% 5
98% 6
99% 7
consmer2:
Concurrency Level: 10
Time taken for tests: 339.512 seconds
Complete requests: 1000000
Requests per second: 2945.41 [#/sec] (mean)
Transfer rate: 313.52 [Kbytes/sec] received
Percentage of the requests served within a certain time (ms)
50% 3
66% 4
75% 4
80% 4
90% 5
95% 6
98% 7
99% 8
consmer3:
Concurrency Level: 10
Time taken for tests: 368.969 seconds
Complete requests: 1000000
Requests per second: 2710.26 [#/sec] (mean)
Transfer rate: 288.49 [Kbytes/sec] received
Percentage of the requests served within a certain time (ms)
50% 3
66% 4
75% 4
80% 5
90% 5
95% 6
98% 7
99% 8
TPS差不多近9000,RTT響應時間差不多。
provider 伺服器情況(CPU爆高):
top - 17:53:54 up 1141 days, 2:07, 2 users, load average: 7.14, 5.33, 4.19
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu0 : 84.3%us, 8.7%sy, 0.0%ni, 0.7%id, 0.0%wa, 0.0%hi, 6.4%si, 0.0%st
Cpu1 : 89.4%us, 9.6%sy, 0.0%ni, 1.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3924680k total, 3889636k used, 35044k free, 135212k buffers
Swap: 2097144k total, 5988k used, 2091156k free, 1784112k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1089 root 20 0 4908m 1.5g 13m S 198.1 39.4 114:41.32 java -Dfile.encoding=utf-8 -server -Xmx2g -Xms2g -XX:+UseG1GC -Xloggc:/data/dubboStress/logs/dubbo_gc.log
2.2、100k資料
情況一:
consumer1 2h4g -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer2 8h8g -server -Xmx4g -Xms4g -XX:+UseG1GC
provider 2h4g -server -Xmx2g -Xms2g -XX:+UseG1GC
10w請求,20併發(100k請求太慢了。。)
ab -n 100000 -c 20 http://127.0.0.1:8091/consumer/stressTest/string100k
consumer1:
Concurrency Level: 20
Time taken for tests: 184.131 seconds
Requests per second: 543.09 [#/sec] (mean)
Transfer rate: 57.81 [Kbytes/sec] received
Percentage of the requests served within a certain time (ms)
50% 36
66% 38
75% 39
80% 40
90% 42
95% 44
98% 48
99% 50
consumer2:
Concurrency Level: 20
Time taken for tests: 185.186 seconds
Complete requests: 100000
Requests per second: 540.00 [#/sec] (mean)
Transfer rate: 57.48 [Kbytes/sec] received
Percentage of the requests served within a certain time (ms)
50% 36
66% 38
75% 39
80% 40
90% 42
95% 45
98% 49
99% 52
tps差不多1000,RTT 95% 45ms
情況二:
consumer1 2h4g -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer2 8h8g -server -Xmx4g -Xms4g -XX:+UseG1GC
provider 2h4g -server -Xmx2g -Xms2g -XX:+UseG1GC
10w請求 10併發
ab -n 100000 -c 10 http://127.0.0.1:8091/consumer/stressTest/string100k
consumer1:
Concurrency Level: 10
Time taken for tests: 184.276 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 10900000 bytes
HTML transferred: 400000 bytes
Requests per second: 542.66 [#/sec] (mean)
Time per request: 18.428 [ms] (mean)
Time per request: 1.843 [ms] (mean, across all concurrent requests)
Transfer rate: 57.76 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 7 18 2.8 18 67
Waiting: 7 18 2.8 18 67
Total: 7 18 2.8 18 67
Percentage of the requests served within a certain time (ms)
50% 18
66% 19
75% 20
80% 20
90% 21
95% 23
98% 25
99% 28
consumer2:
Concurrency Level: 10
Time taken for tests: 183.730 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 10900000 bytes
HTML transferred: 400000 bytes
Requests per second: 544.28 [#/sec] (mean)
Time per request: 18.373 [ms] (mean)
Time per request: 1.837 [ms] (mean, across all concurrent requests)
Transfer rate: 57.94 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 7 18 2.8 18 56
Waiting: 7 18 2.8 18 55
Total: 7 18 2.8 18 56
Percentage of the requests served within a certain time (ms)
50% 18
66% 19
75% 20
80% 20
90% 22
95% 23
98% 25
99% 27
TPS:1100 ,RTT 95% 23ms
說明並不是併發數越高越好。
情況三:
加一臺機器。
consumer1 2h4g -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer2 4h8g -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer3 8h8g -server -Xmx4g -Xms4g -XX:+UseG1GC
provider 2h4g -server -Xmx2g -Xms2g -XX:+UseG1GC
10w請求 10併發
ab -n 100000 -c 10 http://127.0.0.1:8091/consumer/stressTest/string100k
consumer1先執行完成,手動停止 2、3 ,存在2s誤差。
consumer1:
Concurrency Level: 10
Time taken for tests: 179.200 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 10900000 bytes
HTML transferred: 400000 bytes
Requests per second: 558.04 [#/sec] (mean)
Time per request: 17.920 [ms] (mean)
Time per request: 1.792 [ms] (mean, across all concurrent requests)
Transfer rate: 59.40 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 7 18 4.3 17 55
Waiting: 7 18 4.3 17 54
Total: 7 18 4.3 17 55
Percentage of the requests served within a certain time (ms)
50% 17
66% 19
75% 21
80% 22
90% 23
95% 25
98% 27
99% 29
consumer2:
Concurrency Level: 10
Time taken for tests: 190.898 seconds
Complete requests: 73115
Failed requests: 0
Write errors: 0
Total transferred: 7969535 bytes
HTML transferred: 292460 bytes
Requests per second: 383.01 [#/sec] (mean)
Time per request: 26.109 [ms] (mean)
Time per request: 2.611 [ms] (mean, across all concurrent requests)
Transfer rate: 40.77 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 7 26 5.6 26 67
Waiting: 7 26 5.6 26 66
Total: 7 26 5.6 26 67
Percentage of the requests served within a certain time (ms)
50% 26
66% 28
75% 29
80% 30
90% 33
95% 35
98% 38
99% 41
consumer3:
Concurrency Level: 10
Time taken for tests: 188.066 seconds
Complete requests: 44403
Failed requests: 0
Write errors: 0
Total transferred: 4839927 bytes
HTML transferred: 177612 bytes
Requests per second: 236.10 [#/sec] (mean)
Time per request: 42.354 [ms] (mean)
Time per request: 4.235 [ms] (mean, across all concurrent requests)
Transfer rate: 25.13 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 11 42 13.0 42 182
Waiting: 11 42 13.0 42 181
Total: 11 42 13.0 42 182
Percentage of the requests served within a certain time (ms)
50% 42
66% 48
75% 51
80% 53
90% 58
95% 64
98% 71
99% 76
tps 1100左右,RTT 要視機器而定。平均就是95% 41ms
provider 伺服器情況:
top - 10:18:21 up 1141 days, 18:31, 3 users, load average: 5.02, 4.67, 4.14
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.1%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.1%us, 0.0%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3924680k total, 3892608k used, 32072k free, 141252k buffers
Swap: 2097144k total, 5988k used, 2091156k free, 1726716k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1089 root 20 0 4908m 1.5g 13m S 198.5 40.8 188:01.30 java -Dfile.encoding=utf-8 -server -Xmx2g -Xms2g -XX:+UseG1GC
100w 請求 10 併發這裡不放了,TPS差不多,RTT會稍微高一點
3、分析
2.1、dubbo執行緒 WAITING
試過調大了 dubbo (預設是200)的執行緒數,發現 效果並沒有顯著的提升。
"DubboServerHandler-172.16.179.198:20890-thread-15" #54 daemon prio=5 os_prio=0 tid=0x00007f32c0015800 nid=0x4a4 waiting on condition [0x00007f32b5046000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000080e805a8> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:41)
at java.lang.Thread.run(Thread.java:748)
暫時無解,網上說法是執行緒都在同步佇列,太多反而出現資源等待的情況。
2.2、調大ab的併發數
超過20執行緒,RTT會變得很大,原因:
- 執行緒的上下文切換嚴重
- 執行緒等待CPU排程(ab發出請求)
隨後TPS會達到瓶頸,並不是十分顯著。
4、結果彙總
provider機器:
2h4g
CentOS release 6.4 (Final)
model name : QEMU Virtual CPU version 2.5+
stepping : 3
cpu MHz : 2099.998
cache size : 4096 KB
JVM:
jdk1.8
-server -Xmx2g -Xms2g -XX:+UseG1GC
1、多機,以壓滿CPU為目的
ab -n 1000000 -c 10
詳情見上。
1k dubbo+hessian2 | 100k dubbo+hessian2 | |
---|---|---|
TPS | 8500~9000 | 1000~1100 |
RTT | 95% 5ms | 95% 41ms |
OOM | 無 | 無 |
2、單機,不壓滿CPU
單機:(沒有壓滿CPU,provider機器同上,使用consumer2)
過程省略。
ab -n 1000000 -c 10
1k dubbo+hessian2 | 100k dubbo+hessian2 | |
---|---|---|
TPS | 4000~4500 | 600~700 |
RTT | 95% 4ms | 95% 19ms |
OOM | 無 | 無 |
CPU | 125%+ | 150% |
參考:
- 壓測dubbo:https://blog.csdn.net/u013815546/article/details/101385888
- 壓測dubbo:(有點不一樣)https://blog.csdn.net/cyjs1988/article/details/84258046
- https://blog.51cto.com/ydhome/1861956
- 官網:https://jmeter-plugins.org/wiki/Start
- jmeter使用:https://blog.csdn.net/github_27109687/article/details/71968662
- 使用 rest 協議:https://dangdangdotcom.github.io/dubbox/rest.html
- 基於Dubbo的Hessian協議實現遠端呼叫:http://shiyanjun.cn/archives/349.html
Java資源分享: