1. 程式人生 > 其它 >(六)使用ab(Apache-BenchMark)壓測 Dubbo

(六)使用ab(Apache-BenchMark)壓測 Dubbo

1、準備

使用ab(Apache-BenchMark)壓測consumer的Controller,觸發呼叫。

dubbo的consumer發起呼叫,呼叫provider暴露的介面。

provider做1w次迴圈,生成隨機數做累加。

2、ab壓測情況

2.1、1k 資料

情況一:

三臺同機房的機器,利用兩臺consumer請求同一臺provider

機器:

consumer1  2h4g   -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer2  4h8g   -server -Xmx4g -Xms4g -XX:+UseG1GC
provider   2h4g	  -server -Xmx2g -Xms2g -XX:+UseG1GC 

因為單機無法讓provider機器cpu壓滿

取10併發,consumer2 進行100w次請求,consumer1 進行 80w次(為了和consumer2 在同一時間完成)請求 。

 ab -n 1000000 -c  10 http://127.0.0.1:8091/consumer/stressTest/string1k

provider情況:

top - 09:51:00 up 1137 days, 18:04,  4 users,  load average: 7.07, 3.55, 2.35
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu0  : 80.4%us, 11.0%sy,  0.0%ni,  1.0%id,  0.0%wa,  0.0%hi,  7.6%si,  0.0%st
Cpu1  : 88.6%us,  9.4%sy,  0.0%ni,  2.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3924680k total,  3124704k used,   799976k free,   102508k buffers
Swap:  2097144k total,     6156k used,  2090988k free,  1074068k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                       
 1089 root      20   0 4905m 1.5g  13m S 196.8 39.3  49:05.17 java -Dfile.encoding=utf-8 -server -Xmx2g -Xms2g -XX:+UseG1GC -Xloggc:/data/dubboStress/logs/dubbo_gc.log -XX

可以看到CPU負載過高。

consumer1、consumer2 幾乎同時執行結束。

consumer1 2h4g 結果:

Concurrency Level:      10
Time taken for tests:   197.835 seconds
Complete requests:      800000
Failed requests:        0
Write errors:           0
Total transferred:      87200000 bytes
HTML transferred:       3200000 bytes
Requests per second:    4043.77 [#/sec] (mean)
Time per request:       2.473 [ms] (mean)
Time per request:       0.247 [ms] (mean, across all concurrent requests)
Transfer rate:          430.44 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      3
  75%      3
  80%      3
  90%      4
  95%      4
  98%      5
  99%      6
 100%    224 (longest request)

consumer2 4h8g:

Concurrency Level:      10
Time taken for tests:   220.944 seconds
Complete requests:      1000000
Failed requests:        0
Write errors:           0
Total transferred:      109000000 bytes
HTML transferred:       4000000 bytes
Requests per second:    4526.03 [#/sec] (mean)
Time per request:       2.209 [ms] (mean)
Time per request:       0.221 [ms] (mean, across all concurrent requests)
Transfer rate:          481.77 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      3
  80%      3
  90%      3
  95%      4
  98%      4
  99%      5
 100%    261 (longest request)

可以看到 兩臺機tps加起來有 8500多,95%響應時間都在 5ms以內。

情況二:

我再加一臺跨機房的機器測試一下。

consumer1  2h4g   -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer2  4h8g   -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer3  2h4g   -server -Xmx2g -Xms2g -XX:+UseG1GC (跨網段)
provider   2h4g	  -server -Xmx2g -Xms2g -XX:+UseG1GC 

consumer1:

Concurrency Level:      10
Time taken for tests:   269.711 seconds
Complete requests:      1000000
Failed requests:        0
Write errors:           0
Total transferred:      109000000 bytes
HTML transferred:       4000000 bytes
Requests per second:    3707.68 [#/sec] (mean)
Time per request:       2.697 [ms] (mean)
Time per request:       0.270 [ms] (mean, across all concurrent requests)
Transfer rate:          394.66 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      3
  75%      3
  80%      3
  90%      4
  95%      5
  98%      5
  99%      6
 100%    233 (longest request)

consumer2:

Concurrency Level:      10
Time taken for tests:   251.411 seconds
Complete requests:      1000000
Failed requests:        0
Write errors:           0
Total transferred:      109000000 bytes
HTML transferred:       4000000 bytes
Requests per second:    3977.55 [#/sec] (mean)
Time per request:       2.514 [ms] (mean)
Time per request:       0.251 [ms] (mean, across all concurrent requests)
Transfer rate:          423.39 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       2
Processing:     1    2   2.8      2     201
Waiting:        0    2   2.8      2     201
Total:          1    2   2.8      2     201

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      3
  75%      3
  80%      3
  90%      4
  95%      4
  98%      5
  99%      6
 100%    201 (longest request)

consumer3:

(跨機房,RTT較慢)

Concurrency Level:      10
Time taken for tests:   253.211 seconds
Complete requests:      220941
Failed requests:        0
Write errors:           0
Total transferred:      24082569 bytes
HTML transferred:       883764 bytes
Requests per second:    872.56 [#/sec] (mean)
Time per request:       11.461 [ms] (mean)
Time per request:       1.146 [ms] (mean, across all concurrent requests)
Transfer rate:          92.88 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms)
  50%     11
  66%     11
  75%     12
  80%     12
  90%     13
  95%     14
  98%     15
  99%     17
 100%    281 (longest request)

可以看到tps累加也差不多是 8500多。

情況三:

換成都是同機房的機器。

consumer1  2h4g   -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer2  4h8g   -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer3  8h8g   -server -Xmx4g -Xms4g -XX:+UseG1GC
provider   2h4g	  -server -Xmx2g -Xms2g -XX:+UseG1GC 

結果差不多,TPS有增加,誤差來自於 consumer1執行完成,consumer2、consumer3還沒執行完成,但是時間差在5秒以內。

consumer1:

Concurrency Level:      10
Time taken for tests:   299.224 seconds
Complete requests:      1000000
Requests per second:    3341.98 [#/sec] (mean)
Transfer rate:          355.74 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      3
  75%      3
  80%      4
  90%      4
  95%      5
  98%      6
  99%      7

consmer2:

Concurrency Level:      10
Time taken for tests:   339.512 seconds
Complete requests:      1000000
Requests per second:    2945.41 [#/sec] (mean)
Transfer rate:          313.52 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      4
  75%      4
  80%      4
  90%      5
  95%      6
  98%      7
  99%      8

consmer3:

Concurrency Level:      10
Time taken for tests:   368.969 seconds
Complete requests:      1000000
Requests per second:    2710.26 [#/sec] (mean)
Transfer rate:          288.49 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      4
  75%      4
  80%      5
  90%      5
  95%      6
  98%      7
  99%      8

TPS差不多近9000,RTT響應時間差不多。

provider 伺服器情況(CPU爆高):

top - 17:53:54 up 1141 days,  2:07,  2 users,  load average: 7.14, 5.33, 4.19
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu0  : 84.3%us,  8.7%sy,  0.0%ni,  0.7%id,  0.0%wa,  0.0%hi,  6.4%si,  0.0%st
Cpu1  : 89.4%us,  9.6%sy,  0.0%ni,  1.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3924680k total,  3889636k used,    35044k free,   135212k buffers
Swap:  2097144k total,     5988k used,  2091156k free,  1784112k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                       
 1089 root      20   0 4908m 1.5g  13m S 198.1 39.4 114:41.32 java -Dfile.encoding=utf-8 -server -Xmx2g -Xms2g -XX:+UseG1GC -Xloggc:/data/dubboStress/logs/dubbo_gc.log

2.2、100k資料

情況一:

consumer1  2h4g   -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer2  8h8g   -server -Xmx4g -Xms4g -XX:+UseG1GC
provider   2h4g	  -server -Xmx2g -Xms2g -XX:+UseG1GC 

10w請求,20併發(100k請求太慢了。。)

ab -n 100000 -c 20 http://127.0.0.1:8091/consumer/stressTest/string100k

consumer1:

Concurrency Level:      20
Time taken for tests:   184.131 seconds
Requests per second:    543.09 [#/sec] (mean)
Transfer rate:          57.81 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms)
  50%     36
  66%     38
  75%     39
  80%     40
  90%     42
  95%     44
  98%     48
  99%     50

consumer2:

Concurrency Level:      20
Time taken for tests:   185.186 seconds
Complete requests:      100000
Requests per second:    540.00 [#/sec] (mean)
Transfer rate:          57.48 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms)
  50%     36
  66%     38
  75%     39
  80%     40
  90%     42
  95%     45
  98%     49
  99%     52

tps差不多1000,RTT 95% 45ms

情況二:

consumer1  2h4g   -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer2  8h8g   -server -Xmx4g -Xms4g -XX:+UseG1GC
provider   2h4g	  -server -Xmx2g -Xms2g -XX:+UseG1GC 

10w請求 10併發

ab -n 100000 -c 10 http://127.0.0.1:8091/consumer/stressTest/string100k

consumer1:

Concurrency Level:      10
Time taken for tests:   184.276 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      10900000 bytes
HTML transferred:       400000 bytes
Requests per second:    542.66 [#/sec] (mean)
Time per request:       18.428 [ms] (mean)
Time per request:       1.843 [ms] (mean, across all concurrent requests)
Transfer rate:          57.76 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     7   18   2.8     18      67
Waiting:        7   18   2.8     18      67
Total:          7   18   2.8     18      67

Percentage of the requests served within a certain time (ms)
  50%     18
  66%     19
  75%     20
  80%     20
  90%     21
  95%     23
  98%     25
  99%     28

consumer2:

Concurrency Level:      10
Time taken for tests:   183.730 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      10900000 bytes
HTML transferred:       400000 bytes
Requests per second:    544.28 [#/sec] (mean)
Time per request:       18.373 [ms] (mean)
Time per request:       1.837 [ms] (mean, across all concurrent requests)
Transfer rate:          57.94 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     7   18   2.8     18      56
Waiting:        7   18   2.8     18      55
Total:          7   18   2.8     18      56

Percentage of the requests served within a certain time (ms)
  50%     18
  66%     19
  75%     20
  80%     20
  90%     22
  95%     23
  98%     25
  99%     27

TPS:1100 ,RTT 95% 23ms

說明並不是併發數越高越好。

情況三:

加一臺機器。

consumer1  2h4g   -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer2  4h8g   -server -Xmx4g -Xms4g -XX:+UseG1GC
consumer3  8h8g   -server -Xmx4g -Xms4g -XX:+UseG1GC
provider   2h4g	  -server -Xmx2g -Xms2g -XX:+UseG1GC 

10w請求 10併發

ab -n 100000 -c 10 http://127.0.0.1:8091/consumer/stressTest/string100k

consumer1先執行完成,手動停止 2、3 ,存在2s誤差。

consumer1:

Concurrency Level:      10
Time taken for tests:   179.200 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      10900000 bytes
HTML transferred:       400000 bytes
Requests per second:    558.04 [#/sec] (mean)
Time per request:       17.920 [ms] (mean)
Time per request:       1.792 [ms] (mean, across all concurrent requests)
Transfer rate:          59.40 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     7   18   4.3     17      55
Waiting:        7   18   4.3     17      54
Total:          7   18   4.3     17      55

Percentage of the requests served within a certain time (ms)
  50%     17
  66%     19
  75%     21
  80%     22
  90%     23
  95%     25
  98%     27
  99%     29

consumer2:

Concurrency Level:      10
Time taken for tests:   190.898 seconds
Complete requests:      73115
Failed requests:        0
Write errors:           0
Total transferred:      7969535 bytes
HTML transferred:       292460 bytes
Requests per second:    383.01 [#/sec] (mean)
Time per request:       26.109 [ms] (mean)
Time per request:       2.611 [ms] (mean, across all concurrent requests)
Transfer rate:          40.77 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     7   26   5.6     26      67
Waiting:        7   26   5.6     26      66
Total:          7   26   5.6     26      67

Percentage of the requests served within a certain time (ms)
  50%     26
  66%     28
  75%     29
  80%     30
  90%     33
  95%     35
  98%     38
  99%     41

consumer3:

Concurrency Level:      10
Time taken for tests:   188.066 seconds
Complete requests:      44403
Failed requests:        0
Write errors:           0
Total transferred:      4839927 bytes
HTML transferred:       177612 bytes
Requests per second:    236.10 [#/sec] (mean)
Time per request:       42.354 [ms] (mean)
Time per request:       4.235 [ms] (mean, across all concurrent requests)
Transfer rate:          25.13 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:    11   42  13.0     42     182
Waiting:       11   42  13.0     42     181
Total:         11   42  13.0     42     182

Percentage of the requests served within a certain time (ms)
  50%     42
  66%     48
  75%     51
  80%     53
  90%     58
  95%     64
  98%     71
  99%     76

tps 1100左右,RTT 要視機器而定。平均就是95% 41ms

provider 伺服器情況:

top - 10:18:21 up 1141 days, 18:31,  3 users,  load average: 5.02, 4.67, 4.14
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.1%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.1%us,  0.0%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3924680k total,  3892608k used,    32072k free,   141252k buffers
Swap:  2097144k total,     5988k used,  2091156k free,  1726716k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                       
 1089 root      20   0 4908m 1.5g  13m S 198.5 40.8 188:01.30 java -Dfile.encoding=utf-8 -server -Xmx2g -Xms2g -XX:+UseG1GC 

100w 請求 10 併發這裡不放了,TPS差不多,RTT會稍微高一點

3、分析

2.1、dubbo執行緒 WAITING

試過調大了 dubbo (預設是200)的執行緒數,發現 效果並沒有顯著的提升。

"DubboServerHandler-172.16.179.198:20890-thread-15" #54 daemon prio=5 os_prio=0 tid=0x00007f32c0015800 nid=0x4a4 waiting on condition [0x00007f32b5046000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000080e805a8> (a java.util.concurrent.SynchronousQueue$TransferStack)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:41)
        at java.lang.Thread.run(Thread.java:748)

暫時無解,網上說法是執行緒都在同步佇列,太多反而出現資源等待的情況。

2.2、調大ab的併發數

超過20執行緒,RTT會變得很大,原因:

  • 執行緒的上下文切換嚴重
  • 執行緒等待CPU排程(ab發出請求)

隨後TPS會達到瓶頸,並不是十分顯著。

4、結果彙總

provider機器:

2h4g

CentOS release 6.4 (Final)
model name      : QEMU Virtual CPU version 2.5+
stepping        : 3
cpu MHz         : 2099.998
cache size      : 4096 KB

JVM:

jdk1.8
-server -Xmx2g -Xms2g -XX:+UseG1GC 

1、多機,以壓滿CPU為目的

ab -n 1000000 -c  10 

詳情見上。

1k dubbo+hessian2 100k dubbo+hessian2
TPS 8500~9000 1000~1100
RTT 95% 5ms 95% 41ms
OOM

2、單機,不壓滿CPU

單機:(沒有壓滿CPU,provider機器同上,使用consumer2)

過程省略。

ab -n 1000000 -c  10 
1k dubbo+hessian2 100k dubbo+hessian2
TPS 4000~4500 600~700
RTT 95% 4ms 95% 19ms
OOM
CPU 125%+ 150%

參考:


Java資源分享:

Java學習路線思維導圖+Java學習視訊+簡歷模板+Java電子書