1. 程式人生 > >Linux效能分析

Linux效能分析

生產環境伺服器變慢,診斷思路和效能評估

整機:top

程式碼

public class JavaDemo2 {
    public static void main(String[] args) {
        while (true){
            System.out.println(new java.util.Random().nextInt(77778888));
        }
    }
}

top命令檢視

[root@192 ~]# top
top - 11:28:50 up 7 min,  2 users,  load average: 0.73, 0.39, 0.18
Tasks: 230 total,   1 running, 229 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.0%us,  2.3%sy,  0.0%ni, 96.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1906280k total,   649948k used,  1256332k free,    21164k buffers
Swap:  2097148k total,        0k used,  2097148k free,   203820k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                              
  3672 root      20   0 3158m  94m  11m S 15.6  5.1   0:13.67 java                                                                                 
  3397 root      20   0  100m 5388 3396 S  9.3  0.3   0:09.00 sshd                                                                                 
    39 root      20   0     0    0    0 S  1.7  0.0   0:00.25 events/4                                                                             
    37 root      20   0     0    0    0 S  1.0  0.0   0:00.39 events/2                                                                             
    35 root      20   0     0    0    0 S  0.3  0.0   0:01.27 events/0                                                                             
    36 root      20   0     0    0    0 S  0.3  0.0   0:00.34 events/1                                                                             
    41 root      20   0     0    0    0 S  0.3  0.0   0:00.24 events/6   
[root@192 ~]# uptime
11:29:09 up 7 min,  2 users,  load average: 0.52, 0.36, 0.18

CPU:vmstat

[root@192 ~]# vmstat -n 2  3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 1313028  21228 203868    0    0    54     3  112 1620  0  1 98  0  0       
 0  0      0 1312732  21228 203896    0    0     0     0 2579 25940  1  3 96  0  0      
 0  0      0 1312856  21228 203896    0    0     0     0 2677 24290  1  3 97  0  0  

id:處於空閒的CPU百分比

wa:系統等待IO的CPU時間百分比

st:來自於一個虛擬機器偷取的CPU時間百分比

檢視額外的CPU資訊

所有CPU核資訊

[root@192 ~]# mpstat -P  ALL 2
Linux 2.6.32-642.el6.x86_64 (192.168.1.101)     08/05/2019      _x86_64_        (8 CPU)


11:42:40 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
11:42:42 AM  all    1.07    0.00    2.89    0.00    0.00    0.00    0.00    0.00   96.04
11:42:42 AM    0    0.00    0.00    1.52    0.00    0.00    0.00    0.00    0.00   98.48
11:42:42 AM    1    6.09    0.00    9.14    0.00    0.00    0.00    0.00    0.00   84.77
11:42:42 AM    2    0.50    0.00    4.02    0.00    0.00    0.00    0.00    0.00   95.48
11:42:42 AM    3    1.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00   97.00

每個程序使用CPU的用量分解資訊

[root@192 ~]# pidstat -u 1 -p 3790
Linux 2.6.32-642.el6.x86_64 (192.168.1.101)     08/05/2019      _x86_64_        (8 CPU)

11:45:00 AM       PID    %usr %system  %guest    %CPU   CPU  Command
11:45:01 AM      3790    6.00   10.00    0.00   16.00     7  java
11:45:02 AM      3790    4.00   11.00    0.00   15.00     7  java
11:45:03 AM      3790    9.00   15.00    0.00   24.00     7  java
11:45:04 AM      3790    7.00   14.00    0.00   21.00     7  java

記憶體:free

應用程式可用記憶體數

推薦使用free -m

[root@192 ~]# free -g
             total       used       free     shared    buffers     cached
Mem:             1          0          1          0          0          0
-/+ buffers/cache:          0          1
Swap:            1          0          1
[root@192 ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          1861        553       1308          1         20        199
-/+ buffers/cache:        333       1528
Swap:         2047          0       2047

檢視額外

[root@192 ~]# pidstat -p 3823 -r 2
Linux 2.6.32-642.el6.x86_64 (192.168.1.101)     08/05/2019      _x86_64_        (8 CPU)

11:51:16 AM       PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
11:51:18 AM      3823      1.00      0.00 3234728  47068   2.47  java
11:51:20 AM      3823      2.50      0.00 3234728  47068   2.47  java
11:51:22 AM      3823      0.50      0.00 3234728  47068   2.47  java
11:51:24 AM      3823      2.50      0.00 3234728  47068   2.47  java
11:51:26 AM      3823      0.50      0.00 3234728  47068   2.47  java

硬碟:df

檢視磁碟剩餘空間

[root@192 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        15G  5.6G  8.4G  41% /
tmpfs           931M   72K  931M   1% /dev/shm
/dev/sda1       190M   39M  142M  22% /boot

磁碟IO:iostat

[root@192 ~]# iostat -xdk 2 3
Linux 2.6.32-642.el6.x86_64 (192.168.1.101)     08/05/2019      _x86_64_        (8 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
scd0              0.00     0.00    0.38    0.00     1.50     0.00     8.00     0.00    0.67    0.67    0.00   0.67   0.03
sda              34.33     4.94   47.36    3.18  1631.23    32.44    65.84     0.07    1.38    1.22    3.72   0.88   4.43

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
scd0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    0.50    0.00     4.00     0.00    16.00     0.00    1.00    1.00    0.00   1.00   0.05

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
scd0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

檢視額外

[root@192 ~]# pidstat -d 2 -p 3485
Linux 2.6.32-642.el6.x86_64 (192.168.1.101)     08/05/2019      _x86_64_        (8 CPU)

02:16:10 PM       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
02:16:12 PM      3485      0.00      0.00      0.00  java
02:16:14 PM      3485      0.00      0.00      0.00  java
02:16:16 PM      3485      0.00      0.00      0.00  java
02:16:18 PM      3485      0.00      0.00      0.00  java
02:16:20 PM      3485      0.00      0.00      0.00  java
02:16:22 PM      3485      0.00      0.00      0.00  java
02:16:24 PM      3485      0.00      0.00      0.00  java

網路IO:ifstat

預設本地沒有,需要下載ifstat

安裝ifstat

[root@192 ifstat-1.1]# yum install flex byacc  libpcap ncurses ncurses-devel libpcap-devel gcc-c++
wget http://gael.roualland.free.fr/ifstat/ifstat-1.1.tar.gz
tar xzvf ifstat-1.1.tar.gz
cd ifstat-1.1
./configure
make
make install
[root@192 ifstat-1.1]# ifstat 1
       eth0       
 KB/s in  KB/s out
    0.06      0.13
    0.06      0.12
    0.06      0.12
    0.06      0.12
    0.06      0.12
    0.06      0.12
    0.12      0.17
    0.06      0.12

生產環境出現CPU佔用過高,分析思路和定位

public class JavaDemo2 {
    public static void main(String[] args) {
        while (true){
            System.out.println(new java.util.Random().nextInt(77778888));
        }
    }
}

先用top命令找出CPU佔比最高的

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                              
  4294 root      20   0 3158m  37m  11m S 16.6  2.0   0:03.28 java                                                                                 
  3428 root      20   0  100m 5444 3396 S 14.3  0.3   0:22.85 sshd                                                                                 
    35 root      20   0     0    0    0 S  1.0  0.0   0:03.18 events/0                                                                             
    41 root      20   0     0    0    0 S  0.3  0.0   0:00.52 events/6                                                                             
  2012 root      20   0  199m 5192 4288 S  0.3  0.3   0:00.32 ManagementAgent                                                                      
     1 root      20   0 19344 1544 1228 S  0.0  0.1   0:01.83 init                                                                                 
     2 root      20   0     0    0    0 S  0.0  0.0   0:00.02 kthreadd   

ps -ef或者jps進一步定位

[root@192 ifstat-1.1]# jps
4320 Jps
4294 JavaDemo2

定位到具體執行緒或者程式碼

[root@192 ifstat-1.1]# ps -mp 4294 -o THREAD,tid,time
USER     %CPU PRI SCNT WCHAN  USER SYSTEM    TID     TIME
root     18.6   -    - -         -      -      - 00:00:15
root      0.0  19    - futex_    -      -   4294 00:00:00
root     15.6  19    - n_tty_    -      -   4295 00:00:13
root      0.2  19    - futex_    -      -   4296 00:00:00
root      0.2  19    - futex_    -      -   4297 00:00:00
root      0.3  19    - futex_    -      -   4298 00:00:00
root      0.2  19    - futex_    -      -   4299 00:00:00
root      0.3  19    - futex_    -      -   4300 00:00:00
root      0.3  19    - futex_    -      -   4301 00:00:00
root      0.3  19    - futex_    -      -   4302 00:00:00
root      0.2  19    - futex_    -      -   4303 00:00:00
root      0.0  19    - futex_    -      -   4304 00:00:00
root      0.0  19    - futex_    -      -   4305 00:00:00
root      0.0  19    - futex_    -      -   4306 00:00:00
root      0.0  19    - futex_    -      -   4307 00:00:00
root      0.0  19    - futex_    -      -   4308 00:00:00
root      0.1  19    - futex_    -      -   4309 00:00:00
root      0.0  19    - futex_    -      -   4310 00:00:00
root      0.0  19    - futex_    -      -   4311 00:00:00
root      0.0  19    - futex_    -      -   4312 00:00:00
root      0.0  19    - futex_    -      -   4313 00:00:00

-m:顯示所有的執行緒

-p: pid程序使用CPU的時間

-o:該引數後是使用者自定義格式

執行緒ID轉換為16進位制格式(英文小寫格式)

[root@192 ifstat-1.1]# printf "%x\n" 4295
10c7

jstack 程序ID|grep tid(16進位制執行緒ID小寫英文) -A60

[root@192 ifstat-1.1]# jstack 4294 | grep 10c7 -A60
"main" #1 prio=5 os_prio=0 tid=0x00007f7f68009000 nid=0x10c7 runnable [0x00007f7f6f117000]
   java.lang.Thread.State: RUNNABLE
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:326)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        - locked <0x00000000e2e0e6e0> (a java.io.BufferedOutputStream)
        at java.io.PrintStream.write(PrintStream.java:482)
        - locked <0x00000000e2e0bbd8> (a java.io.PrintStream)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
        - locked <0x00000000e2e0e800> (a java.io.OutputStreamWriter)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
        at java.io.PrintStream.newLine(PrintStream.java:546)
        - eliminated <0x00000000e2e0bbd8> (a java.io.PrintStream)
        at java.io.PrintStream.println(PrintStream.java:737)
        - locked <0x00000000e2e0bbd8> (a java.io.PrintStream)
        at JavaDemo2.main(JavaDemo2.java:4)

"VM Thread" os_prio=0 tid=0x00007f7f68080000 nid=0x10d0 runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f7f6801e800 nid=0x10c8 runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f7f68020000 nid=0x10c9 runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f7f68022000 nid=0x10ca runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f7f68024000 nid=0x10cb runnable 

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f7f68025800 nid=0x10cc runnable 

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f7f68027800 nid=0x10cd runnable 

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f7f68029800 nid=0x10ce runnable 

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f7f6802b000 nid=0x10cf runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007f7f680e8000 nid=0x10d9 waiting on condition 

JNI global references: 5

參考

找到最耗CPU的java執行緒 - - ITeye部落格

本文由部落格一文多發平臺 OpenWrite 釋出!