網絡卡多佇列，中斷配置
阿新 • • 發佈：2021-01-23
1. 檢視每一個cpu的負載以及中斷指標：（參考：https://www.cnblogs.com/tcicy/p/10197136.html）
業務方在使用KVM虛擬機器進行效能壓測時，發現某一個核的softirq佔比特別高，如下所示：
mpstat -P ALL 1 Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle Average: all 9.96 0.00 3.17 0.01 0.71 2.00 0.00 0.00 0.00 84.15 Average: 0 0.30 0.00 0.51 0.00 0.05 1.24 0.00 0.00 0.00 97.89 Average: 1 55.32 0.00 18.52 0.00 0.53 0.14 0.00 0.00 0.00 25.49 Average: 2 37.03 0.00 15.00 0.00 11.73 12.93 0.00 0.00 0.00 23.30 Average: 3 56.76 0.00 18.98 0.00 0.50 0.14 0.00 0.00 0.00 23.62 Average: 4 50.96 0.00 11.05 0.00 1.41 0.07 0.00 0.00 0.00 36.52 Average: 5 55.81 0.00 19.03 0.00 0.47 0.14 0.00 0.00 0.00 24.56 Average: 6 43.61 0.00 10.13 0.00 0.49 0.09 0.00 0.00 0.00 45.68 Average: 7 57.33 0.00 19.45 0.00 0.47 0.13 0.00 0.00 0.00 22.62 Average: 8 44.77 0.00 9.83 0.00 0.42 0.09 0.00 0.00 0.00 44.90 Average: 9 1.08 0.00 2.77 0.00 1.42 0.08 0.00 0.00 0.00 94.65 Average: 10 0.04 0.00 0.17 0.00 1.21 6.78 0.00 0.00 0.00 91.79 Average: 11 0.02 0.00 0.06 0.00 0.01 0.01 0.00 0.00 0.00 99.89 Average: 12 0.02 0.00 0.16 0.00 1.39 7.04 0.00 0.00 0.00 91.39 Average: 13 0.02 0.00 0.08 0.01 0.01 0.00 0.00 0.00 0.00 99.88 Average: 14 0.00 0.00 0.07 0.00 0.00 0.00 0.00 0.00 0.00 99.92 Average: 15 0.04 0.00 0.09 0.00 0.00 0.00 0.00 0.00 0.00 99.86 Average: 16 0.02 0.00 0.05 0.00 0.00 0.00 0.00 0.00 0.00 99.92 Average: 17 0.04 0.00 0.03 0.00 0.01 0.01 0.00 0.00 0.00 99.91 Average: 18 0.00 0.00 0.04 0.00 0.01 0.01 0.00 0.00 0.00 99.94 Average: 19 0.05 0.00 0.23 0.03 0.00 0.00 0.00 0.00 0.00 99.68 Average: 20 0.03 0.00 0.13 0.00 1.05 7.76 0.00 0.00 0.00 91.03 Average: 21 0.02 0.00 0.03 0.00 0.00 0.00 0.00 0.00 0.00 99.94 Average: 22 0.01 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 99.98 Average: 23 0.02 0.00 0.02 0.09 0.00 0.00 0.00 0.00 0.00 99.86 Average: 24 0.00 0.00 0.12 0.00 3.17 15.49 0.00 0.00 0.00 81.22 Average: 25 0.00 0.00 0.18 0.00 0.00 0.00 0.00 0.00 0.00 99.82 Average: 26 0.03 0.00 0.06 0.00 0.00 0.04 0.00 0.00 0.00 99.86 Average: 27 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.99 Average: 28 0.02 0.00 0.03 0.00 0.00 0.05 0.00 0.00 0.00 99.90 Average: 29 0.44 0.00 0.94 0.00 0.02 0.01 0.00 0.00 0.00 98.59 Average: 30 0.01 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 99.97 Average: 31 0.01 0.00 0.08 0.05 0.00 0.00 0.00 0.00 0.00 99.85 Average: 32 0.05 0.00 0.08 0.00 0.00 0.00 0.00 0.00 0.00 99.85 Average: 33 0.03 0.00 0.04 0.00 0.00 0.00 0.00 0.00 0.00 99.91 Average: 34 0.00 0.00 0.10 0.00 1.33 6.77 0.00 0.00 0.00 91.79 Average: 35 0.06 0.00 0.15 0.00 0.01 0.00 0.00 0.00 0.00 99.77 Average: 36 0.06 0.00 0.10 0.00 1.30 10.50 0.00 0.00 0.00 88.05 Average: 37 0.02 0.00 0.07 0.00 0.02 0.02 0.00 0.00 0.00 99.86 Average: 38 0.05 0.00 0.14 0.00 1.35 10.39 0.00 0.00 0.00 88.07 Average: 39 0.02 0.00 0.13 0.18 0.01 0.01 0.00 0.00 0.00 99.64
其實，看到前面mpstat的顯示，如果對網絡卡多佇列，RPS／RFS很熟悉，就會意識到他們在這裡不適用。 一句話解釋：這個kvm虛擬機器只有一個網絡卡，有網路包到達這個網絡卡後，它會給某一個cpu(如果沒有設定親和性，這個可以認為是隨機的一個cpu，然後就會一直固定在這個cpu上)發中斷，通知該cpu來處理這個包，然後cpu就會觸發一個軟中斷把該包送到tcp/ip協議棧(對於tcp包而言)裡去處理，該包被放入某一個socket的receive buffer中（如果是一個數據包），軟中斷結束。
%soft就是指的CPU耗在軟中斷處理上的時間。
可以看到核1的%soft很高，其他的核的%soft基本為0.
我們想到的方法是網絡卡多佇列，或者RPS／RFS。用這種手段來把網絡卡軟中斷給均攤到其他的核上去。
2. 檢視每個CPU上的中斷變化：
watch -d -n 1 'cat /proc/interrupts'
watch -d -n 1 'cat /proc/interrupts |grep -E "enp5s0f3"'Every 1.0s: cat /proc/interrupts |grep -E "enp5s0f3" localhost.localdomain: Thu Jan 21 20:46:10 2021 95: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 2627584-edge enp5s0f3 96: 0 0 568542973 0 16929902 0 3284654 0 182591750 0 2263993 0 3753337 0 74401525 0 216870268 0 552241418 0 15685 0 3674387 0 112125 0 2881645 0 3546554 0 5439101 0 5256804 0 45385219 0 1656062 0 69829199 0 IR-PCI-MSI 2627585-edge enp5s0f3-TxRx-0 97: 0 0 4607649 0 1479663 0 2786155 0 46627891 0 736533668 0 3750151 0 106562556 0 3765484 0 1693397 0 554069952 0 508746 0 18870 0 3239848 0 9600866 0 71546816 0 1272910 0 13978513 0 12613923 0 13481427 0 IR-PCI-MSI 2627586-edge enp5s0f3-TxRx-1 98: 0 0 0 0 510 0 11880 0 1803 0 14090 0 96163845 0 0 0 360800202 0 3084618 0 696758357 0 9 0 570 0 374160885 0 95515 0 1907818 0 6936473 0 190 0 196122799 0 8473283 0 IR-PCI-MSI 2627587-edge enp5s0f3-TxRx-2 99: 0 0 2245 0 8332855 0 36497 0 6115511 0 25791519 0 0 0 7455 0 167804562 0 543627622 0 0 0 18982920 0 541808143 0 2219164 0 294297212 0 1431021 0 27269 0 10280 0 114868812 0 6056541 0 IR-PCI-MSI 2627588-edge enp5s0f3-TxRx-3 100: 0 0 9155 0 69422794 0 18162315 0 4466926 0 189110717 0 0 0 11995261 0 41574430 0 1680987 0 8875 0 1525786 0 521395 0 2322062 0 364377557 0 823854973 0 40 0 17336324 0 940570 0 22848418 0 IR-PCI-MSI 2627589-edge enp5s0f3-TxRx-4 101: 0 0 4931536 0 53429823 0 631213311 0 5700165 0 371677111 0 12402 0 1100738 0 65434558 0 896685 0 378432869 0 7574076 0 5807676 0 5615317 0 8297189 0 14207899 0 14640 0 194774810 0 3579341 0 5431105 0 IR-PCI-MSI 2627590-edge enp5s0f3-TxRx-5 102: 0 0 761536 0 18302532 0 60965 0 2862940 0 7080043 0 4744878 0 295883 0 0 0 3719027 0 0 0 124772699 0 859454102 0 46192 0 2767191 0 6519441 0 0 0 700 0 684847631 0 8185540 0 IR-PCI-MSI 2627591-edge enp5s0f3-TxRx-6 103: 0 0 9565 0 4940735 0 1352508 0 55642 0 61360 0 554759678 0 82639147 0 360683 0 1632161 0 0 0 2458 0 25397625 0 4196539 0 185625265 0 1520 0 4347106 0 848968539 0 65254 0 32430039 0 IR-PCI-MSI 2627592-edge enp5s0f3-TxRx-7
網絡卡多佇列配置如下：
[[email protected] ~]# cat /proc/irq/99/smp_affinity
0000,00000400
[[email protected] ~]# cat /proc/irq/100/smp_affinity
0040,00000000
[[email protected] ~]# cat /proc/irq/101/smp_affinity
0004,00000000
[[email protected] ~]# cat /proc/irq/102/smp_affinity
0000,01000000
[[email protected] ~]# cat /proc/irq/103/smp_affinity
0000,00001000
[[email protected] ~]# cat /proc/irq/104/smp_affinity
0000,10000000
[[email protected] ~]# cat /proc/irq/103/smp_affinity_list
12
[[email protected] ~]# cat /proc/irq/102/smp_affinity_list
24
[[email protected] ~]# cat /proc/irq/101/smp_affinity_list
34
[[email protected] ~]# cat /proc/irq/100/smp_affinity_list
38
[[email protected] ~]# cat /proc/irq/99/smp_affinity_list
10
[[email protected] ~]# cat /proc/irq/98/smp_affinity_list
36
[[email protected] ~]# cat /proc/irq/97/smp_affinity_list
20
[[email protected] ~]# cat /proc/irq/96/smp_affinity_list
2
[[email protected] ~]# echo 0080,00000000 > /proc/irq/103/smp_affinity
[[email protected] ~]# echo 0040,00000000 > /proc/irq/102/smp_affinity
[[email protected] ~]# echo 0020,00000000 > /proc/irq/101/smp_affinity
[[email protected] ~]# echo 0010,00000000 > /proc/irq/100/smp_affinity
[[email protected] ~]# cat /proc/irq/103/smp_affinity_list
39
[[email protected] ~]# cat /proc/irq/102/smp_affinity_list
38
[[email protected] ~]# cat /proc/irq/101/smp_affinity_list
37
[[email protected] ~]# cat /proc/irq/100/smp_affinity_list
36
[[email protected] ~]# echo 000f,00000000 > /proc/irq/99/smp_affinity
[[email protected] ~]# echo 0008,00000000 > /proc/irq/98/smp_affinity
[[email protected] ~]# echo 0004,00000000 > /proc/irq/97/smp_affinity
[[email protected] ~]# echo 0002,00000000 > /proc/irq/96/smp_affinity需要先把irqbalance關閉：
關閉方式：
停止irqbalance服務。
systemctl stop irqbalance.service
關閉irqbalance服務。
systemctl disable irqbalance.service
檢視irqbalance服務狀態是否已關閉。
systemctl status irqbalance.service
網絡卡多佇列配置指令碼：
cat /proc/irq/96/smp_affinity
cat /proc/irq/97/smp_affinity
cat /proc/irq/98/smp_affinity
cat /proc/irq/99/smp_affinity
cat /proc/irq/100/smp_affinity
cat /proc/irq/101/smp_affinity
cat /proc/irq/102/smp_affinity
cat /proc/irq/103/smp_affinity
cat /proc/irq/103/smp_affinity_list
cat /proc/irq/102/smp_affinity_list
cat /proc/irq/101/smp_affinity_list
cat /proc/irq/100/smp_affinity_list
cat /proc/irq/99/smp_affinity_list
cat /proc/irq/98/smp_affinity_list
cat /proc/irq/97/smp_affinity_list
cat /proc/irq/96/smp_affinity_list
echo 0080,00000000 > /proc/irq/103/smp_affinity
echo 0040,00000000 > /proc/irq/102/smp_affinity
echo 0020,00000000 > /proc/irq/101/smp_affinity
echo 0010,00000000 > /proc/irq/100/smp_affinity
echo 0008,00000000 > /proc/irq/99/smp_affinity
echo 0004,00000000 > /proc/irq/98/smp_affinity
echo 0002,00000000 > /proc/irq/97/smp_affinity
echo 0001,00000000 > /proc/irq/96/smp_affinity
cat /proc/irq/96/smp_affinity
cat /proc/irq/97/smp_affinity
cat /proc/irq/98/smp_affinity
cat /proc/irq/99/smp_affinity
cat /proc/irq/100/smp_affinity
cat /proc/irq/101/smp_affinity
cat /proc/irq/102/smp_affinity
cat /proc/irq/103/smp_affinity
cat /proc/irq/103/smp_affinity_list
cat /proc/irq/102/smp_affinity_list
cat /proc/irq/101/smp_affinity_list
cat /proc/irq/100/smp_affinity_list
cat /proc/irq/99/smp_affinity_list
cat /proc/irq/98/smp_affinity_list
cat /proc/irq/97/smp_affinity_list
cat /proc/irq/96/smp_affinity_list