記一次bond引起的網絡故障
阿新 • • 發佈：2019-04-28
policy for tput nat etc mon 6.0 arping 技術分享
本案中3個關鍵服務器
物理服務器：192.168.6.63，簡稱P，(Physical server)
KVM-VM：192.168.6.150，是物理服務器P上的一個KVM虛機，簡稱VM
NAS：外部NAS服務器，用來做ping/arp測試服務器，簡稱NAS
物理服務器P的配置：
#uname -a Linux cz63 4.15.18-11-pve #1 SMP PVE 4.15.18-34 (Mon, 25 Feb 2019 14:51:06 +0100) x86_64 GNU/Linux #cat /etc/network/interfaces auto lo iface lo inet loopback auto enp3s0f0 iface enp3s0f0 inet manual auto enp3s0f1 iface enp3s0f1 inet manual auto ens1f0 iface ens1f0 inet manual auto ens1f1 iface ens1f1 inet manual auto bond0 iface bond0 inet manual bond-slaves enp3s0f0 enp3s0f1 bond-miimon 100 bond-mode balance-rr auto bond1 iface bond1 inet manual bond-slaves ens1f0 ens1f1 bond-miimon 100 bond-mode balance-rr auto vmbr0 iface vmbr0 inet static address 192.168.6.63 netmask 255.255.255.0 gateway 192.168.6.1 bridge-ports bond0 bridge-stp off bridge-fd 0 auto vmbr1 iface vmbr1 inet static address 10.1.1.63 netmask 255.255.255.0 bridge-ports bond1 bridge-stp off bridge-fd 0 #brctl show bridge name bridge id STP enabled interfaces vmbr0 8000.ac1f6b342094 no bond0 tap401000001i0 vmbr1 8000.74a4b500e768 no bond1
安裝KVM-VM，之後VM與外部服務器的連接非常不穩定，90%情況下會出現ping不可達。這裏測試了centos、ubuntu、win7都是一樣的不穩定。
在VM內執行ping 192.168.6.40（NAS）
結果ping不通。
此時，在P上執行
tcpdump -leni vmbr0 arp
tcpdump -leni tap401000001i0 arp
發現情況如下：
P上 #tcpdump -leni vmbr0 arp | grep 2a:f0:5f:ae:c9:8b 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46 00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46 00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46 00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46 00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46 00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46 P上 #tcpdump -leni tap401000001i0 arp | grep 2a:f0:5f:ae:c9:8b 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28 NAS上 # tcpdump -leni vmx0 arp | grep c9:8b 02:03:50.930907 2a:f0:5f:ae:c9:8b > 00:50:56:87:86:b9, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 tell 192.168.6.150, length 46 02:03:50.930923 00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 42: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 28 02:04:02.669823 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46 02:04:02.670131 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46 02:04:03.670770 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46 02:04:03.671059 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46 02:04:04.672736 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46 02:04:04.672992 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46 02:04:06.671878 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46 02:04:06.672021 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46 02:04:07.674726 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46 02:04:07.674773 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46 02:04:08.676733 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46 02:04:08.676868 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46 02:04:10.673678 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46 02:04:10.674026 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46
vmbr0上發現了2個request包和1個reply包
tap401000001i0上僅發現了request包，而且length不一致，未發現reply包。
NAS上一切正常
看到的現象就是VM在發送icmp之前需要先獲得NAS的MAC地址，所以需要先通過ARP廣播獲取到NAS的MAC地址。
- P的vmbr0異常：接收到2個request，1個reply
- P的vmbr0異常：接收到reply未發送至tap401000001i0
由於以上2個原因，導致VM無法收到reply，所以icmp不能進行，ping失敗。
問題定位在數據鏈路層的ARP協議上。
在VM上執行arping -c 10 192.168.6.40
現象依據。再次證明以上2個現象。
# brctl show
bridge name bridge id STP enabled interfaces
vmbr0 8000.ac1f6b342094 no bond0
tap401000001i0
vmbr1 8000.74a4b500e768 no bond1
# brctl showstp vmbr0
vmbr0
bridge id 8000.ac1f6b342094
designated root 8000.ac1f6b342094
root port 0 path cost 0
max age 20.00 bridge max age 20.00
hello time 2.00 bridge hello time 2.00
forward delay 0.00 bridge forward delay 0.00
ageing time 300.00
hello timer 0.00 tcn timer 0.00
topology change timer 0.00 gc timer 4.18
flags
bond0 (1)
port id 8001 state forwarding
designated root 8000.ac1f6b342094 path cost 4
designated bridge 8000.ac1f6b342094 message age timer 0.00
designated port 8001 forward delay timer 0.00
designated cost 0 hold timer 0.00
flags
tap401000001i0 (2)
port id 8002 state forwarding
designated root 8000.ac1f6b342094 path cost 100
designated bridge 8000.ac1f6b342094 message age timer 0.00
designated port 8002 forward delay timer 0.00
designated cost 0 hold timer 0.00
flags
# brctl showmacs vmbr0
port no mac addr is local? ageing timer
1 2a:f0:5f:ae:c9:8b no 1.13
1 00:50:56:87:e3:6d no 0.00
# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
# ebtables -L
Bridge table: filter
Bridge chain: INPUT, entries: 0, policy: ACCEPT
Bridge chain: FORWARD, entries: 0, policy: ACCEPT
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
# ip rule show
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
# ip route show table all
default via 192.168.6.1 dev vmbr0 onlink
10.1.1.0/24 dev vmbr1 proto kernel scope link src 10.1.1.63
192.168.6.0/24 dev vmbr0 proto kernel scope link src 192.168.6.63
broadcast 10.1.1.0 dev vmbr1 table local proto kernel scope link src 10.1.1.63
local 10.1.1.63 dev vmbr1 table local proto kernel scope host src 10.1.1.63
broadcast 10.1.1.255 dev vmbr1 table local proto kernel scope link src 10.1.1.63
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.6.0 dev vmbr0 table local proto kernel scope link src 192.168.6.63
local 192.168.6.63 dev vmbr0 table local proto kernel scope host src 192.168.6.63
broadcast 192.168.6.255 dev vmbr0 table local proto kernel scope link src 192.168.6.63
# bridge vlan show
port vlan ids
bond0 1 PVID Egress Untagged
vmbr0 1 PVID Egress Untagged
bond1 1 PVID Egress Untagged
vmbr1 1 PVID Egress Untagged
tap401000001i0 1 PVID Egress Untagged以上一切正常。
sequenceDiagram
A->>B: How are you?
B->>A: Great!
記一次bond引起的網絡故障