Centos雙網絡卡冗餘繫結
1. 前言
關於雙網絡卡繫結,前前後後踩過不少的坑,雖然這是 RHCE 中的一道題,但是在實踐中碰到問題也夠喝一壺的。
在實踐中,虛擬機器、物理機都做過,但是不盡相同,大部分的坑也集中在這裡,本文長期更新關於網絡卡繫結中遇到的問題及處理方法。
現在的伺服器預設都配備 4 張 千兆網絡卡,在生產環境中,無論遇到多大的困難,都必須做到雙網絡卡繫結(至少要實現主備模式),最基本的主備這是基本原則。因此在前期的規劃,網路裝置(交換機)是必須要考慮到的。
2. 概念
概念性的東西的最容易遺忘的,這裡得反覆強調:
鏈路聚合最常見的三種模式:
- mode0(平衡負載模式):兩張網絡卡同時均衡工作,且自動備援,但是需要在與伺服器本地網絡卡相連的交換機裝置上進行埠聚合來支援繫結技術。
- mode1(自動備援模式):1張網絡卡工作,在工作網絡卡故障時,第二張網絡卡替換故障網絡卡,單純的主備模式。
- mode6(平衡負載模式):兩張網絡卡均衡工作,且自動備援,無需交換機裝置提供輔助支援。
以上三種模式,最推薦的當然是 mode6 技能負載均衡又能主備切換,最最重要的是不用浪費時間去聯絡網路工程師。記住上面三種常見模式就夠用了。
3. 實踐#
3.1 虛擬機器網絡卡繫結
環境介紹:
虛擬化軟體:VMware® Workstation 15 Pro
虛擬機器作業系統:CentOS Linux release 7.7.1908 (Core)
在 Centos 7 中有兩種技術來實現網絡卡繫結:teaming 和 bonding ,下面通過三種方式來實現網絡卡繫結:
3.1.1 手動配置,使用 bond 技術#
檢視物理網絡卡及連線狀態
1 2 3 4 5 6 7 8 9 10 11 |
[[email protected] ~] #nmcli dev
DEVICE TYPE STATE CONNECTION
eth0 ethernet connected System eth0
eth1 ethernet connected Wired connection 1
eth2 ethernet connected Wired connection 2
lo loopback unmanaged --
[[email protected] ~] #nmcli con
NAME UUID TYPE DEVICE
System eth0 078df8ff-b848-4c23-b212-3213a74bc5d5 ethernet eth0
Wired connection 1 24d053c5-d908-3e0d-9e61-4d92efcd6f3b ethernet eth1
Wired connection 2 b4cea59c-59ea-3b6c-a343-e2578ede5034 ethernet eth2
|
通過上面的命令得知:該主機有3張網絡卡,1張本地迴環lo 且 三張物理網絡卡都處於連線中,檢視網絡卡配置檔案:
1 2 3 |
[[email protected] ~] #ll /etc/sysconfig/network-scripts/ifcfg-*
-rw-r--r-- 1 root root 259 Mar 30 19:21 /etc/sysconfig/network-scripts/ifcfg-eth0
-rw-r--r--. 1 root root 254 Mar 29 2019 /etc/sysconfig/network-scripts/ifcfg-lo
|
檢視配置網絡卡配置檔案卻只有 eth0 的, eth1 和 eth2 沒有。這裡可以通過新建連線來生成配置檔案,比自己手動編寫方便太多了,建議使用。
首先通過連線名 ‘Wired connection 1’ ‘Wired connection 2’ 刪除連線
1 2 3 4 5 6 7 8 9 10 11 |
[[email protected] ~] #nmcli con
NAME UUID TYPE DEVICE
System eth0 078df8ff-b848-4c23-b212-3213a74bc5d5 ethernet eth0
Wired connection 1 24d053c5-d908-3e0d-9e61-4d92efcd6f3b ethernet eth1
Wired connection 2 b4cea59c-59ea-3b6c-a343-e2578ede5034 ethernet eth2
[[email protected] ~] #nmcli con del 'Wired connection 1' 'Wired connection 2'
Connection 'Wired connection 1' (24d053c5-d908-3e0d-9e61-4d92efcd6f3b) successfully deleted.
Connection 'Wired connection 2' (b4cea59c-59ea-3b6c-a343-e2578ede5034) successfully deleted.
[[email protected] ~] #nmcli con
NAME UUID TYPE DEVICE
System eth0 078df8ff-b848-4c23-b212-3213a74bc5d5 ethernet eth0
|
然後在通過物理網絡卡 eth1 和 eth2 重建連線
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
[[email protected] ~] #nmcli con
NAME UUID TYPE DEVICE
System eth0 078df8ff-b848-4c23-b212-3213a74bc5d5 ethernet eth0
[[email protected] ~] #nmcli con add type ethernet ifname eth1 con-name eth1
Connection 'eth1' (8c47d2f2-1373-425e-b6e2-fa5e6b9eb8f6) successfully added.
[[email protected] ~] #nmcli con add type ethernet ifname eth2 con-name eth2
Connection 'eth2' (5d8dc9b0-3b4f-4c6d-919b-a7754d86b1f8) successfully added.
[[email protected] ~] #nmcli con
NAME UUID TYPE DEVICE
System eth0 078df8ff-b848-4c23-b212-3213a74bc5d5 ethernet eth0
eth1 8c47d2f2-1373-425e-b6e2-fa5e6b9eb8f6 ethernet eth1
eth2 5d8dc9b0-3b4f-4c6d-919b-a7754d86b1f8 ethernet eth2
[[email protected] ~] #ll /etc/sysconfig/network-scripts/ifcfg-*
-rw-r--r-- 1 root root 259 Mar 30 19:21 /etc/sysconfig/network-scripts/ifcfg-eth0
-rw-r--r-- 1 root root 278 May 21 22:58 /etc/sysconfig/network-scripts/ifcfg-eth1
-rw-r--r-- 1 root root 278 May 21 22:58 /etc/sysconfig/network-scripts/ifcfg-eth2
-rw-r--r--. 1 root root 254 Mar 29 2019 /etc/sysconfig/network-scripts/ifcfg-lo
|
新建網絡卡連線命令:
1 |
nmcli con add type ethernet ifname eth2 con-name eth2
|
ifname : 物理網絡卡名,通過 nmcli dev 檢視
con-name:網絡卡配置檔名,邏輯網絡卡名
通過上面的配置已得到三張網絡卡的配置檔案,接下來進行配置檔案的修改實現網絡卡繫結
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
----------------------eth1----------------------
[[email protected] /etc/sysconfig/network-scripts ] #vim ifcfg-eth1
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE= yes
IPV4_FAILURE_FATAL=no
NAME=eth1
UUID=8c47d2f2-1373-425e-b6e2-fa5e6b9eb8f6
DEVICE=eth1
ONBOOT= yes
MASTER=bond6
SLAVE= yes
----------------------eth2----------------------
[[email protected] /etc/sysconfig/network-scripts ] #vim ifcfg-eth2
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE= yes
IPV4_FAILURE_FATAL=no
NAME=eth2
UUID=5d8dc9b0-3b4f-4c6d-919b-a7754d86b1f8
DEVICE=eth2
ONBOOT= yes
MASTER=bond6
SLAVE= yes
|
修改及新增的部分用黃(和諧)色標註,新增 ifcfg-bond6 配置檔案如下:
1 2 3 4 5 6 7 8 9 10 11 |
[[email protected] /etc/sysconfig/network-scripts ] #vim ifcfg-bond6
TYPE=Bond
BOOTPROTO=none
DEVICE=bond6
ONBOOT= yes
IPADDR=192.168.1.100
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=114.114.114.114
BONDING_OPTS= 'miimon=100 mode=6 fail_over_mac=1'
|
這裡需要特別注意的地方:BONDING_OPTS='miimon=100 mode=6 fail_over_mac=1' ,如果是虛擬機器,fail_over_mac=1是必須要帶上的,否則vmware會出現告警資訊,配置起來能正常用,但是在進行準備切換時,是無法進行的。切記!
vmware 出現這樣的提示基本可以確定 fail_over_mac 沒有生效,所配置的網絡卡也無法做到故障切換。
注意:在vmware 虛擬機器環境中,常用的三種方式(mode-0 mode-1 mode-6) 只有 mode 1 實現了故障切換。
mode 1 - ifcfg-bond6 配置如下:
1 2 3 4 5 6 7 8 9 10 11 |
[[email protected] /etc/sysconfig/network-scripts ] #vim ifcfg-bond6
TYPE=Bond
BOOTPROTO=none
DEVICE=bond6
ONBOOT= yes
IPADDR=192.168.1.100
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=114.114.114.114
BONDING_OPTS= "miimon=100 mode=1 fail_over_mac=1"
|
bond 配置檔案中可以通過 mode = 模式號 來進行切換,這裡修改為 mode-1 模式,檢視配置資訊:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
[[email protected] /etc/sysconfig/network-scripts ] #cat /proc/net/bonding/bond6
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: fault-tolerance (active-backup) (fail_over_mac active)
Primary Slave: None
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:f4:f4:0f
Slave queue ID: 0
Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:f4:f4:19
Slave queue ID: 0
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
[[email protected] /etc/sysconfig/network-scripts ] #ethtool bond6
Settings for bond6:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 1000Mb /s
Duplex: Full
Port: Other
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Link detected: yes
|
可以看到橙色部分:
1 |
Bonding Mode: fault-tolerance (active-backup) (fail_over_mac active) 說明這裡 fail_over_mac 生效了。
|
1 |
Currently Active Slave: eth1 當前活動的網絡卡是第一張網絡卡,也就是當第一張網絡卡 down,就會切換到 eth2
|
進行網絡卡故障切換,虛擬機器可以通過 ifdown / ifup 來實現
3.1.2 通過 nmcli 命令實現網絡卡繫結,使用技術:bonding
網絡卡資訊及連線狀態如下:
1 2 3 4 5 6 7 8 9 |
[[email protected] ~] #nmcli dev
DEVICE TYPE STATE CONNECTION
eth0 ethernet connected System eth0
eth1 ethernet disconnected --
eth2 ethernet disconnected --
lo loopback unmanaged --
[[email protected] ~] #nmcli con
NAME UUID TYPE DEVICE
System eth0 078df8ff-b848-4c23-b212-3213a74bc5d5 ethernet eth0
|
目前只有 eth0 網絡卡連線網路
通過nmcli 建立 bond1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
[[email protected] ~] #nmcli con add type bond ifname bond1 con-name bond1 mode 1 ipv4.method manual ipv4.address 192.168.1.100/24 ipv4.gateway 192.168.1.1 ipv4.dns 114.114.114.114
Connection 'bond1' (f922008c-75b1-4f5e-a27d-d3e054acde0d) successfully added.
[[email protected] ~] #nmcli con add type bond-slave ifname eth1 con-name bond1-port1 master bond1
Connection 'bond1-port1' (06489850-82d7-4827-bb16-13d5abb84c58) successfully added.
[[email protected] ~] #nmcli con add type bond-slave ifname eth2 con-name bond1-port2 master bond1
Connection 'bond1-port2' (80f43a85-32cc-4302-b0f1-8cb87e9e3e17) successfully added.
[[email protected] ~] #ll /etc/sysconfig/network-scripts/ifcfg-*
-rw-r--r-- 1 root root 399 May 21 23:50 /etc/sysconfig/network-scripts/ifcfg-bond1
-rw-r--r-- 1 root root 119 May 21 23:50 /etc/sysconfig/network-scripts/ifcfg-bond1-port1
-rw-r--r-- 1 root root 119 May 21 23:50 /etc/sysconfig/network-scripts/ifcfg-bond1-port2
-rw-r--r-- 1 root root 259 Mar 30 19:21 /etc/sysconfig/network-scripts/ifcfg-eth0
-rw-r--r--. 1 root root 254 Mar 29 2019 /etc/sysconfig/network-scripts/ifcfg-lo
[[email protected] ~] #nmcli con
NAME UUID TYPE DEVICE
System eth0 078df8ff-b848-4c23-b212-3213a74bc5d5 ethernet eth0
bond1 f922008c-75b1-4f5e-a27d-d3e054acde0d bond bond1
bond1-port2 80f43a85-32cc-4302-b0f1-8cb87e9e3e17 ethernet eth2
bond1-port1 06489850-82d7-4827-bb16-13d5abb84c58 ethernet --
|
建立 bond1 並將 兩張物理網絡卡新增到 bond1 , 檢視配置檔案已經生成。
注意:在虛擬機器環境中一定要加上 fail_over_mac = 1
1 2 |
sed -i '/BONDING_OPTS/d' ifcfg-bond1
echo 'BONDING_OPTS="miimon=100 mode=1 fail_over_mac=1"' >> ifcfg-bond1
|
重啟網路
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
[[email protected] /etc/sysconfig/network-scripts ] #systemctl restart network
[[email protected] /etc/sysconfig/network-scripts ] #cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: fault-tolerance (active-backup) (fail_over_mac active)
Primary Slave: None
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:f4:f4:0f
Slave queue ID: 0
Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:f4:f4:19
Slave queue ID: 0
|
成功實現了虛擬機器裝網絡卡繫結,測試和前一種方式一樣。
3.1.3 通過 nmcli 命令實現網絡卡繫結,使用技術:team[ 在Centos 7 之後的版本推薦使用這種方式 ]
網絡卡資訊及連線狀態如下:
1 2 3 4 5 6 7 8 9 |
[[email protected] ~] #nmcli dev
DEVICE TYPE STATE CONNECTION
eth0 ethernet connected System eth0
eth1 ethernet disconnected --
eth2 ethernet disconnected --
lo loopback unmanaged --
[[email protected] ~] #nmcli con
NAME UUID TYPE DEVICE
System eth0 078df8ff-b848-4c23-b212-3213a74bc5d5 ethernet eth0
|
目前只有eth0 建立連線,通過nmcli 新增 team1
1 2 3 4 5 6 |
[[email protected] ~] #nmcli con add type team ifname team1 con-name team1 config '{"runner":{"name":"activebackup", "hwaddr_policy":"by_active"}}' \ipv4.method manual ipv4.address 192.168.1.100/24 ipv4.gateway 192.168.1.1 ipv4.dns 114.114.114.114
Connection 'team1' (25434d73-0224-47e1-80f4-bbb3faae53fe) successfully added.
[[email protected] ~] #nmcli con add type team-slave ifname eth1 con-name team1-port1 master team1
Connection 'team1-port1' (c9e216c2-7668-487b-b6cd-e67631b8a3f9) successfully added.
[[email protected] ~] #nmcli con add type team-slave ifname eth2 con-name team1-port2 master team1
Connection 'team1-port2' (cf8d3150-b3e0-433c-8c4e-ba4feaa4bd6d) successfully added.
|
配置完畢,在虛擬機器環境中一定要注意:"hwaddr_policy":"by_active" 這個引數意義和 fail_over_mac =1 是一致的,在虛擬機器環境中必須新增上,否則網絡卡高可用失敗。
檢視網絡卡連線狀態和繫結狀態:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
[[email protected] ~] #nmcli con
NAME UUID TYPE DEVICE
System eth0 078df8ff-b848-4c23-b212-3213a74bc5d5 ethernet eth0
team1 25434d73-0224-47e1-80f4-bbb3faae53fe team team1
team1-port1 c9e216c2-7668-487b-b6cd-e67631b8a3f9 ethernet eth1
team1-port2 cf8d3150-b3e0-433c-8c4e-ba4feaa4bd6d ethernet eth2
[[email protected] ~] #teamdctl team1 st
setup:
runner: activebackup
ports:
eth1
link watches:
link summary: up
instance[link_watch_0]:
name: ethtool
link: up
down count: 0
eth2
link watches:
link summary: up
instance[link_watch_0]:
name: ethtool
link: up
down count: 0
runner:
active port: eth1
|
本次建立, 網絡卡直接建立了連線如果沒有建立連線,也就是 nmcli con 檢視 DEVICE 項沒有物理網絡卡連線,則執行如下:
1 2 3 4 5 6 |
[[email protected] ~] #nmcli con up team1-port1
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/74 )
[[email protected] ~] #nmcli con up team1-port2
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/75 )
[[email protected] ~] #nmcli con up team1
Connection successfully activated (master waiting for slaves) (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/76 )
|
做完了如上配置,建議直接重啟網絡卡檢視是否有報錯,或者無法連線的情況:
測試
直接通過 ifdown 掉正在使用的 eth1 網路僅僅延遲了不到1ms的時間就恢復了正常,現在檢視網路繫結狀態:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
[[email protected] ~] #teamdctl team1 st
setup:
runner: activebackup
ports:
eth2
link watches:
link summary: up
instance[link_watch_0]:
name: ethtool
link: up
down count: 0
runner:
active port: eth2
|
活動網絡卡已經切換到 eth2 ,重啟 eth1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
[[email protected] ~] #ifup eth1
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/87 )
[[email protected] ~] #teamdctl team1 st
setup:
runner: activebackup
ports:
eth1
link watches:
link summary: up
instance[link_watch_0]:
name: ethtool
link: up
down count: 0
eth2
link watches:
link summary: up
instance[link_watch_0]:
name: ethtool
link: up
down count: 0
runner:
active port: eth2
|
啟動 eth1 以後,活動網絡卡依然是 eth2 ,而 eth1 則成為備用網絡卡。
3.1.4 總結
三種方式:
- 手動配置 - bonding
- nmcli 配置 - bonding
- nmcli 配置 - team
在 Centos 7 以後的版本建議使用 nmcli - team 的方式,簡便,更易管理和檢視。
在虛擬機器的環境中,使用 bond 和 team 都只能實現 mode-1模式的故障切換。
- bond 虛擬機器中必要引數: fail_over_mac=1
- team 虛擬機器中必要引數:"hwaddr_policy":"by_active"