KVM虛擬機&openVSwitch雜記(2)
阿新 • • 發佈:2018-04-11
kvm 外部通信 openvswitch一、兩個計算機節點(宿主機)的虛擬機通信【GRE&oepnVSwitch】
1、移除之前的配置
1.1 關閉之前master1上的三臺虛擬機
[root@master1 ~]# for i in `ps aux | grep qemu-kvm | grep -v grep | awk ‘{print $2}‘`;do kill $i;done 確保ovs(openVSwitch上,之前的虛擬機網口已經刪除) [root@master1 ~]# ovs-vsctl show 85eb6797-6dd5-4c5f-aaf8-b7b242b633ae Bridge br-in Port "s0" trunks: [10] Interface "s0" Port br-in Interface br-in type: internal Bridge br-test Port br-test Interface br-test type: internal Port "s1" Interface "s1" 如果有刪除失敗,直接用命令刪除,例: [root@master1 ~]# ovs-vsctl del-port br-in vif1.0 移除之前的br-test和s0,s1 [root@master1 ~]# ovs-vsctl del-port br-test s1 [root@master1 ~]# ovs-vsctl del-port br-in s0 [root@master1 ~]# ovs-vsctl del-br br-test [root@master1 ~]# ovs-vsctl show 85eb6797-6dd5-4c5f-aaf8-b7b242b633ae Bridge br-in Port br-in Interface br-in type: internal
1.2 復制啟動和刪除網卡腳本到master2上面
[root@master1 ~]# cat /etc/if-up #!/bin/bash bridge=br-in if [ -n "$1" ]; then ip link set $1 up sleep 1 ovs-vsctl add-port $bridge $1 [ $? -eq 0 ] && exit 0 || exit 1 else echo "Error:no port specified." exit 2 fi [root@master1 ~]# cat /etc/if-down #!/bin/bash bridge=br-in if [ -n "$1" ]; then ip link set $1 down sleep 1 ovs-vsctl del-port $bridge $1 [ $? -eq 0 ] && exit 0 || exit 1 else echo "Error:no port specified." exit 2 fi 復制到master2: [root@master1 ~]# scp -p /etc/if-up /etc/if-down [email protected]:/etc/ 查看: [root@master2 ~]# ls /etc/if-up /etc/if-down /etc/if-down /etc/if-up
1.3 master2也裝上qemu-kvm和openvswitch
[root@master2 ~]# yum install qemu-kvm [root@master2 ~]# ln -sv /usr/libexec/qemu-kvm /usr/bin/ 安裝依賴包: yum -y install make gcc openssl-devel autoconf automake rpm-build redhat-rpm-config oepnVSwtich 下載包: [root@master2 ~]# wget http://openvswitch.org/releases/openvswitch-2.5.0.tar.gz 準備構建rpm包 [root@master2 ~]# mkdir -p ~/rpmbuild/SOURCES [root@master2 ~]# cp openvswitch-2.5.0.tar.gz ~/rpmbuild/SOURCES/ 解壓: [root@master2 ~]# tar xf openvswitch-2.5.0.tar.gz 修改配置文件 [root@master1 ~]# sed ‘s/openvswitch-kmod, //g‘ openvswitch-2.5.0/rhel/openvswitch.spec > openvswitch-2.5.0/rhel/openvswitch_no_kmod.spec 構建RPM包: [root@master2 ~]# rpmbuild -bb --nocheck ~/openvswitch-2.5.0/rhel/openvswitch_no_kmod.spec 安裝 [root@master2 ~]# yum localinstall ~/rpmbuild/RPMS/x86_64/openvswitch-2.5.0-1.x86_64.rpm 啟動服務 [root@master2 ~]# systemctl start openvswitch.service
2、配置兩臺宿主機的虛擬機互通
2.1 master2配置
[root@master2 ~]# ovs-vsctl add-br br-in
[root@master2 ~]# ovs-vsctl list-br
br-in
準備目錄存放磁盤映像文件:
[root@master2 ~]# mkdir -pv /vm/images
[root@master2 images]# mv cirros-no_cloud-0.3.0-i386-disk.img c1.img
[root@master2 images]# mv cirros-no_cloud-0.3.0-x86_64-disk.img c2.img
[root@master2 images]# ls
c1.img c2.img
2.2 搭建DHCP服務,創建虛擬機自動獲取IP
創建網絡名稱空間:
[root@master2 ~]# ip netns add r0
創建一對網卡,rif0連接名稱空間,sif0連接到橋上:
[root@master2 ~]# ip link add sif0 type veth peer name rif0
[root@master2 ~]# ip link set sif0 up
rif0網卡添加到r0網絡名稱空間:
[root@master2 ~]# ip link set rif0 netns r0
[root@master2 ~]# ip netns exec r0 ip link set rif0 up
sif0網卡添加到br-in橋:
[root@master2 ~]# ovs-vsctl add-port br-in sif0
[root@master2 ~]# ovs-vsctl show
35f025aa-0395-4fa5-b111-553aa9c651e5
Bridge br-in
Port "sif0"
Interface "sif0"
Port br-in
Interface br-in
給rif0添加IP地址
[root@master2 ~]# ip netns exec r0 ip addr add 10.0.4.254/24 dev rif0
啟動DHCP服務:
[root@master2 ~]# ip netns exec r0 dnsmasq -F 10.0.4.200,10.0.4.220,86400 --dhcp-option=option:router,10.0.4.254 -i rif0
查看DHCP服務是否有監聽:
[root@master2 ~]# ip netns exec r0 ss -unlp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 0 0 *:53 *:* users:(("dnsmasq",pid=3801,fd=6))
UNCONN 0 0 *:67 *:* users:(("dnsmasq",pid=3801,fd=4))
UNCONN 0 0 :::53 :::* users:(("dnsmasq",pid=3801,fd=8))
master1同上,或者不配置DHCP服務也行,自己手動配置IP。
2.3 master2啟動虛擬機測試
[root@master2 ~]# qemu-kvm -name "c2" -m 128 -smp 1 -drive file=/vm/images/c2.img,media=disk,if=virtio -net nic,model=virtio,macaddr=52:54:00:00:00:04 -net tap,ifname=vif1.0,script=/etc/if-up,downscript=/etc/if-down -daemonize
VNC server running on `::1:5900‘
[root@master2 ~]# yum install tigervnc
[root@master2 ~]# vncviewer :5900
已經自動獲取到IP
2.4 master1也啟動一臺虛擬機
[root@master1 ~]# qemu-kvm -name "c1" -m 128 -smp 1 -drive file=/vm/images/cirros-no_cloud-0.3.0-i386-disk.img,media=disk,if=virtio -net nic,model=virtio,macaddr=52:54:00:00:00:01 -net tap,ifname=vif1.0,script=/etc/if-up,downscript=/etc/if-down -daemonize
VNC server running on `::1:5900‘
2.5 GRE隧道通信
2.5.1 配置用於隧道功能的網卡eth1,配置IP並激活
[root@master1 ~]# ip addr add 192.168.20.1/24 dev eth1
[root@master2 ~]# ip addr add 192.168.20.2/24 dev eth1
兩臺宿主機的eth1處於同一vmnet網絡,可以ping通
2.5.2 br-in網橋添加1個port,設置為GRE類型
添加port,名字為gre0
[root@master1 ~]# ovs-vsctl add-port br-in gre0
[root@master1 ~]# ifconfig gre0 up
查看
[root@master1 ~]# ovs-vsctl list port gre0 | grep name
name : "gre0"
[root@master1 ~]# ovs-vsctl show
85eb6797-6dd5-4c5f-aaf8-b7b242b633ae
Bridge br-in
Port "vif1.0"
Interface "vif1.0"
Port "gre0"
Interface "gre0"
設置端口為gre類型,並設置遠端IP值
[root@master1 ~]# ovs-vsctl set interface gre0 type=gre options:remote_ip=192.168.20.2
查看:
[root@master1 ~]# ovs-vsctl list interface gre0 | egrep "type|options"
options : {remote_ip="192.168.20.2"}
type : gre
節點2配置:
[root@master2 ~]# ovs-vsctl add-port br-in gre0 -- set interface gre0 type=gre options:remote_ip=192.168.20.1
查看隧道建立狀態,如果狀態異常可以嘗試重啟oepnvswitch服務:
[root@master1 ~]# ovs-vsctl list interface gre0
link_state : up
mac_in_use : "b2:33:5e:d3:e6:5d"
mtu : []
name : "gre0"
ofport : 9
ofport_request : []
options : {remote_ip="192.168.20.2"}
other_config : {}
statistics : {collisions=0, rx_bytes=0, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=0, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
status : {tunnel_egress_iface="eth1", tunnel_egress_iface_carrier=up}
type : gre
[root@master2 ~]# ovs-vsctl list interface gre0
2.5.3 重啟master1的虛擬機看能否從master2上動態獲取IP
[root@master1 ~]# killall qemu-kvm
[root@master1 ~]# qemu-kvm -name "c1" -m 128 -smp 1 -drive file=/vm/images/cirros-no_cloud-0.3.0-i386-disk.img,media=disk,if=virtio -net nic,model=virtio,macaddr=52:54:00:00:00:01 -net tap,ifname=vif1.0,script=/etc/if-up,downscript=/etc/if-down -daemonize
VNC server running on `::1:5900‘
可以獲取到IP,並且ping通master2上的虛擬機
2.5.4 抓包查看
上圖的ping持續,在master2抓包,下面是一個隧道通信過程。外面是eth1的IP之間通信,裏面是虛擬機的IP間通信
2.6 在master1創建c3(虛擬機名稱),在master2創建c4(虛擬機名稱).實現c1和c2通信,c3和c4通信,其他無法通信
2.6.1 在master1創建c3,在master2創建c4
[root@master1 ~]# qemu-kvm -name "c3" -m 128 -smp 1 -drive file=/vm/images/c3.img,media=disk,if=virtio -net nic,model=virtio,macaddr=52:54:00:00:00:03 -net tap,ifname=vif3.0,script=/etc/if-up,downscript=/etc/if-down -daemonize
VNC server running on `::1:5901‘
[root@master2 ~]# qemu-kvm -name "c4" -m 128 -smp 1 -drive file=/vm/images/c4.img,media=disk,if=virtio -net nic,model=virtio,macaddr=52:54:00:00:00:21 -net tap,ifname=vif4.0,script=/etc/if-up,downscript=/etc/if-down -daemonize
VNC server running on `::1:5901‘
2.6.2 設置c1,c2在vlan10,c3,c4在vlan20
[root@master1 ~]# ovs-vsctl set port vif1.0 tag=10
[root@master1 ~]# ovs-vsctl set port vif3.0 tag=20
[root@master2 ~]# ovs-vsctl set port vif1.0 tag=10
[root@master2 ~]# ovs-vsctl set port vif4.0 tag=20
現在的IP是
c1:10.0.4.203,宿主機:master1
c2:10.0.4.206,宿主機:master2
c3:10.0.4.205,宿主機:master1
c4:10.0.4.214,宿主機:master2
測試:
c1能夠ping通另一臺宿主機的c2,但是無法ping通同個宿主機的c3,和另一臺宿主機的c4
測試:
c3能夠ping通另一臺宿主機的c4,但是無法ping通同個宿主機的c1,和另一臺宿主機的c2
2.6.4 關閉所有vlan,下面做其他測試
[root@master1 ~]# ovs-vsctl remove port vif1.0 tag 10
[root@master1 ~]# ovs-vsctl remove port vif3.0 tag 20
[root@master1 ~]# ovs-vsctl show
85eb6797-6dd5-4c5f-aaf8-b7b242b633ae
Bridge br-in
Port "gre0"
Interface "gre0"
type: gre
options: {remote_ip="192.168.20.2"}
Port "vif3.0"
Interface "vif3.0"
Port "vif1.0"
Interface "vif1.0"
[root@master2 ~]# ovs-vsctl remove port vif1.0 tag 10
[root@master2 ~]# ovs-vsctl remove port vif4.0 tag 20
[root@master2 ~]# ovs-vsctl show
35f025aa-0395-4fa5-b111-553aa9c651e5
Bridge br-in
Port "gre0"
Interface "gre0"
type: gre
options: {remote_ip="192.168.20.1"}
Port "vif1.0"
Interface "vif1.0"
Port "sif0"
Interface "sif0"
Port "vif4.0"
Interface "vif4.0"
繼續關掉GRE,刪除端口即可:
[root@master1 ~]# ovs-vsctl del-port br-in gre0
[root@master2 ~]# ovs-vsctl del-port br-in gre0
刪除後,不同宿主機的虛擬機之間無法通信了
3、VxLAN(本身可以實現GRE的隧道功能)
3.1 vxlan配置
在br-in交換機添加端口:
[root@master1 ~]# ovs-vsctl add-port br-in vx0 -- set interface vx0 type=vxlan options:remote_ip=192.168.20.2
查看,如果狀態異常可以嘗試重啟oepnvswitch服務:
[root@master1 ~]# ovs-vsctl list port vx0
[root@master1 ~]# ovs-vsctl list interface vx0
[root@master2 ~]# ovs-vsctl add-port br-in vx0 -- set interface vx0 type=vxlan options:remote_ip=192.168.20.1
[root@master2 ~]# ovs-vsctl list interface vx0
兩端xvlan配置完畢後,宿主機間的虛擬機又可以通信了:
抓包
4、讓虛擬機能夠訪問互聯網
4.1 再開一個master4作為網絡節點
4.1.1 安裝openvswitch
[root@master1 x86_64]# scp openvswitch-2.5.0-1.x86_64.rpm [email protected]:/root/
安裝
[root@master4 ~]# yum localinstall openvswitch-2.5.0-1.x86_64.rpm
啟動服務
[root@master4 ~]# systemctl start openvswitch.service
4.1.2 master4準備三塊網卡
從上往下,第一塊網卡(僅主機模式)為管理IP,第二塊(VMnet3)為業務IP,主要走GRE或者VxLAN。第三塊網卡IP連接互聯網
配置IP如下:
能夠ping通,另外兩臺宿主機的eth1的IP即可
4.1.3 拆除之前的xvlan
[root@master1 ~]# ovs-vsctl del-port br-in vx0
[root@master2 ~]# ovs-vsctl del-port br-in vx0
4.1.4 創建外部交換機(外部橋,brctl),橋接到外網網卡
設置外網網卡橋接到br-ex橋
[root@master4 network-scripts]# vim ifcfg-eth2
TYPE="Ethernet"
NAME="eth2"
DEVICE="eth2"
ONBOOT="yes"
NM_CONTOLLED="no"
BRIDGE=br-ex
設置br-ex橋的IP:
[root@master4 network-scripts]# vim ifcfg-br-ex
TYPE="Bridge"
BOOTPROTO="static"
NAME="br-ex"
DEVICE="br-ex"
ONBOOT="yes"
IPADDR=10.201.106.134
NETMASK=255.255.255.0
GATEWAY=10.201.106.2
DNS1=10.201.106.2
NM_CONTOLLED="no"
[root@master4 ~]# systemctl restart network
4.2 創建內部橋和netns
4.2.1 配置內部橋(openvswitch)
添加內部橋:
[root@master4 ~]# ovs-vsctl add-br br-in
跟master2建立GRE隧道:
master1配置:
[root@master2 ~]# ovs-vsctl add-port br-in gre0
[root@master2 ~]# ovs-vsctl set interface gre0 type=gre options:remote_ip=192.168.20.254
查看狀態:
[root@master2 ~]# ovs-vsctl list interfaces gre0
如果配置GRE時,使用ovs-vsctl show查看有配置未找到報錯,可以考慮重裝openvswitch或者換一個臨近版本
master4(網絡節點)配置:
[root@master4 ~]# ovs-vsctl add-port br-in gre0
[root@master4 ~]# ovs-vsctl set interface gre0 type=gre options:remote_ip=192.168.20.2
4.2.2 netns配置(兩對網卡,一對是連接到外部橋br-ex,一對是連接到內部橋br-in)
創建網絡名稱空間r0
[root@master4 ~]# ip netns add r0
創建兩對內部網卡:
內部網卡:
[root@master4 ~]# ip link add sin0 type veth peer name rin0
外部網卡:
[root@master4 ~]# ip link add sex0 type veth peer name rex0
激活關聯到橋上的網卡,並關聯到橋:
[root@master4 ~]# ip link set sin0 up
[root@master4 ~]# ip link set sex0 up
將內部網卡添加到內部橋:
[root@master4 ~]# ovs-vsctl add-port br-in sin0
將外部網卡添加到外部橋:
[root@master4 ~]# brctl addif br-ex sex0
ps:關聯到名稱空間的,需要先加入名稱空間後,再在名稱空間裏面激活網卡
將要關聯到名稱空間的網卡,加入到名稱空間裏面:
[root@master4 ~]# ip link set rin0 netns r0
[root@master4 ~]# ip link set rex0 netns r0
配置rin0地址(內部業務地址):
[root@master4 ~]# ip netns exec r0 ifconfig rin0 10.0.4.253/24 up
測試在名稱空間ping虛擬機地址
抓包:
配置rex0地址(外網地址):
[root@master4 ~]# ip netns exec r0 ifconfig rex0 10.201.106.88/24 up
master4(網絡節點IP示意圖如下)
網絡名稱空間已經ping到外網網關:
4.2.3 打開master4的路由轉發
[root@master4 ~]# ip netns exec r0 sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1
4.2.4 虛擬機將網關指向網絡名稱空間(10.0.4.253)
4.3.5 做flowing IP(內外網IP映射)
網絡名稱空間裏面專門新建一個網卡IP,作為映射地址:
[root@master4 ~]# ip netns exec r0 ifconfig rex0:0 10.201.106.66/24 up
將來自內網10.0.4.0的地址轉換為10.201.106.66
[root@master4 ~]# ip netns exec r0 iptables -t nat -A POSTROUTING -s 10.0.4.0/24 -j SNAT --to-source 10.201.106.66
將外部訪問10.201.106.66的數據,轉換為虛擬機IP地址
[root@master4 ~]# ip netns exec r0 iptables -t nat -A PREROUTING -d 10.201.106.66 -j DNAT --to-destination 10.0.4.206
還需要在名稱空間添加去往外網的默認路由:
[root@master4 ~]# ip netns exec r0 route add default gw 10.201.106.2
測試:外部ping 10.201.106.66,能夠轉換為10.0.4.206
抓網絡名稱空間的包:
在業務網口抓包:
如果虛擬機想訪問互聯網,還需要在網絡名稱空間添加默認路由:
[root@master4 ~]# ip netns exec r0 route add default gw 10.201.106.2
KVM虛擬機&openVSwitch雜記(2)