docker overlay network測試
阿新 • • 發佈：2018-12-30
docker自1.9版本後，引入了overlay網路（本文不具體分析其背後使用的技術）。重點解決之前docker網路在跨主機通訊方面的不足。本文記錄，參考官方指導文件，搭建測試overlay網路的過程。
文中使用的os為centos7，核心版本為3.10。而docker 1.9版本overlay網路要求核心版本在3.19以上。自docker 1.10版本後，docker overlay 支援3.10版本核心。因此，本文中使用的docker版本為1.10.3。
測試環境工包含三臺virtualbox虛擬機器。1臺作為key-value store儲存的機器，本文測試中選用的是etcd。另外，兩臺用來測試跨host的通訊,分別為net1和net2。
docker info：
[[email protected] vagrant]# docker info
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 1
Server Version: 1.10.3
Storage Driver: devicemapper
Pool Name: docker-253:0-469034-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/vg-docker/data
Metadata file: /dev/vg-docker/metadata
Data Space 注：使用預設的loop的時候，overlay網路測試存在問題。無法建立成功。
參考官方文件，需要配置docker daemon的如下引數：
--cluster-store=PROVIDER://URL Describes the location of the KV service.
--cluster-advertise=HOST_IP|HOST_IFACE:PORT
The IP address or interface of the HOST used for clustering.
--cluster-store-opt=KEY-VALUE OPTIONS
Options such as TLS certificate or tuning discovery Timersdocker daemon引數
/usr/bin/docker daemon -H fd:// --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata --cluster-store=etcd://172.28.0.2:4001 --cluster-advertise=eth1:0埠，7946控制面，4789資料面。
firewall-cmd --permanent --add-port=7946/tcp
firewall-cmd --permanent --add-port=7946/udp
firewall-cmd --permanent --add-port=4789/udp配置完上述引數後，即可建立網路。
docker network create -d overlay mutihost
[[email protected] vagrant]# docker network ls
NETWORK ID NAME DRIVER
15bb57daf277 multihost overlay
3cd7ab7018e9 docker_gwbridge bridge
a874aa0d9e0b bridge bridge
9fe04ff37f6f none null
010a53c2bf04 host host
[[email protected] vagrant]# docker network inspect multihost
[
{
"Name": "multihost",
"Id": "15bb57daf27731da102c8a5c5bf903e574daa33f5286e938009734a8cd5ce93c",
"Scope": "global",
"Driver": "overlay",
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.0.0/24",
"Gateway": "10.0.0.1/24"
}
]
},
"Containers": {},
"Options": {}
}
]
[[email protected] vagrant]# docker network inspect multihost
[
{
"Name": "multihost",
"Id": "15bb57daf27731da102c8a5c5bf903e574daa33f5286e938009734a8cd5ce93c",
"Scope": "global",
"Driver": "overlay",
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.0.0/24",
"Gateway": "10.0.0.1/24"
}
]
},
"Containers": {
"37162168dca4ad715d12f6bc78d1bf0678ff9128fe5d55178e39ed08e847f80a": {
"Name": "tender_kirch",
"EndpointID": "765451d1201d570c626470d16c92515de55f4ea1df2a58a03bef8e1767873897",
"MacAddress": "02:42:0a:00:00:05",
"IPv4Address": "10.0.0.5/24",
"IPv6Address": ""
}
},
"Options": {}
}
]
[[email protected] vagrant]# docker run -it --rm=true --net=multihost centos /bin/bash
[[email protected]37162168dca4 /]# ping 10.0.0.3
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
64 bytes from 10.0.0.3: icmp_seq=1 ttl=64 time=0.505 ms
64 bytes from 10.0.0.3: icmp_seq=2 ttl=64 time=0.619 ms
64 bytes from 10.0.0.3: icmp_seq=3 ttl=64 time=0.632 ms
64 bytes from 10.0.0.3: icmp_seq=4 ttl=64 time=0.660 ms
64 bytes from 10.0.0.3: icmp_seq=5 ttl=64 time=0.663 ms觀察容器內的鏈路
[root@37162168dca4 /]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
11: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT
link/ether 02:42:0a:00:00:05 brd ff:ff:ff:ff:ff:ff
13: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
[root@37162168dca4 /]# ethtool -S eth1
NIC statistics:
peer_ifindex: 14
[root@37162168dca4 /]# ethtool -S eth0
NIC statistics:
peer_ifindex: 12host上的網橋
[root@net2 vagrant]# brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.024297afd372 no
docker_gwbridge 8000.0242117ceeda no veth2cef6db
ov-000100-15bb5 8000.96f96b0c7379 no vetha6b50db
vx-000100-15bb5
[root@net2 vagrant]# ip -d link
12: vetha6b50db: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master ov-000100-15bb5 state UP mode DEFAULT
link/ether be:c0:b4:39:e5:fc brd ff:ff:ff:ff:ff:ff promiscuity 1
veth addrgenmode eui64
14: veth2cef6db: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP mode DEFAULT
link/ether ca:d6:b0:d9:e9:5c brd ff:ff:ff:ff:ff:ff promiscuity 1
veth addrgenmode eui64
vx-000100-15bb5
[root@net2 vagrant]# ip -d link show vx-000100-15bb5
10: vx-000100-15bb5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ov-000100-15bb5 state UNKNOWN mode DEFAULT
link/ether 96:f9:6b:0c:73:79 brd ff:ff:ff:ff:ff:ff promiscuity 1
vxlan id 256 srcport 0 0 dstport 4789 proxy l2miss l3miss ageing 300 addrgenmode eui64 從容器內查詢的結果看，連結overaly網路的時候，會建立兩個網橋，其中ov-000100-15bb5 網橋用來中有兩個裝置,一個veth peer用來連結容器和網橋，還有一個vxlan裝置，從查詢資料看vxlan id為256。
另外，還有一個docker_gwbridge網橋，容器也通過veth pair裝置連線到了該網橋。這個網路的作用主要是方便容器對外提供服務。