Docker 與 K8S學習筆記（八）—— 自定義容器網路
阿新 • • 發佈：2021-10-23
我們在上一篇中介紹了Docker中三種網路，none、host和bridge，除了這三種網路，Docker還允許我們建立自定義網路，當我們要建立自定義網路時，Docker提供了三種網路驅動供我們選擇：bridge、macvlan和overlay，其中macvlan和ovelay都是用於建立跨主機網路，我們後面在研究，本篇我們主要使用bridge驅動。
一、建立自定義網路
我們通過dockernetwork create命令建立test_net1網路：
$ sudo docker network create --driver bridge test_net1 c7c20444a940135c92958f4434ca2b7428ba17f70a2a8a954a6cf160a011b513 $sudo docker network ls NETWORK ID NAME DRIVER SCOPE 6f0087fd32cd bridge bridge local 464b3d11003c host host local faa8eb8310b4 none null local c7c20444a940 test_net1 bridge local $ brctl show bridge name bridge id STP enabled interfaces br-c7c20444a940 8000.024200e01cd2 no docker0 8000.0242cef1fc32 no
我們可以看到，此時系統中新增了一個網橋br-c7c20444a940，其中c7c20444a940就是我們剛剛建立的橋接網路的ID。我們可以看下test_net1的詳細配置資訊：
$ sudo docker network inspect test_net1 [ { "Name": "test_net1", "Id": "c7c20444a940135c92958f4434ca2b7428ba17f70a2a8a954a6cf160a011b513", "Created": "2021-10-23T07:07:47.870947684Z", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": {}, "Config": [ { "Subnet": "172.19.0.0/16", "Gateway": "172.19.0.1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": {}, "Options": {}, "Labels": {} } ]
如果我們在建立網路時，不指定--subnet和--gateway引數時，Docker會自動為其分配網段和閘道器，我們接著再建立一個網路並指定其網段和閘道器：
$ sudo docker network create --driver bridge --subnet 173.20.0.0/16 --gateway 173.20.0.1 test_net2 47542e86dc4474779a3b105f1d22f005fc39a529b1f19aa7af2c68297c1c0a41 $ sudo docker network inspect test_net2 [ { "Name": "test_net2", "Id": "47542e86dc4474779a3b105f1d22f005fc39a529b1f19aa7af2c68297c1c0a41", "Created": "2021-10-23T07:21:47.684305897Z", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": {}, "Config": [ { "Subnet": "173.20.0.0/16", "Gateway": "173.20.0.1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": {}, "Options": {}, "Labels": {} } ] $ brctl show bridge name bridge id STP enabled interfaces br-47542e86dc44 8000.024273a34080 no br-c7c20444a940 8000.024200e01cd2 no docker0 8000.0242cef1fc32 no $ ifconfig br-47542e86dc44 br-47542e86dc44: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 173.20.0.1 netmask 255.255.0.0 broadcast 173.20.255.255 ether 02:42:73:a3:40:80 txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
二、自定義網路的使用
容易要使用自定義的網路，同樣需在啟動時通過--network引數指定即可：
$ sudo docker run -it --network=test_net2 busybox / # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 50: eth0@if51: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether 02:42:ad:14:00:02 brd ff:ff:ff:ff:ff:ff inet 173.20.0.2/16 brd 173.20.255.255 scope global eth0 valid_lft forever preferred_lft forever
我們可以看到Docker會根據容易所使用的網路的網段自動為容器分配IP，當然，容器IP也是可以指定的，即通過--ip引數，但是要注意兩點：
-
只有使用--subnet引數指定網段建立的網路才能指定IP
-
所指定的IP一定要在所使用的網路的網段中
下圖所示的錯誤就是容器使用的網路並沒有通過--subnet指定網段：
$ sudo docker run -it --network=test_net1 --ip 172.19.0.5 busybox docker: Error response from daemon: user specified IP address is supported only when connecting to networks with user configured subnets.
下圖所示的錯誤是容器所指定IP不在其網路的指定網段內：
$ sudo docker run -it --network=test_net2 --ip 172.20.0.2 busybox ERRO[0000] error waiting for container: context canceled docker: Error response from daemon: Invalid address 172.20.0.2: It does not belong to any of this network's subnets.
以下是正確的指定容器IP的例子：
$ sudo docker run -it --network=test_net2 --ip 173.20.0.4 busybox / # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 54: eth0@if55: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether 02:42:ad:14:00:04 brd ff:ff:ff:ff:ff:ff inet 173.20.0.4/16 brd 173.20.255.255 scope global eth0 valid_lft forever preferred_lft forever
三、Docker網路隔離及原理
到目前位置我們建立使用test_net2網路建立了兩個容器，那麼這兩個容器網路應該是互通的，我們來驗證一下：
$ sudo docker run -it --network=test_net2 --ip 173.20.0.4 busybox / # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 54: eth0@if55: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether 02:42:ad:14:00:04 brd ff:ff:ff:ff:ff:ff inet 173.20.0.4/16 brd 173.20.255.255 scope global eth0 valid_lft forever preferred_lft forever / # ping 173.20.0.2 PING 173.20.0.2 (173.20.0.2): 56 data bytes 64 bytes from 173.20.0.2: seq=0 ttl=64 time=1.319 ms 64 bytes from 173.20.0.2: seq=1 ttl=64 time=0.106 ms 64 bytes from 173.20.0.2: seq=2 ttl=64 time=0.107 ms ^C --- 173.20.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.106/0.510/1.319 ms
所以，我們得出結論，使用同一網路建立的容器是能夠互通的。相反，如果是不同網路建立的容器應該是無法通訊，我們使用test_net1建立一個容器驗證一下：
$ sudo docker run -it --network=test_net1 busybox / # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 58: eth0@if59: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether 02:42:ac:13:00:02 brd ff:ff:ff:ff:ff:ff inet 172.19.0.2/16 brd 172.19.255.255 scope global eth0 valid_lft forever preferred_lft forever / # ping 173.20.0.2 PING 173.20.0.2 (173.20.0.2): 56 data bytes ^C --- 173.20.0.2 ping statistics --- 146 packets transmitted, 0 packets received, 100% packet loss
從結果來看確實無法通訊，但是如果我們開啟系統的ip forwarding，把當前主機當作路由，那麼不同網路應該是能通訊的，我們首先來看下系統的配置：
$ ip r default via 172.16.194.2 dev ens33 proto dhcp src 172.16.194.135 metric 100 172.16.194.0/24 dev ens33 proto kernel scope link src 172.16.194.135 172.16.194.2 dev ens33 proto dhcp scope link src 172.16.194.135 metric 100 172.19.0.0/16 dev br-c7c20444a940 proto kernel scope link src 172.19.0.1 173.20.0.0/16 dev br-47542e86dc44 proto kernel scope link src 173.20.0.1 $ sysctl net.ipv4.ip_forward net.ipv4.ip_forward = 1
看來我們的主機已經開啟了ip forwarding並且172.19.0.0/16和173.20.0.0/16的路由也定義好了，那為什麼網路還是不通呢？我們看看iptables：
$ sudo iptables-save # Generated by iptables-save v1.6.1 on Sat Oct 23 08:19:03 2021 *filter ... # 如果資料包是從br-47542e86dc44流入但不從br-47542e86dc44流出則跳轉到DOCKER-ISOLATION-STAGE-2處理 -A DOCKER-ISOLATION-STAGE-1 -i br-47542e86dc44 ! -o br-47542e86dc44 -j DOCKER-ISOLATION-STAGE-2 # 如果資料包是從br-c7c20444a940流入但不從br-c7c20444a940流出則跳轉到DOCKER-ISOLATION-STAGE-2處理 -A DOCKER-ISOLATION-STAGE-1 -i br-c7c20444a940 ! -o br-c7c20444a940 -j DOCKER-ISOLATION-STAGE-2 # 如果資料包是從docker0流入但不從docker0流出則跳轉到DOCKER-ISOLATION-STAGE-2處理 -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2 # 如果非以上三種情況則返回上一層處理 -A DOCKER-ISOLATION-STAGE-1 -j RETURN # 如果資料包流出到br-47542e86dc44則drop -A DOCKER-ISOLATION-STAGE-2 -o br-47542e86dc44 -j DROP # 如果資料包流出到br-c7c20444a940則drop -A DOCKER-ISOLATION-STAGE-2 -o br-c7c20444a940 -j DROP # 如果資料包流出到docker0則drop -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP # 如果非以上三種情況則返回上一層處理 -A DOCKER-ISOLATION-STAGE-2 -j RETURN -A DOCKER-USER -j RETURN ...
看來原因就在這裡了，我們發現Docker建立了一系列網路隔離策略，當資料包流入的網路與流出的網路相同則不作處理，不同則棄資料包，我們通過一個流程圖來說明DOCKER-ISOLATION-STAGE-1和DOCKER-ISOLATION-STAGE-2規則的邏輯：
如果我一定要兩個使用不同網路的容器通訊，該怎麼處理呢？也很簡單，我們只需要給容器新增對應的網路的的網絡卡即可，我們通過docker network connect命令可以實現：
# 0da5e260a4b6容器使用的是test_net1，另外兩個使用的是test_net2， # 如果大家在這一步不確定的話可以使用docker inspect 容器id進行確定 $ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0da5e260a4b6 busybox "sh" 2 hours ago Up 2 hours objective_blackburn c6871db3efbb busybox "sh" 2 hours ago Up 2 hours goofy_swartz c683eacc4eae busybox "sh" 2 hours ago Up 2 hours modest_swartz yangye@ayato:~$ sudo docker network connect test_net2 0da5e260a4b6
我們在容器0da5e260a4b6中先ping下173.20.0.2發現網路是通的，我們再看看其網路配置，發現容器中添加了my_net2網路的網絡卡eth1：
/ # ping 173.20.0.2 PING 173.20.0.2 (173.20.0.2): 56 data bytes 64 bytes from 173.20.0.2: seq=0 ttl=64 time=0.291 ms 64 bytes from 173.20.0.2: seq=1 ttl=64 time=0.100 ms 64 bytes from 173.20.0.2: seq=2 ttl=64 time=0.097 ms 64 bytes from 173.20.0.2: seq=3 ttl=64 time=0.100 ms 64 bytes from 173.20.0.2: seq=4 ttl=64 time=0.147 ms ^C --- 173.20.0.2 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max = 0.097/0.147/0.291 ms / # ifconfig eth0 Link encap:Ethernet HWaddr 02:42:AC:13:00:02 inet addr:172.19.0.2 Bcast:172.19.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:25 errors:0 dropped:0 overruns:0 frame:0 TX packets:150 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1830 (1.7 KiB) TX bytes:14476 (14.1 KiB) eth1 Link encap:Ethernet HWaddr 02:42:AD:14:00:03 inet addr:173.20.0.3 Bcast:173.20.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:18 errors:0 dropped:0 overruns:0 frame:0 TX packets:7 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1440 (1.4 KiB) TX bytes:574 (574.0 B) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
好的，以上就是本篇的內容，下一篇我們再看看容器間通訊方式。