Neutron L2 Gateway + HP 5930 switch OVSDB integration, for VXLAN bridging and routing
轉載http://kimizhang.com/neutron-l2-gateway-hp-5930-switch-ovsdb-integration/
NeutronL2 Gatewayis a new Openstack project working as a Neutron service plugin. It supports only one use case at the moment, which is to bridge Neutron VxLAN tenant network to physical VLAN network via OVSDB Hardware VTEP enabled physical switch.
This blog will go through all steps to configure Openstack and physical switch to make Neutron L2-GW working.
HP 5930 switch is used in this integration.
Red Hat Openstack Platform 7 (Kilo) is used as Openstack environment.
L2 population mechanism driver needs to be enabled to make l2-GW working.
Here's network diagram:
HP 5930 switch configuration
First, let's configure HP 5930 switch.
Get into system-view:
<VGP_LCC_EOR_02>system-view
[VGP_LCC_EOR_02]
Enable l2vpn:
l2vpn enable
Configure a passive TCP connection for OVSDB, we use port 6632 here:
ovsdb server ptcp port 6632
Enable OVSDB server:
ovsdb server enable
Enable VTEP process:
vtep enable
Configure 10.7.31.1 as VTEP source IP:
tunnel global source-address 10.7.31.1
Configure vtep access port, in our example, we use Bridge-Aggregation1:
interface Bridge-Aggregation1
vtep access port
Now HP switch part configuration is done, we can dump the OVSDB to see what it looks like, run ovsdb-client from any Linux machine:
[stack@under ~(stackrc)]$ ovsdb-client dump --pretty tcp:10.7.31.1:6632
Arp_Sources_Local table
_uuid locator src_mac
----- ------- -------
Arp_Sources_Remote table
_uuid locator src_mac
----- ------- -------
Global table
_uuid managers switches
------------------------------------ -------- --------------------------------------
58708319-c85e-44a5-b221-4c2d3cde6004 [] [8f3d25db-f219-443e-9771-1d8e5219b5d9]
Logical_Binding_Stats table
_uuid bytes_from_local bytes_to_local packets_from_local packets_to_local
----- ---------------- -------------- ------------------ ----------------
Logical_Router table
_uuid description name static_routes switch_binding
----- ----------- ---- ------------- --------------
Logical_Switch table
_uuid description name tunnel_key
----- ----------- ---- ----------
Manager table
_uuid inactivity_probe is_connected max_backoff other_config status target
----- ---------------- ------------ ----------- ------------ ------ ------
Mcast_Macs_Local table
MAC _uuid ipaddr locator_set logical_switch
--- ----- ------ ----------- --------------
Mcast_Macs_Remote table
MAC _uuid ipaddr locator_set logical_switch
--- ----- ------ ----------- --------------
Physical_Locator table
_uuid dst_ip encapsulation_type
----- ------ ------------------
Physical_Locator_Set table
_uuid locators
----- --------
Physical_Port table
_uuid description name port_fault_status vlan_bindings vlan_stats
------------------------------------ ----------- --------------------- ----------------- ------------- ----------
28d3da91-7666-492e-9c20-fc960249a7b6 "" "Bridge-Aggregation1" [UP] {} {}
Physical_Switch table
_uuid description management_ips name ports switch_fault_status tunnel_ips tunnels
------------------------------------ ----------- -------------- ---------------- -------------------------------------- ------------------- ------------- -------
8f3d25db-f219-443e-9771-1d8e5219b5d9 "" [] "VGP_LCC_EOR_02" [28d3da91-7666-492e-9c20-fc960249a7b6] [] ["10.7.31.1"] []
Tunnel table
_uuid bfd_config_local bfd_config_remote bfd_params bfd_status local remote
----- ---------------- ----------------- ---------- ---------- ----- ------
Ucast_Macs_Local table
MAC _uuid ipaddr locator logical_switch
--- ----- ------ ------- --------------
Ucast_Macs_Remote table
MAC _uuid ipaddr locator logical_switch
--- ----- ------ ------- --------------
We can see this switch inPhysical_Switchtable and a port inPhysical_Porttable.
L2-GW installation and integration with Neutron
On controller node, fetch latestnetworking-l2gwcodes from stable/kilo branch
git clone -b stable/kilo https://github.com/openstack/networking-l2gw.git
Install it bypip:
pip install ~/networking-l2gw/
Add l2-gw service plugin to neutron server:
[root@overcloud-controller-0 ~]# grep ^service_pl /etc/neutron/neutron.conf
service_plugins =router,networking_l2gw.services.l2gateway.plugin.L2GatewayPlugin
Copy l2-gw example config files to/etc/neutron/:
[root@overcloud-controller-0 ~]#cp /usr/etc/neutron/l2g* /etc/neutron/
Add HP 5930 OVSDB connection tol2gateway_agent.ini,ovsdb1is user defined identifier of physical switch.
[root@overcloud-controller-0 ~]# grep ^ovsdb /etc/neutron/l2gateway_agent.ini
ovsdb_hosts = 'ovsdb1:10.7.31.1:6632'
Make neutron-l2gateway-agent.service systemd unit file, so that we can control the service by systemctl:
[root@overcloud-controller-0 system]# cat /usr/lib/systemd/system/neutron-l2gateway-agent.service
[Unit]
Description=OpenStack Neutron L2 Gateway Agent
After=syslog.target network.target
[Service]
Type=simple
User=neutron
ExecStart=/usr/bin/neutron-l2gateway-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/l2gateway_agent.ini
PrivateTmp=false
KillMode=process
[Install]
WantedBy=multi-user.target
Load the unit file we created:
[root@overcloud-controller-0]# systemctl daemon-reload
Stop neutron services:
pcs resource disable neutron-server-clone
Update neutron database schema:
neutron-l2gw-db-manage --config-file /etc/neutron/neutron.conf upgrade head
Start neutron services:
pcs resource enable neutron-server-clone
Start neutron-l2-gateway service:
systemctl start neutron-l2gateway-agent.service
Now neutron l2-GW is enable, we should be able to see theL2 Gateway agentfromneutron agent-list:
[root@overcloud-controller-0 ~]# neutron agent-list
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
| id | agent_type | host | alive | admin_state_up | binary |
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
| 097320d1-9e7c-468a-bcee-948c560b65c9 | Metadata agent | overcloud-controller-0.localdomain | :-) | True | neutron-metadata-agent |
| 0ef52d27-9b91-40d4-b9dd-d24ff4f86ef7 | Open vSwitch agent | overcloud-controller-0.localdomain | :-) | True | neutron-openvswitch-agent |
| 304a3e9b-3a6f-4ae6-b2af-02891fa63d0f | L2 Gateway agent | overcloud-controller-0.localdomain | :-) | True | neutron-l2gateway-agent |
| 4f81e156-e679-4d76-a816-9b976cf9d62f | Open vSwitch agent | overcloud-compute-0.localdomain | :-) | True | neutron-openvswitch-agent |
| 71f7e395-045f-47a2-9575-9976604eac7c | L3 agent | overcloud-controller-0.localdomain | :-) | True | neutron-l3-agent |
| 81f254a4-dbf2-48e2-afb1-9456201c2631 | DHCP agent | overcloud-controller-0.localdomain | :-) | True | neutron-dhcp-agent |
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
Also we can see the physical switch is registered into neutron DB:
MariaDB [ovs_neutron]> select * from physical_switches;
+--------------------------------------+----------------+-----------+------------------+---------------------+
| uuid | name | tunnel_ip | ovsdb_identifier | switch_fault_status |
+--------------------------------------+----------------+-----------+------------------+---------------------+
| 8f3d25db-f219-443e-9771-1d8e5219b5d9 | VGP_LCC_EOR_02 | 10.7.31.1 | ovsdb1 | NULL |
+--------------------------------------+----------------+-----------+------------------+---------------------+
L2-GW usage
Create a neutron network (VxLAN overlay):
[root@overcloud-controller-0 ~(admin)]# neutron net-create test-l2gw
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 49485e07-77db-4d43-8b20-068597f744a5 |
| mtu | 0 |
| name | test-l2gw |
| provider:network_type | vxlan |
| provider:physical_network | |
| provider:segmentation_id | 100 |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | 7128bb366ffd47ffb445965daab45751 |
+---------------------------+--------------------------------------+
We can see neutron randomly allocates VNI ID100to this tenant network.
Let's create subnet for it:
neutron subnet-create --allocation-pool start=10.7.30.201,end=10.7.30.210 test-l2gw 10.7.30.0/24
Launch a VM in this network:
nova boot --image cirros --flavor m1.tiny --nic net-id=49485e07-77db-4d43-8b20-068597f744a5 test-vm
Check VM launching status and IP address allocated:
[root@overcloud-controller-0 ~(admin)]# nova list
+--------------------------------------+---------+--------+------------+-------------+-----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+-----------------------+
| 15e9aefb-cddd-4be6-a74e-a3098e17f26c | test-vm | ACTIVE | - | Running | test-l2gw=10.7.30.202 |
+--------------------------------------+---------+--------+------------+-------------+-----------------------+
We can see VM is running, with IP10.7.30.202.
We can check VXLAN tunnels created on controller and compute nodes:
[root@overcloud-controller-0 ~(admin)]# ovs-vsctl show | grep vxlan -A1
Port "vxlan-0a14699a"
Interface "vxlan-0a14699a"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="10.20.105.153", out_key=flow, remote_ip="10.20.105.154"}
[root@overcloud-compute-0 ~]# ovs-vsctl show | grep vxlan -A1
Port "vxlan-0a146999"
Interface "vxlan-0a146999"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="10.20.105.154", out_key=flow, remote_ip="10.20.105.153"}
So we can see at the moment, both controller and compute node has only 1 tunnel created to each other. This makes VM to DHCP server traffic working.
Now let's create a l2-gateway usingneutron-l2gwclient:
neutron-l2gw l2-gateway-create --device name="VGP_LCC_EOR_02",interface_names="Bridge-Aggregation1" gw1
Then we can now create a l2-gateway-connection, to bridgetest-l2gwtenant network togw1for VLAN2000:
neutron-l2gw l2-gateway-connection-create --default-segmentation-id 2000 gw1 test-l2gw
We can try to ping hosts in VLAN 2000, from controller dhcp namespace and VM.
[root@overcloud-controller-0 ~(admin)]# ip netns exec qdhcp-49485e07-77db-4d43-8b20-068597f744a5 ping 10.7.30.32
PING 10.7.30.32 (10.7.30.32) 56(84) bytes of data.
64 bytes from 10.7.30.32: icmp_seq=61 ttl=64 time=0.493 ms
64 bytes from 10.7.30.32: icmp_seq=62 ttl=64 time=0.400 ms
64 bytes from 10.7.30.32: icmp_seq=63 ttl=64 time=0.443 ms
...
cirros$ ping 10.7.30.33
PING 10.7.30.33 (10.7.30.33): 56 data bytes
64 bytes from 10.7.30.33: seq=0 ttl=64 time=2.007 ms
64 bytes from 10.7.30.33: seq=1 ttl=64 time=1.034 ms
...
cirros$ ping 10.7.30.32
PING 10.7.30.32 (10.7.30.32): 56 data bytes
64 bytes from 10.7.30.32: seq=0 ttl=64 time=1.824 ms
64 bytes from 10.7.30.32: seq=1 ttl=64 time=1.007 ms
...
Pings work!!!
Let's try to check what's making that working under the hood.
Check VXLAN tunnels on controller and compute nodes, we will see tunnels created between them and HP 5930 switch.
[root@overcloud-controller-0 ~(admin)]# ovs-vsctl show | grep vxlan -A1
Port "vxlan-0a14699a"
Interface "vxlan-0a14699a"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="10.20.105.153", out_key=flow, remote_ip="10.20.105.154"}
Port "vxlan-0a071f01"
Interface "vxlan-0a071f01"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="10.20.105.153", out_key=flow, remote_ip="10.7.31.1"}
[root@overcloud-compute-0 ~]# ovs-vsctl show | grep vxlan -A1
Port "vxlan-0a146999"
Interface "vxlan-0a146999"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="10.20.105.154", out_key=flow, remote_ip="10.20.105.153"}
Port "vxlan-0a0713f4"
Interface "vxlan-0a0713f4"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="10.20.105.154", out_key=flow, remote_ip="10.7.31.1"}
Dump the OVSDB of physical switch:
[stack@under ~(stackrc)]$ ovsdb-client dump --pretty tcp:10.7.31.1:6632
Arp_Sources_Local table
_uuid locator src_mac
----- ------- -------
Arp_Sources_Remote table
_uuid locator src_mac
----- ------- -------
Global table
_uuid managers switches
------------------------------------ -------- --------------------------------------
58708319-c85e-44a5-b221-4c2d3cde6004 [] [8f3d25db-f219-443e-9771-1d8e5219b5d9]
Logical_Binding_Stats table
_uuid bytes_from_local bytes_to_local packets_from_local packets_to_local
----- ---------------- -------------- ------------------ ----------------
Logical_Router table
_uuid description name static_routes switch_binding
----- ----------- ---- ------------- --------------
Logical_Switch table
_uuid description name tunnel_key
------------------------------------ ----------- -------------------------------------- ----------
c08ad894-886f-452a-a528-216f36d48958 "test-l2gw" "49485e07-77db-4d43-8b20-068597f744a5" 100
Manager table
_uuid inactivity_probe is_connected max_backoff other_config status target
----- ---------------- ------------ ----------- ------------ ------ ------
Mcast_Macs_Local table
MAC _uuid ipaddr locator_set logical_switch
----------- ------------------------------------ ------ ------------------------------------ ------------------------------------
unknown-dst aeaab3ec-a0ae-4937-8cda-14ca48039be8 "" e542b5fe-70c4-4872-a741-f334d1bf7021 c08ad894-886f-452a-a528-216f36d48958
Mcast_Macs_Remote table
MAC _uuid ipaddr locator_set logical_switch
--- ----- ------ ----------- --------------
Physical_Locator table
_uuid dst_ip encapsulation_type
------------------------------------ --------------- ------------------
de8e7621-023a-4dfe-bc79-1bbd97962c71 "10.20.105.153" "vxlan_over_ipv4"
6047ff6f-7662-44d0-8e49-b1c6afb51e68 "10.20.105.154" "vxlan_over_ipv4"
7e83e09d-2ee3-4c8d-8784-efae6b780c15 "10.7.31.1" "vxlan_over_ipv4"
Physical_Locator_Set table
_uuid locators
------------------------------------ --------------------------------------
e542b5fe-70c4-4872-a741-f334d1bf7021 [7e83e09d-2ee3-4c8d-8784-efae6b780c15]
Physical_Port table
_uuid description name port_fault_status vlan_bindings vlan_stats
------------------------------------ ----------- --------------------- ----------------- ------------------------------------------- ----------
28d3da91-7666-492e-9c20-fc960249a7b6 "" "Bridge-Aggregation1" [UP] {2000=c08ad894-886f-452a-a528-216f36d48958} {}
Physical_Switch table
_uuid description management_ips name ports switch_fault_status tunnel_ips tunnels
------------------------------------ ----------- -------------- ---------------- ---------------------------------------------------------------------------- ------------------- ------------- ----------------------------------------------------------------------------
8f3d25db-f219-443e-9771-1d8e5219b5d9 "" [] "VGP_LCC_EOR_02" [28d3da91-7666-492e-9c20-fc960249a7b6] [] ["10.7.31.1"] [340ae640-ad8c-4dc7-9c61-1f84c9c462aa, 68629911-a3d0-4793-8a06-583b544dd827]
Tunnel table
_uuid bfd_config_local bfd_config_remote bfd_params bfd_status local remote
------------------------------------ -------------------------------------------------------------- ------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------ ------------------------------------ ------------------------------------
68629911-a3d0-4793-8a06-583b544dd827 {bfd_dst_ip="10.7.31.1", bfd_dst_mac="188:234:250:108:94:141"} {bfd_dst_ip="10.20.105.153", bfd_dst_mac="0:0:0:0:0:0"} {check_tnl_key="0", cpath_down="0", decay_min_rx="100", enable="1", forwarding_if_rx="1", min_rx="100", min_tx="100"} {diagnostic="", forwarding="0", remote_diagnostic="", remote_state=down, state=down} 7e83e09d-2ee3-4c8d-8784-efae6b780c15 de8e7621-023a-4dfe-bc79-1bbd97962c71
340ae640-ad8c-4dc7-9c61-1f84c9c462aa {bfd_dst_ip="10.7.31.1", bfd_dst_mac="188:234:250:108:94:141"} {bfd_dst_ip="10.20.105.154", bfd_dst_mac="0:0:0:0:0:0"} {check_tnl_key="0", cpath_down="0", decay_min_rx="100", enable="1", forwarding_if_rx="1", min_rx="100", min_tx="100"} {diagnostic="", forwarding="0", remote_diagnostic="", remote_state=down, state=down} 7e83e09d-2ee3-4c8d-8784-efae6b780c15 6047ff6f-7662-44d0-8e49-b1c6afb51e68
Ucast_Macs_Local table
MAC _uuid ipaddr locator logical_switch
------------------- ------------------------------------ ------ ------------------------------------ ------------------------------------
"2c:60:0c:c0:97:c9" 1b74b46b-5afb-41ad-ab84-faf1331d0c72 "" 7e83e09d-2ee3-4c8d-8784-efae6b780c15 c08ad894-886f-452a-a528-216f36d48958
"2c:60:0c:c0:a6:c1" 13996ca9-3de0-4bea-b815-036a07aa54d4 "" 7e83e09d-2ee3-4c8d-8784-efae6b780c15 c08ad894-886f-452a-a528-216f36d48958
"2c:60:0c:c0:a9:3b" 9d1c3c28-8ced-4e74-a1a9-bb3a27a41267 "" 7e83e09d-2ee3-4c8d-8784-efae6b780c15 c08ad894-886f-452a-a528-216f36d48958
Ucast_Macs_Remote table
MAC _uuid ipaddr locator logical_switch
------------------- ------------------------------------ ------------- ------------------------------------ ------------------------------------
"fa:16:3e:96:3b:ce" 18dd6787-d5dc-42dc-97eb-285faf5b23ff "10.7.30.202" 6047ff6f-7662-44d0-8e49-b1c6afb51e68 c08ad894-886f-452a-a528-216f36d48958
"fa:16:3e:d1:44:85" 109c752e-1b97-4169-b939-6ada0a8ec703 "10.7.30.201" de8e7621-023a-4dfe-bc79-1bbd97962c71 c08ad894-886f-452a-a528-216f36d48958
FromTunneltable, we can see 2 VXLAN tunnels created between physical switch and controller and compute nodes.
FromLogical_Switchtable, a logical switch is created with VNI ID(tunnel_key)100, then fromPhysical_Porttable, we can see VLAN2000is bound to that logical switch.
FromUcast_Macs_Remotetable, we can see MAC of10.7.30.201and10.7.30.202, which are VM and dhcp server's MACs.
Check tunnel interfaces from HP 5930 console:
[VGP_LCC_EOR_02]dis interface Tunnel
Tunnel0
Current state: UP
Line protocol state: UP
Description: created by ovsdb
Bandwidth: 64 kbps
Maximum transmission unit: 1464
Internet protocol processing: Disabled
Last clearing of counters: Never
Tunnel source unknown, destination 10.20.105.153
Tunnel protocol/transport UDP_VXLAN/IP
Tunnel1
Current state: UP
Line protocol state: UP
Description: created by ovsdb
Bandwidth: 64 kbps
Maximum transmission unit: 1464
Internet protocol processing: Disabled
Last clearing of counters: Never
Tunnel source unknown, destination 10.20.105.154
Tunnel protocol/transport UDP_VXLAN/IP
2 Tunnels are shown there as well.
VXLAN routing problem
On HP 5930 switch, for VLAN2000, we have a L3 IP address working as gateway of this network:
[VGP_LCC_EOR_02]display current-configuration interface Vlan-interface 2000
interface Vlan-interface2000
ip address 10.7.30.1 255.255.255.0
After l2-gateway-connection is created, this default GW IP can not be reached from VM, it's because of limitation of TRIDENT 2 chipset in HP 5930 switch, details can be checked from this blog:LAYER-3 SWITCHING OVER VXLAN REVISITED
Workaround of VXLAN routing problem by adding a loopback cable in the switch
To workaround the HW limitation, we could make a loopback cable on HP 5930 switch, bridge out VXLAN to VLAN first, then through the loopback cable, reaching in the L3 GW as normal VLAN routing.
The diagram is:
Loopback cable is connected between FGE1/0/27 and FGE1/0/28, here we configure FGE1/0/27 as vtep access port, and FGE1/0/28 as normal VLAN trunk port. Dotted red line shows the packet transit path between VM and L3 GW.
On HP 5930 console:
[VGP_LCC_EOR_02]dis current-configuration interface FortyGigE 1/0/27
#
interface FortyGigE1/0/27
port link-mode bridge
port link-type trunk
undo port trunk permit vlan 1
vtep access port
#
return
[VGP_LCC_EOR_02]dis current-configuration interface FortyGigE 1/0/28
#
interface FortyGigE1/0/28
port link-mode bridge
port link-type trunk
undo port trunk permit vlan 1
port trunk permit vlan 2000
#
return
We need to remove previously created l2-gateway-connection and l2-gateway:
neutron-l2gw l2-gateway-connection-delete 4fb1c9eb-de29-442c-a946-190f5f99cf17
neutron-l2gw l2-gateway-delete gw1
Re-create l2-gatewaygw1with interfaceFortyGigE1/0/27:
neutron-l2gw l2-gateway-create --device name="VGP_LCC_EOR_02",interface_names="FortyGigE1/0/27" gw1
Create l2-gateway-connection, to bridgetest-l2gwtenant network togw1for VLAN2000
neutron-l2gw l2-gateway-connection-create --default-segmentation-id 2000 gw1 test-l2gw
Now try to ping L3 GW 10.7.30.1 from VM.
$ ping 10.7.30.1
PING 10.7.30.1 (10.7.30.1): 56 data bytes
64 bytes from 10.7.30.1: seq=1 ttl=64 time=1.463 ms
64 bytes from 10.7.30.1: seq=2 ttl=64 time=2.606 ms
64 bytes from 10.7.30.1: seq=3 ttl=64 time=1.036 ms
...
It works now!!!