Neutron总结-openvswitch+vxlan网络

最新推荐文章于 2023-08-08 09:59:22 发布

dylloveyou

最新推荐文章于 2023-08-08 09:59:22 发布

阅读量1w

点赞数 6

本文链接：https://blog.csdn.net/dylloveyou/article/details/72639187

版权

云计算同时被 3 个专栏收录

56 篇文章 8 订阅

订阅专栏

OpenStack

54 篇文章 17 订阅

订阅专栏

OpenStack

33 篇文章 16 订阅

订阅专栏

本篇文章介绍如何规划及创建openvswitch+vxlan网络，实现实例间及实例与外部的通讯。读者应对OpenStack及其网络服务Neutron有初步的了解。

规划网络

部署节点为一个controller节点(包含网络节点)，两个compute节点。controller节点有3个网卡，分别为eth0(管理和API网络，CIDR为192.168.128.0/24)、eth1(租户网络，CIDR为10.10.10.0/24)、eth2(外部网络，CIDR为11.11.11.0/24，不设置IP)；compute节点有2个网卡，分别为eth0(管理和API网络，CIDR为192.168.128.0/24)、eth1(租户网络，CIDR为10.10.10.0/24)。
这里写图片描述

这里用VMware Workstation下面的三个虚拟机分别作为controller、compute1、compute2节点，其网卡设置为：

每个主机的eth0网卡作为 API&Management 网络，使用的是“NAT模式”，可以连接到外网下载OpenStack软件
每个主机的eth1网卡作为 Tenant(VM) 网络，是承载VxLAN的底层网络，使用的是“仅主机模式 ”
controller节点也作为网络节点，需要模拟OpenStack的外网，增加网卡eth2，这里也选择“仅主机模式”

这里写图片描述

具体IP设置为：

controller节点

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.128.31
netmask 255.255.255.0
gateway 192.168.128.2

# The provider network interface
auto eth1
iface eth1 inet static
address 10.10.10.10
netmask 255.255.255.0

# The external network interface
auto eth2
iface eth2 inet manual

compute1节点

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.128.32
netmask 255.255.255.0
gateway 192.168.128.2

# The provider network interface
auto eth1
iface eth1 inet static
address 10.10.10.11
netmask 255.255.255.0

compute2节点

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.128.34
netmask 255.255.255.0
gateway 192.168.128.2

# The provider network interface
auto eth1
iface eth1 inet static
address 10.10.10.12
netmask 255.255.255.0

Neutron网络方案为：

Neutron ML2的Type Driver为vxlan
Neutron ML2的Mechanism Driver为Open vSwitch
Neutron L2 Agent为Open vSwitch

创建虚拟网络

修改配置文件

controller节点

/etc/neutron/plugins/ml2/ml2_conf.ini

[ml2]
type_drivers = flat,vxlan
tenant_network_types = vxlan
mechanism_drivers = openvswitch,l2population
extension_drivers = port_security

[ml2_type_vxlan]
vni_ranges = 1001:2000

[securitygroup]
enable_ipset = true

/etc/neutron/l3_agent.ini

[DEFAULT]
external_network_bridge = br-ex
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

/etc/neutron/dhcp_agent.ini

[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq
enable_isolated_metadata = True

/etc/neutron/plugins/ml2/openvswitch_agent.ini

[agent]
tunnel_types = vxlan
l2_population = True

[ovs]
bridge_mappings = 
tunnel_bridge = br-tun
local_ip = 10.10.10.10

[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_security_group = true

compute1节点

/etc/neutron/plugins/ml2/openvswitch_agent.ini

[agent]
tunnel_types = vxlan
l2_population = True

[ovs]
bridge_mappings = 
tunnel_bridge = br-tun
local_ip = 10.10.10.11

[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_security_group = true

compute2节点

/etc/neutron/plugins/ml2/openvswitch_agent.ini

[agent]
tunnel_types = vxlan
l2_population = True

[ovs]
bridge_mappings = 
tunnel_bridge = br-tun
local_ip = 10.10.10.12

[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_security_group = true

提前准备ovs网桥

root@controller:~# ovs-vsctl add-br br-ex
root@controller:~# ovs-vsctl add-port br-ex eth2

配置文件修改后，需要重启网络相关服务。

登陆到dashboard，可以查看网络服务信息。
controller节点启动了 neturon-metadata-agent、neutron-l3-agent、neutron-openvswitch-agent、neutron-dhcp-agent
compute节点启动了 neutron-openvswitch-agent
这里写图片描述

创建vxlan100网络，网络地址为172.16.100.0/24
这里写图片描述

创建vxlan101网络，网络地址为172.16.101.0/24
这里写图片描述

创建外部网络（外部网络为flat网络），物理网络为external，网络地址为11.11.11.0/24
这里写图片描述

创建路由器，把vxlan网络和外部网络连通
这里写图片描述

我们的网络环境就准备好了。

测试网络

创建3个实例，2个选择vxlan100网络，1个选择vxlan101网络。
其中：test1、test3被调度到compute1节点，test2被调度到compute2节点。
这里写图片描述

查看网络拓扑。两个vxlan网络均通过路由器连接到了外网，并且两个vxlan网络也通过路由器连接。所以理论上test1、test2、test3能互通，并且也能连通外部网络。
这里写图片描述

为了保证外部能ping通以及能ssh登陆到实例，需要在安全组那加两条规则。
这里写图片描述

下面测试网络的连通性：

同一vxlan：172.16.100.11 ping 172.16.100.12
这里写图片描述

不同vxlan：172.16.100.11 ping 172.16.101.11
这里写图片描述

外网：172.16.100.11 ping 11.11.11.111 (确保外网相同网段有一台机器)
这里写图片描述

查看各个节点上面的虚拟网络设备

controller节点

root@controller:~# ovs-vsctl show
cf60e0c1-d418-41ed-82e4-c16609fd4817
    Bridge br-tun
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vxlan-0a0a0a0b"
            Interface "vxlan-0a0a0a0b"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.10.10.10", out_key=flow, remote_ip="10.10.10.11"}
        Port br-tun
            Interface br-tun
                type: internal
        Port "vxlan-0a0a0a0c"
            Interface "vxlan-0a0a0a0c"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.10.10.10", out_key=flow, remote_ip="10.10.10.12"}
    Bridge br-ex
        Port "qg-3f57bf51-cd"
            Interface "qg-3f57bf51-cd"
                type: internal
        Port "eth2"
            Interface "eth2"
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-int
        fail_mode: secure
        Port "qr-154f0704-67"
            tag: 3
            Interface "qr-154f0704-67"
                type: internal
        Port "tap70544d00-52"
            tag: 3
            Interface "tap70544d00-52"
                type: internal
        Port "qr-6a034a9d-a0"
            tag: 2
            Interface "qr-6a034a9d-a0"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf3ec4aee-3d"
            tag: 2
            Interface "tapf3ec4aee-3d"
                type: internal
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.5.0"

compute1节点

root@controller:~# ovs-vsctl show
cf60e0c1-d418-41ed-82e4-c16609fd4817
    Bridge br-tun
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vxlan-0a0a0a0b"
            Interface "vxlan-0a0a0a0b"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.10.10.10", out_key=flow, remote_ip="10.10.10.11"}
        Port br-tun
            Interface br-tun
                type: internal
        Port "vxlan-0a0a0a0c"
            Interface "vxlan-0a0a0a0c"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.10.10.10", out_key=flow, remote_ip="10.10.10.12"}
    Bridge br-ex
        Port "qg-3f57bf51-cd"
            Interface "qg-3f57bf51-cd"
                type: internal
        Port "eth2"
            Interface "eth2"
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-int
        fail_mode: secure
        Port "qr-154f0704-67"
            tag: 3
            Interface "qr-154f0704-67"
                type: internal
        Port "tap70544d00-52"
            tag: 3
            Interface "tap70544d00-52"
                type: internal
        Port "qr-6a034a9d-a0"
            tag: 2
            Interface "qr-6a034a9d-a0"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapf3ec4aee-3d"
            tag: 2
            Interface "tapf3ec4aee-3d"
                type: internal
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.5.0"

root@compute1:~# brctl show
bridge name bridge id       STP enabled interfaces
qbr5c00dd50-28      8000.5ebc610f1574   no      qvb5c00dd50-28
                            tap5c00dd50-28
qbrb3770721-b5      8000.4e6230493787   no      qvbb3770721-b5
                            tapb3770721-b5
virbr0      8000.52540066ffc3   yes     virbr0-nic

compute2节点

root@compute2:~# ovs-vsctl show
65d9004a-fb60-4554-aff1-2f91e40fa076
    Bridge br-tun
        fail_mode: secure
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vxlan-0a0a0a0a"
            Interface "vxlan-0a0a0a0a"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.10.10.12", out_key=flow, remote_ip="10.10.10.10"}
        Port "vxlan-0a0a0a0b"
            Interface "vxlan-0a0a0a0b"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.10.10.12", out_key=flow, remote_ip="10.10.10.11"}
    Bridge br-int
        fail_mode: secure
        Port "qvofb1e821c-a9"
            tag: 2
            Interface "qvofb1e821c-a9"
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    ovs_version: "2.5.0"

root@compute2:~# brctl show
bridge name bridge id       STP enabled interfaces
qbrfb1e821c-a9      8000.e62d95f3538a   no      qvbfb1e821c-a9
                            tapfb1e821c-a9
virbr0      8000.52540066ffc3   yes     virbr0-nic

controller节点1个Router、2个DHCP分别在各自的namespace下

root@controller:~# ip netns
qrouter-b65df04e-908e-457f-b52b-323db01cff6f
qdhcp-eea49c4f-c5ca-4d33-b73e-43e4f01aa3ec
qdhcp-b9c98245-eb3f-41b8-9f0b-b4c4b063cccd

可以通过exec查看router细节

root@controller:~# ip netns exec qrouter-b65df04e-908e-457f-b52b-323db01cff6f ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
14: qr-154f0704-67: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:55:37:6d brd ff:ff:ff:ff:ff:ff
    inet 172.16.101.1/24 brd 172.16.101.255 scope global qr-154f0704-67
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe55:376d/64 scope link 
       valid_lft forever preferred_lft forever
15: qr-6a034a9d-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:58:be:77 brd ff:ff:ff:ff:ff:ff
    inet 172.16.100.1/24 brd 172.16.100.255 scope global qr-6a034a9d-a0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe58:be77/64 scope link 
       valid_lft forever preferred_lft forever
18: qg-3f57bf51-cd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:d8:9a:d4 brd ff:ff:ff:ff:ff:ff
    inet 11.11.11.21/24 brd 11.11.11.255 scope global qg-3f57bf51-cd
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fed8:9ad4/64 scope link 
       valid_lft forever preferred_lft forever

查看router的路由

root@controller:~# ip netns exec qrouter-b65df04e-908e-457f-b52b-323db01cff6f route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         11.11.11.1      0.0.0.0         UG    0      0        0 qg-3f57bf51-cd
11.11.11.0      *               255.255.255.0   U     0      0        0 qg-3f57bf51-cd
172.16.100.0    *               255.255.255.0   U     0      0        0 qr-6a034a9d-a0
172.16.101.0    *               255.255.255.0   U     0      0        0 qr-154f0704-67

router的iptables

root@controller:~# ip netns exec qrouter-b65df04e-908e-457f-b52b-323db01cff6f iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-POSTROUTING ! -i qg-3f57bf51-cd ! -o qg-3f57bf51-cd -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -o qg-3f57bf51-cd -j SNAT --to-source 11.11.11.21
-A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 11.11.11.21
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat

可以通过exec查看dhcp细节
vxlan100的dhcp

root@controller:~# ip netns exec qdhcp-b9c98245-eb3f-41b8-9f0b-b4c4b063cccd ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
11: tapf3ec4aee-3d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:f1:3f:c8 brd ff:ff:ff:ff:ff:ff
    inet 172.16.100.10/24 brd 172.16.100.255 scope global tapf3ec4aee-3d
       valid_lft forever preferred_lft forever
    inet 169.254.169.254/16 brd 169.254.255.255 scope global tapf3ec4aee-3d
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fef1:3fc8/64 scope link 
       valid_lft forever preferred_lft forever

vxlan101的dhcp

root@controller:~# ip netns exec qdhcp-eea49c4f-c5ca-4d33-b73e-43e4f01aa3ec ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
12: tap70544d00-52: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:4f:e5:47 brd ff:ff:ff:ff:ff:ff
    inet 172.16.101.10/24 brd 172.16.101.255 scope global tap70544d00-52
       valid_lft forever preferred_lft forever
    inet 169.254.169.254/16 brd 169.254.255.255 scope global tap70544d00-52
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe4f:e547/64 scope link 
       valid_lft forever preferred_lft forever

再看下更详细的网络结构
controller节点
这里写图片描述

compute节点
这里写图片描述

如果需要从外部网络访问虚拟机，则要添加Floating IP
test1添加floating IP为11.11.11.22
这里写图片描述

测试网络连通：
外网：11.11.11.111 ping 11.11.11.22

root@ubuntu:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:0d:7d:0f brd ff:ff:ff:ff:ff:ff
    inet 192.168.128.10/24 brd 192.168.128.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe0d:7d0f/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:0d:7d:19 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.10/24 brd 192.168.1.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe0d:7d19/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:0d:7d:23 brd ff:ff:ff:ff:ff:ff
    inet 11.11.11.111/24 brd 11.11.11.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe0d:7d23/64 scope link 
       valid_lft forever preferred_lft forever
root@ubuntu:~# ping 11.11.11.22
PING 11.11.11.22 (11.11.11.22) 56(84) bytes of data.
64 bytes from 11.11.11.22: icmp_seq=1 ttl=63 time=17.0 ms
64 bytes from 11.11.11.22: icmp_seq=2 ttl=63 time=2.34 ms
64 bytes from 11.11.11.22: icmp_seq=3 ttl=63 time=0.856 ms
64 bytes from 11.11.11.22: icmp_seq=4 ttl=63 time=3.00 ms

外网：11.11.11.111 ssh登录到 11.11.11.22

root@ubuntu:~# ssh cirros@11.11.11.22
The authenticity of host '11.11.11.22 (11.11.11.22)' can't be established.
RSA key fingerprint is 41:b8:b4:8b:87:b6:99:f0:61:93:65:42:12:58:67:24.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '11.11.11.22' (RSA) to the list of known hosts.
cirros@11.11.11.22's password: 
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:7f:29:10 brd ff:ff:ff:ff:ff:ff
    inet 172.16.100.11/24 brd 172.16.100.255 scope global eth0
    inet6 fe80::f816:3eff:fe7f:2910/64 scope link 
       valid_lft forever preferred_lft forever
$

查看controller节点router的变化

root@controller:~# ip netns exec qrouter-b65df04e-908e-457f-b52b-323db01cff6f ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
14: qr-154f0704-67: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:55:37:6d brd ff:ff:ff:ff:ff:ff
    inet 172.16.101.1/24 brd 172.16.101.255 scope global qr-154f0704-67
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe55:376d/64 scope link 
       valid_lft forever preferred_lft forever
15: qr-6a034a9d-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:58:be:77 brd ff:ff:ff:ff:ff:ff
    inet 172.16.100.1/24 brd 172.16.100.255 scope global qr-6a034a9d-a0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe58:be77/64 scope link 
       valid_lft forever preferred_lft forever
18: qg-3f57bf51-cd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:d8:9a:d4 brd ff:ff:ff:ff:ff:ff
    inet 11.11.11.21/24 brd 11.11.11.255 scope global qg-3f57bf51-cd
       valid_lft forever preferred_lft forever
    inet 11.11.11.22/32 brd 11.11.11.22 scope global qg-3f57bf51-cd
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fed8:9ad4/64 scope link 
       valid_lft forever preferred_lft forever

iptables

root@controller:~# ip netns exec qrouter-b65df04e-908e-457f-b52b-323db01cff6f iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 11.11.11.22/32 -j DNAT --to-destination 172.16.100.11
-A neutron-l3-agent-POSTROUTING ! -i qg-3f57bf51-cd ! -o qg-3f57bf51-cd -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 11.11.11.22/32 -j DNAT --to-destination 172.16.100.11
-A neutron-l3-agent-float-snat -s 172.16.100.11/32 -j SNAT --to-source 11.11.11.22
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -o qg-3f57bf51-cd -j SNAT --to-source 11.11.11.21
-A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 11.11.11.21
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat

vxlan的flow规则

下面介绍一下vxlan的flow规则。
这里分析控制节点上的 flow rule，计算节点类似。

br-int 的 flow rule

root@controller:~# ovs-ofctl dump-flows br-int 
NXST_FLOW reply (xid=0x4):
 cookie=0xa1d717d78c22379b, duration=18487.092s, table=0, n_packets=1504, n_bytes=114157, idle_age=819, priority=0 actions=NORMAL
 cookie=0xa1d717d78c22379b, duration=18486.909s, table=23, n_packets=0, n_bytes=0, idle_age=18486, priority=0 actions=drop
 cookie=0xa1d717d78c22379b, duration=18486.680s, table=24, n_packets=0, n_bytes=0, idle_age=18486, priority=0 actions=drop

br-int 的 rule 逻辑很简单，br-int 被当作一个二层交换机，其重要的 rule 是下面这条：

cookie=0xa1d717d78c22379b, duration=18487.092s, table=0, n_packets=1504, n_bytes=114157, idle_age=819, priority=0 actions=NORMAL

此规则的含义是：根据 vlan 和 mac 进行转发。

br-tun 的 flow rule

root@controller:~# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
 cookie=0xbf6d56e97edab3a4, duration=18590.954s, table=0, n_packets=393, n_bytes=42787, idle_age=926, priority=1,in_port=1 actions=resubmit(,2)
 cookie=0xbf6d56e97edab3a4, duration=15047.815s, table=0, n_packets=795, n_bytes=44848, idle_age=926, priority=1,in_port=4 actions=resubmit(,4)
 cookie=0xbf6d56e97edab3a4, duration=15032.607s, table=0, n_packets=87, n_bytes=4852, idle_age=14821, priority=1,in_port=5 actions=resubmit(,4)
 cookie=0xbf6d56e97edab3a4, duration=18590.954s, table=0, n_packets=0, n_bytes=0, idle_age=18590, priority=0 actions=drop
 cookie=0xbf6d56e97edab3a4, duration=18590.953s, table=2, n_packets=339, n_bytes=38251, idle_age=926, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0xbf6d56e97edab3a4, duration=18590.953s, table=2, n_packets=54, n_bytes=4536, idle_age=13825, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 cookie=0xbf6d56e97edab3a4, duration=18590.953s, table=3, n_packets=0, n_bytes=0, idle_age=18590, priority=0 actions=drop
 cookie=0xbf6d56e97edab3a4, duration=15545.941s, table=4, n_packets=317, n_bytes=21356, idle_age=926, priority=1,tun_id=0x64 actions=mod_vlan_vid:2,resubmit(,10)
 cookie=0xbf6d56e97edab3a4, duration=15406.189s, table=4, n_packets=565, n_bytes=28344, idle_age=12874, priority=1,tun_id=0x65 actions=mod_vlan_vid:3,resubmit(,10)
 cookie=0xbf6d56e97edab3a4, duration=18590.953s, table=4, n_packets=0, n_bytes=0, idle_age=18590, priority=0 actions=drop
 cookie=0xbf6d56e97edab3a4, duration=18590.953s, table=6, n_packets=0, n_bytes=0, idle_age=18590, priority=0 actions=drop
 cookie=0xbf6d56e97edab3a4, duration=18590.952s, table=10, n_packets=1111, n_bytes=71370, idle_age=926, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xbf6d56e97edab3a4,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
 cookie=0xbf6d56e97edab3a4, duration=15047.813s, table=20, n_packets=111, n_bytes=12557, idle_age=926, priority=2,dl_vlan=2,dl_dst=fa:16:3e:7f:29:10 actions=strip_vlan,set_tunnel:0x64,output:4
 cookie=0xbf6d56e97edab3a4, duration=15032.607s, table=20, n_packets=9, n_bytes=1334, idle_age=14821, priority=2,dl_vlan=2,dl_dst=fa:16:3e:1d:eb:0e actions=strip_vlan,set_tunnel:0x64,output:5
 cookie=0xbf6d56e97edab3a4, duration=15019.610s, table=20, n_packets=81, n_bytes=9174, idle_age=12874, priority=2,dl_vlan=3,dl_dst=fa:16:3e:4b:79:a7 actions=strip_vlan,set_tunnel:0x65,output:4
 cookie=0xbf6d56e97edab3a4, duration=18590.952s, table=20, n_packets=0, n_bytes=0, idle_age=18590, priority=0 actions=resubmit(,22)
 cookie=0xbf6d56e97edab3a4, duration=15047.682s, table=22, n_packets=0, n_bytes=0, idle_age=15047, hard_age=15032, dl_vlan=2 actions=strip_vlan,set_tunnel:0x64,output:4,output:5
 cookie=0xbf6d56e97edab3a4, duration=15019.530s, table=22, n_packets=0, n_bytes=0, idle_age=15019, dl_vlan=3 actions=strip_vlan,set_tunnel:0x65,output:4
 cookie=0xbf6d56e97edab3a4, duration=18590.891s, table=22, n_packets=54, n_bytes=4536, idle_age=13825, priority=0 actions=drop

这些才是真正处理 VXLAN 数据包的 rule，流程如下：

这里写图片描述

table 0

cookie=0xbf6d56e97edab3a4, duration=18590.954s, table=0, n_packets=393, n_bytes=42787, idle_age=926, priority=1,in_port=1 actions=resubmit(,2)
cookie=0xbf6d56e97edab3a4, duration=15047.815s, table=0, n_packets=795, n_bytes=44848, idle_age=926, priority=1,in_port=4 actions=resubmit(,4)
cookie=0xbf6d56e97edab3a4, duration=15032.607s, table=0, n_packets=87, n_bytes=4852, idle_age=14821, priority=1,in_port=5 actions=resubmit(,4)
cookie=0xbf6d56e97edab3a4, duration=18590.954s, table=0, n_packets=0, n_bytes=0, idle_age=18590, priority=0 actions=drop

结合如下port编号：

root@controller:~# ovs-ofctl show br-tun
OFPT_FEATURES_REPLY (xid=0x2): dpid:00006a74e5c77644
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 1(patch-int): addr:9e:75:8e:f8:e3:1e
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 4(vxlan-0a0a0a0b): addr:1a:d8:0e:85:bd:74
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 5(vxlan-0a0a0a0c): addr:a2:9b:b0:ec:61:0c
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-tun): addr:6a:74:e5:c7:76:44
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

table 0 flow rule 的含义为：

1.从 port 1（patch-int）进来的包，扔给 table 2 处理：actions=resubmit(,2)
2.从 port 4（vxlan-0a0a0a0b）进来的包，扔给 table 4 处理：actions=resubmit(,4)
3.从 port 5（vxlan-0a0a0a0c）进来的包，扔给 table 4 处理：actions=resubmit(,4)

即第一条 rule 处理来自内部 br-int（这上面挂载着所有的网络服务，包括路由、DHCP 等）的数据；第二、三条 rule 处理来自外部 VXLAN 隧道的数据。

table 4

cookie=0xbf6d56e97edab3a4, duration=15545.941s, table=4, n_packets=317, n_bytes=21356, idle_age=926, priority=1,tun_id=0x64 actions=mod_vlan_vid:2,resubmit(,10)
cookie=0xbf6d56e97edab3a4, duration=15406.189s, table=4, n_packets=565, n_bytes=28344, idle_age=12874, priority=1,tun_id=0x65 actions=mod_vlan_vid:3,resubmit(,10)

table 4 flow rule 的含义为：

1.如果数据包的 VXLAN tunnel ID 为 100（tun_id=0x64），action 是添加内部 VLAN ID 2（tag=2），然后扔给 table 10 去学习；
2.如果数据包的 VXLAN tunnel ID 为 101（tun_id=0x65），action 是添加内部 VLAN ID 3（tag=3），然后扔给 table 10 去学习。

table 10

cookie=0xbf6d56e97edab3a4, duration=18590.952s, table=10, n_packets=1111, n_bytes=71370, idle_age=926, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xbf6d56e97edab3a4,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1

table 10 flow rule 的含义为：
学习外部（从 tunnel）进来的包，往 table 20 中添加对返程包的正常转发规则，然后从 port 1（patch-int）扔给 br-int。

rule 中下面的内容为学习规则，这里就不详细讨论了。

NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]

table 2

cookie=0xbf6d56e97edab3a4, duration=18590.953s, table=2, n_packets=339, n_bytes=38251, idle_age=926, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
cookie=0xbf6d56e97edab3a4, duration=18590.953s, table=2, n_packets=54, n_bytes=4536, idle_age=13825, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)

table 2 flow rule 的含义为：

1.br-int 发过来数据如果是单播包，扔给 table 20 处理：resubmit(,20)
2.br-int 发过来数据如果是多播或广播包，扔 table 22 处理：resubmit(,22)

table 20

cookie=0xbf6d56e97edab3a4, duration=15047.813s, table=20, n_packets=111, n_bytes=12557, idle_age=926, priority=2,dl_vlan=2,dl_dst=fa:16:3e:7f:29:10 actions=strip_vlan,set_tunnel:0x64,output:4
cookie=0xbf6d56e97edab3a4, duration=15032.607s, table=20, n_packets=9, n_bytes=1334, idle_age=14821, priority=2,dl_vlan=2,dl_dst=fa:16:3e:1d:eb:0e actions=strip_vlan,set_tunnel:0x64,output:5
cookie=0xbf6d56e97edab3a4, duration=15019.610s, table=20, n_packets=81, n_bytes=9174, idle_age=12874, priority=2,dl_vlan=3,dl_dst=fa:16:3e:4b:79:a7 actions=strip_vlan,set_tunnel:0x65,output:4
cookie=0xbf6d56e97edab3a4, duration=18590.952s, table=20, n_packets=0, n_bytes=0, idle_age=18590, priority=0 actions=resubmit(,22)

table 20 flow rule 的含义为：

1.第一条规则是 table 10 学习来的结果。内部 VLAN 号为 2（tag=2），目标 MAC 是 fa:16:3e:7f:29:10（test1）的数据包，即发送给 test1 的包，action 是去掉 VLAN 号，添加 VXLAN tunnel ID 100(十六进制 0x64)，并从 port 4 (tunnel 端口 vxlan-0a0a0a0b) 发出。
2.第二条规则也是 table 10 学习来的结果。内部 VLAN 号为 2（tag=2），目标 MAC 是 fa:16:3e:1d:eb:0e（test2）的数据包，即发送给 test2 的包，action 是去掉 VLAN 号，添加 VXLAN tunnel ID 100(十六进制 0x64)，并从 port 5 (tunnel 端口 vxlan-0a0a0a0c) 发出。
3.第三条规则也是 table 10 学习来的结果。内部 VLAN 号为 3（tag=3），目标 MAC 是 fa:16:3e:4b:79:a7（test3）的数据包，即发送给 test3 的包，action 是去掉 VLAN 号，添加 VXLAN tunnel ID 101(十六进制 0x65)，并从 port 4 (tunnel 端口 vxlan-0a0a0a0b) 发出。

对于没学习到规则的数据包，则扔给 table 22 处理。

table 22

cookie=0xbf6d56e97edab3a4, duration=15047.682s, table=22, n_packets=0, n_bytes=0, idle_age=15047, hard_age=15032, dl_vlan=2 actions=strip_vlan,set_tunnel:0x64,output:4,output:5
cookie=0xbf6d56e97edab3a4, duration=15019.530s, table=22, n_packets=0, n_bytes=0, idle_age=15019, dl_vlan=3 actions=strip_vlan,set_tunnel:0x65,output:4

table 22 flow rule 的含义为：

1.如果数据包的内部 VLAN 号为 2（tag=2），action 是去掉 VLAN 号，添加 VXLAN tunnel ID 100(十六进制 0x64)，并从 port 4、5 (tunnel 端口 vxlan-0a0a0a0b、vxlan-0a0a0a0c) 发出。
2.如果数据包的内部 VLAN 号为 3（tag=3），action 是去掉 VLAN 号，添加 VXLAN tunnel ID 101(十六进制 0x65)，并从 port 4 (tunnel 端口 vxlan-0a0a0a0b) 发出。