技术并艺术着

张华的技术Blog

Neutron DVR实现multi-host特性打通东西南北流量提前看(by quqi99)

作者:张华  发表于:2014-03-07
版权声明:可以任意转载,转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明

(http://blog.csdn.net/quqi99 )

Legacy Routing and Distributed Router in Neutron

         先温习下l3-agent原理:l3-agent节点为所有subnet创建内部网关,外部网关,路由等。l3-agent定期同步router时会为将和该router相关联的subnet调度到相应的l3-agent节点上创建网关(根据port的device_owner属性找到对应的port, port里有subnet)。neutron支持在多个节点上启动多个l3-agent, l3-agent的调度是针对router为单位的, 试想, 如果我们创建众多的router, 每一个subnet都关联到一个router的话, 那么也意味着每一个subnet的网关都可被调度到不同的l3-agent上, 从而将不同的subnet的流量分流到不同的l3-agent节点.

         但是上述方案不具有HA特性, 所以出现一个VRRP HA的Blueprint继续使用VRRP+Keepalived+Conntrackd技术解决单点l3-agent的HA问题, VRRP使用广播进行心跳检查, backup节点收不到master节点定期发出的心跳广播时便认为master死掉从而接管master的工作(删除掉原master节点上的网关的IP, 在新master节点上重设网关). 可参见我的另一博文:http://blog.csdn.net/quqi99/article/details/18799877

        但是, 上述两种方案均无法解决相同子网的东西向流量不绕道l3-agent的状况,所以Legacy Routeing in Neutron具有性能瓶颈、水平可扩展性差。

所以出现DVR,设计要点如下:

  1. 东西向流量不再走l3-agent。
  2. 计算节点上无FIP时SNAT/DNAT均走中心化的l3-agent节点。(此时计算节点有qrouter-xxx ns(qr接口为子网网关如192.168.21.1,此处采用策略路由将SNAT流量导到网络节点snat-xxx的sg接口上), 网络节点有qrouter-xxx ns与snat-xxx (qg接口为用于SNAT的外网接口,sg接口为该子网上与网关不同的IP如192.168.21.3, SNAT/DNAT规则设置在snat-xxx ns中)
  3. 计算节点有FIP时SNAT/DNAT均走计算节点。(相对于上面无FIP的情况,此时在计算节点多出fip-xxx ns, fip-xxx与qrouter-xxx通过fpr与rfp这对peer devices相连,fip-xxx中还有fg接口设置外网IP插在br-int上最终和br-ex接起来)对于SNAT, snat规则设置在qrouter-xxx上;对于DNAT,首先fip-xxx会有arp代理欺骗外部FIP的mac就是fg接口的mac地址这样外部流量到达fg接口,然后fip-xxx里的路由再将到FIP的流量导到qrouter-xxx的rfp, 然后在qrouter-xxx中设置DNAT规则导到虚机,由于采用了DNAT规则这样FIP是否设置在rfp接口都是无所谓的,在ocata版本中我看见就没设置)。
  4. DVR对VPNaaS的影响。DVR分legency, dvr, dvr_snat三种模式。dvr只用于计算节点,legency/dvr_snat只用于网络节点。legency模式下只有qrouter-xxx ns(SNAT等设置在此ns里), dvr_snat模式下才有snat_xxx(SNAT设置在此ns里)。由于vpnaas的特性它只能中心化使用,所以对于dvr router应该将VPNaaS服务只设置在中心化的snat_xxx ns中,并且将vpn流量从计算节点导到网络节点)。见:https://review.openstack.org/#/c/143203/
  5. DVR对FWaaS的影响。legency时代neutron只有一个qrouter-xxx,所以东西南北向的防火墙功能都在qrouter-xxx里;而DVR时代在计算节点多出fip-xxx(此ns中不安装防火墙规则),在网络节点多出snat-xxx处理SNAT,我们确保计算节点上的qrouter-xxx里也有南北向防火墙功能,并确保DVR在东西向流量不出问题。见:https://review.openstack.org/#/c/113359/ 与https://specs.openstack.org/openstack/neutron-specs/specs/juno/neutron-dvr-fwaas.html
  6. 自newton起,DVR与VRRP也可以结合使用
  7. DVR对LBaaS的影响,lbaas的vip是没有binding:host属性的,对于这种unbound port也应该中心化。nobound port like allowed_address_pair port and vip port - https://bugs.launchpad.net/neutron/+bug/1583694
    https://www.openstack.org/assets/presentation-media/Neutron-Port-Binding-and-Impact-of-unbound-ports-on-DVR-Routers-with-FloatingIP.pdf
    unbound port(像allowed_address_pair port和vip port),这种port可以和很多VM关联所以它没有binding:host属性(没有binding:host就意味着agentg不会处理它,因为MQ只对有host的agent发消息),这在DVR环境下造成了很多问题:如果这种port继承VM的port-binding属性,因为有多个VM也不知道该继承哪一个;当把FIP关联上这种unbound port之后(LB backend VMs中的各个port都可以关联有相同FIP的allowed_address_pair)会存在GARP问题造成FIP失效。
    解决办法是将这种unbound port中心化,即只有中心化的网络节点来处理unbound port(allowed_address_pair port and vip port),这样网络节点上的snat_xxx名空间也需要处理unbound port的DNAT等事情。代码如下:
    server/db side - https://review.openstack.org/#/c/466434/ - FIP port中没有binding:host时设置DVR_SNAT_BOUND内部变量,然后位于计算节点上的dvr-agent就别处理带DVR_SNAT_BOUND的port了 (server side别返回这种port就行,所以代码主要在server/db side)
    agent side - https://review.openstack.org/#/c/437986/ - l3-agent需要在snat-xxx里添加带DVR_SNAT_BOUND的port,并设置它的SNAT/DNAT
    然后有一个bug - Unbound ports floating ip not working with address scopes in DVR HA - https://bugs.launchpad.net/neutron/+bug/1753434
    经过上面修改后,也顺便在计算节点上多出一个叫dvr_no_external的模式(https://review.openstack.org/#/c/493321/) -  this mode enables only East/West DVR routing functionality for a L3 agent that runs on a compute host, the North/South functionality such as DNAT and SNAT will be provided by the centralized network node that is running in ‘dvr_snat’ mode. This mode should be used when there is no external network connectivity on the compute host.

代码:

server/db side: DVR VIP只是router中的一个属性(const.FLOATINGIP_KEY),它并不是有binding-host一样的普通port, 让这个unbound port中心化处理:
_process_router_update(l3-agent) -> L3PluginApi.get_routers -> sync_routers(rpc) -> list_active_sync_routers_on_active_l3_agent -> _get_active_l3_agent_routers_sync_data(l3_dvrscheduler_db.py) -> get_ha_sync_data_for_host(l3_dvr_db.py)  -> _get_dvr_sync_data -> _process_floating_ips_dvr(router[const.FLOATINGIP_KEY] = router_floatingips)

agent side: l3-agent只处理有DVR_SNAT_BOUND标签(也就是unbount ports)的port
_process_router_update -> _process_router_if_compatible -> _process_added_router(ri.process) -> process_external -> configure_fip_addresses -> process_floating_ip_addresses -> add_floating_ip -> floating_ip_added_dist(only call add_cetralized_floatingip when having DVR_SNAT_BOUND) ->  add_centralized_floatingip -> process_floating_ip_nat_rules_for_centralized_floatingip(floating_ips = self.get_floating_ips())

上面针对的是vip,对于allowed_addr_pair这种unbound port也是一样中心化处理
_dvr_handle_unbound_allowed_addr_pair_add|_notify_l3_agent_new_port|_notify_l3_agent_port_update -> update_arp_entry_for_dvr_service_port (generate arp table for every pairs of this port)


测试环境快速搭建

juju bootstrap
bzr branch lp:openstack-charm-testing && cd openstack-charm-testing/
juju-deployer -c ./next.yaml -d xenial-mitaka
juju status |grep message
juju set neutron-api l2-population=True enable-dvr=True
./configure
source ./novarc
nova boot --flavor 2 --image trusty --nic net-id=$(neutron net-list |grep ' private ' |awk '{print $2}') --poll i1
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
nova secgroup-add-rule default tcp 22 22 0.0.0.0/0


nova floating-ip-create
nova floating-ip-associate i1 $(nova floating-ip-list |grep 'ext_net' |awk '{print $2}')


neutron net-create private_2 --provider:network_type gre --provider:segmentation_id 1012
neutron subnet-create --gateway 192.168.22.1 private_2 192.168.22.0/24 --enable_dhcp=True --name private_subnet_2
ROUTER_ID=$(neutron router-list |grep ' provider-router ' |awk '{print $2}')
SUBNET_ID=$(neutron subnet-list |grep '192.168.22.0/24' |awk '{print $2}')
neutron router-interface-add $ROUTER_ID $SUBNET_ID
nova hypervisor-list
nova boot --flavor 2 --image trusty --nic net-id=$(neutron net-list |grep ' private_2 ' |awk '{print $2}') --hint force_hosts=juju-zhhuabj-machine-16 i2


相关配置

1, neutron.conf on the node hosted neutron-api
    router_distributed = True

2, ml2_conf.ini on the node hosts neutron-api
   mechanism_drivers =openvswitch,l2population

3, l3_agent on the node hosted nova-compute
   agent_mode = dvr

4, l3_agent on the node hosted l3-agent
    agent_mode = dvr_snat

5, openvswitch_agent.ini on all nodes
   [agent]
   l2_population = True
   enable_distributed_routing = True 


基于策略的路由

如路由器上有多个接口,一个为IF1(IP为IP1), 与之相连的子网为P1_NET
a, 创建路由表T1, echo 200 T1 >> /etc/iproute2/rt_tables
b, 设置main表中的缺省路由,ip route add default via $P1
c, 设置main表中的路由,ip route add $P1_NET dev $IF1 src IP1, 注意一定要加src参数, 与步骤d)配合
d, 设置路由规则,      ip rule add from $IP1 table T1, 注意与上步c)配合
e, 设置T1表中的路由,  ip route add $P1_NET dev $IF1 src $IP1 table T1 && ip route flush cache
f, 负载均衡, ip route add default scope global nexthop via $P1 dev $IF1 weight 1 \
nexthop via $P2 dev $IF2 weight 1


1, 中心化SNAT流量 (计算节点没有FIP时会这么走)

 


DVR Deployment without FIP - SNAT goes centralized l3-agent:

  • qr namespace (R ns) in compute node uses table=218103809 policy router to guide SNAT traffic from compute node to centralized network node (eg: qr1=192.168.21.1, sg1=192.168.21.3, VM=192.168.21.5)

    • ip netns exec qr ip route add 192.168.21.0/24 dev qr1 src 192.168.21.1 table main  #ingress

    • ip netns exec qr ip rule add from 192.168.21.1/24 table 218103809                          #egress

    • ip netns exec qr ip route add default via 192.168.21.3 dev qr1 table 218103809

  • SNAT namespace in network node uses the following SNAT rules

    • -A neutron-l3-agent-snat -o qg -j SNAT --to-source <FIP_on_qg>

    • -A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source <FIP_on_qg>


 完整的路径如下:

计算节点只有qrouter-xxx名空间(计算节点有FIP才有fip-xxx名空间)
$ sudo ip netns exec qrouter-9a3594b0-52d1-4f29-a799-2576682e275b ip addr show
19: qr-48e90ad1-5f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc noqueue state UNKNOWN group default qlen 1
    link/ether fa:16:3e:f8:fe:97 brd ff:ff:ff:ff:ff:ff
    inet 192.168.21.1/24 brd 192.168.21.255 scope global qr-48e90ad1-5f
       valid_lft forever preferred_lft forever

$ sudo ip netns exec qrouter-9a3594b0-52d1-4f29-a799-2576682e275b ip rule list
0: from all lookup local 
32766: from all lookup main 
32767: from all lookup default
3232240897: from 192.168.21.1/24 lookup 3232240897

$ sudo ip netns exec qrouter-9a3594b0-52d1-4f29-a799-2576682e275b ip route list table main
192.168.21.0/24 dev qr-48e90ad1-5f proto kernel scope link src 192.168.21.1

$ sudo ip netns exec qrouter-9a3594b0-52d1-4f29-a799-2576682e275b ip route list table 3232240897
default via 192.168.21.3 dev qr-48e90ad1-5f

网络节点, sg-xxx上的接口即为同子网关的一个专用于SNAT的IP
$ sudo ip netns list
qrouter-9a3594b0-52d1-4f29-a799-2576682e275b (id: 2)
snat-9a3594b0-52d1-4f29-a799-2576682e275b (id: 1)
qdhcp-7a8aad5b-393e-4428-b098-066d002b2253 (id: 0)

$ sudo ip netns exec snat-9a3594b0-52d1-4f29-a799-2576682e275b ip addr show
2: qg-b825f548-32@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc noqueue state UP group default qlen 1000
    link/ether fa:16:3e:c4:fd:ad brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.5.150.0/16 brd 10.5.255.255 scope global qg-b825f548-32
       valid_lft forever preferred_lft forever
3: sg-95540cb3-85@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc noqueue state UP group default qlen 1000
    link/ether fa:16:3e:62:d3:26 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.21.3/24 brd 192.168.21.255 scope global sg-95540cb3-85
       valid_lft forever preferred_lft forever

$ sudo ip netns exec snat-9a3594b0-52d1-4f29-a799-2576682e275b iptables-save |grep SNAT
-A neutron-l3-agent-snat -o qg-b825f548-32 -j SNAT --to-source 10.5.150.0
-A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 10.5.150.0

2, 计算节点有FIP时SNAT走计算节点

DVR Deployment without FIP - SNAT goes from qr ns to FIP ns via peer devices rfp and fpr

  • qr namespace in compute node uses table=16 policy router to guide SNAT traffic from compute node to centralized network node (eg: rfp/qr=169.254.109.46/31, fpr/fip=169.254.106.115/31, VM=192.168.21.5)

    • ip netns exec qr ip route add 192.168.21.0/24 dev qr src 192.168.21.1 table main    #ingress

    • ip netns exec qr ip rule add from 192.168.21.5 table 16                                             #egress

    • ip netns exec qr ip route add default via 169.254.106.115 dev rfp table 16

  • qr namespace in compute node also uses the following SNAT rules

-A neutron-l3-agent-float-snat -s 192.168.21.5/32 -j SNAT --to-source <FIP_on_rfp>


具实设置是,这个和下面的DNAT类似,计算节点上因为已经有FIP那SNAT也从计算节点出,故需要将SNAT流量从qrouter-xxx名空间通过rfp-xxx与fpr-xxx这对peer devices采用16这个路由表导到fip-xxx名空间。

$ sudo ip netns list
qrouter-9a3594b0-52d1-4f29-a799-2576682e275b
fip-57282976-a9b2-4c3f-9772-6710507c5c4e

$ sudo ip netns exec qrouter-9a3594b0-52d1-4f29-a799-2576682e275b ip addr show
4: rfp-9a3594b0-5@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc noqueue state UP group default qlen 1000
    link/ether 3a:63:a4:11:4d:c4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.109.46/31 scope global rfp-9a3594b0-5
       valid_lft forever preferred_lft forever
    inet 10.5.150.1/32 brd 10.5.150.1 scope global rfp-9a3594b0-5
       valid_lft forever preferred_lft forever
19: qr-48e90ad1-5f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc noqueue state UNKNOWN group default qlen 1
    link/ether fa:16:3e:f8:fe:97 brd ff:ff:ff:ff:ff:ff
    inet 192.168.21.1/24 brd 192.168.21.255 scope global qr-48e90ad1-5f
       valid_lft forever preferred_lft forever

$ sudo ip netns exec qrouter-9a3594b0-52d1-4f29-a799-2576682e275b ip rule list
0: from all lookup local 
32766: from all lookup main 
32767: from all lookup default
57481:	from 192.168.21.5 lookup 16
3232240897: from 192.168.21.1/24 lookup 3232240897

$ sudo ip netns exec qrouter-9a3594b0-52d1-4f29-a799-2576682e275b ip route list table main
169.254.109.46/31 dev rfp-9a3594b0-5 proto kernel scope link src 169.254.109.46
192.168.21.0/24 dev qr-48e90ad1-5f proto kernel scope link src 192.168.21.1

$ sudo ip netns exec qrouter-9a3594b0-52d1-4f29-a799-2576682e275b ip route list table 16
default via 169.254.106.115 dev rfp-9a3594b0-5

$ sudo ip netns exec qrouter-9a3594b0-52d1-4f29-a799-2576682e275b iptables-save |grep SNAT
-A neutron-l3-agent-float-snat -s 192.168.21.5/32 -j SNAT --to-source 10.5.150.1

$ sudo ip netns exec fip-57282976-a9b2-4c3f-9772-6710507c5c4e ip addr show
5: fpr-9a3594b0-5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc noqueue state UP group default qlen 1000
    link/ether fa:e7:f9:fd:5a:40 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.106.115/31 scope global fpr-9a3594b0-5
       valid_lft forever preferred_lft forever
17: fg-d35ad06f-ae: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc noqueue state UNKNOWN group default qlen 1
    link/ether fa:16:3e:b5:2d:2c brd ff:ff:ff:ff:ff:ff
    inet 10.5.150.2/16 brd 10.5.255.255 scope global fg-d35ad06f-ae
       valid_lft forever preferred_lft forever

$ sudo ip netns exec qrouter-9a3594b0-52d1-4f29-a799-2576682e275b route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
169.254.106.114 0.0.0.0         255.255.255.254 U     0      0        0 rfp-9a3594b0-5
192.168.21.0    0.0.0.0         255.255.255.0   U     0      0        0 qr-48e90ad1-5f

$ sudo ip netns exec fip-57282976-a9b2-4c3f-9772-6710507c5c4e route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.5.0.1        0.0.0.0         UG    0      0        0 fg-d35ad06f-ae
10.5.0.0        0.0.0.0         255.255.0.0     U     0      0        0 fg-d35ad06f-ae
10.5.150.1      169.254.106.114 255.255.255.255 UGH   0      0        0 fpr-9a3594b0-5
169.254.106.114 0.0.0.0         255.255.255.254 U     0      0        0 fpr-9a3594b0-5


3, 计算节点上的DNAT流量(一个计算节点需要两个外网IP)

   网络节点上的namespace是qrouter(管子网网关)和snat(有qg-接口与用于SNAT的,与子网网关不同的sg-接口)。

计算节点上的namespace是qrouter(管子网网关,rpf:169.254.31.29/24,FIP位于rpf接口)和fip(fg:外网网关,fpr:169.254.31.28), qrouter与fip两个名空间通过fpr与rpf两个peer设备关联,流量从fip:fg经peer设备导到qrouter里FIP。

DVR Deployment without FIP - DNAT first via arp proxy then goes from FIP ns to qr ns via peer devices rfp and fpr

  • FIP ns need to proxy ARP for FIP_on_rfp on fg(fg will finally connect to br-ex) because rfp is in another ns

    • ip netns exec fip sysctl net.ipv4.conf.fg-d35ad06f-ae.proxy_arp=1

  • The floating traffic is guided from FIP ns to qr ns via peer devices (rfp and fpr)

  • qr namespace in compute node has the following DNAT

    • ip netns exec qr iptables -A neutron-l3-agent-PREROUTING -d  <FIP_on_rfp>  -j DNAT --to-destination 192.168.21.5

    • ip netns exec qr iptables -A neutron-l3-agent-POSTROUTING ! -i rfp ! -o rfp -m conntrack ! --ctstate DNAT -j ACCEPT


   继续给上面的虚机13.0.0.6配置一个浮动IP(192.168.122.219)
a, 计算节点的qrouter名空间除了有它的网关13.0.0.1外,还会有它的浮动IP(192.168.122.219),和rfp-db5090df-8=169.254.31.28/31
   有了DNAT规则(-A neutron-l3-agent-PREROUTING -d 192.168.122.219/32 -j DNAT --to-destination 13.0.0.6)
   -A neutron-l3-agent-POSTROUTING ! -i rfp-db5090df-8 ! -o rfp-db5090df-8 -m conntrack ! --ctstate DNAT -j ACCEPT
c, 计算节点除了有qrouter名空间外,还多出一个fip名空间。它多用了一个外网IP(192.168.122.220), 和fpr-fpr-db5090df-8=169.254.31.29/31
d, 处理snat流量和之前一样由路由表21813809处理(以下所有命令均是在名空间执行的,为了清晰我去掉了)
   ip route add 13.0.0.0/24 dev rfp-db5090df-8 src 13.0.0.1
   ip rule add from 13.0.0.1 table 218103809
   ip route add 13.0.0.0/24 dev rfp-db5090df-8 src 13.0.0.1 table 218103809
e, qrouter名空间内处理dnat流量新增路由表16处理,把从虚机13.0.0.6出来的流量通过rfp-db5090df-8导到fip名空间
   ip rule add from 13.0.0.6 table 16
   ip route add default gw dev rfp-db5090df-8 table 6
f, fip名空间处理dnat流量,
   ip route add 169.254.31.28/31 dev fpr-db5090df-8
   ip route add 192.168.122.219 dev fpr-db5090df-8
   ip route add 192.168.122.0/24 dev fg-c56eb4c0-b0


4, 东西流量(同一tenant下不同子网)  

让我们先回顾下图中l2-pop的流表(来自:https://wiki.openstack.org/wiki/Ovs-flow-logic ) (实现代码是:

https://review.openstack.org/#/c/41239/2/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py ), patch_int, gre_port, vxlan_port分别是br-tun上的三个port。它的流表设计目的是:1)地址学习,学习虚机及远端tunnel的MAC。2)禁用掉入口广播。3)禁用到除gre/vxlan等tunnel出口广播以外的一切不明广播。

  • 入口流量 - 即从br-tun的gre_port或vxlan_port进入的流量,先转到table 2&3去tunnel id加lvid, 然后转到table 10去学习远程的tunnel地址,最到转到patch-int从而让流量从br-tun到br-int
  • 出口流量 - 即从br-tun的patch-int进入的流量,先转到table 1,单播转到table 20去学习VM的MAC,广播或多播转到table 22, ARP广播则转到table 21

从下图的东西向流量(同一tenant下的不同子网)的流向图可以看出,它的设计的关键是肯定会出现多个计算节点上存在同一子网的网关,所以应该各计算节点上相同子网的所有网关使用相同的IP与MAC地址,或者也可以为每个计算节点生成唯一的DVR MAC地址。所以流表需要在进出br-tun时对DVR-MAC进行封装与解封。



具体的流表在进出计算节点的流量都应该替换到DVR MAC地址,流表要在上图的基础上增加,见:https://wiki.openstack.org/wiki/Neutron/DVR_L2_Agent



 总结后的流表如下:


   

对于出口流量

table=1, priority=4, dl_vlan=vlan1, dl_type=arp, ar_tpa=gw1 actions:drop  #一计算节点所有子网到其他计算节点其网关的arp流量

 table=1, priority=4, dl_vlan=vlan1, dl_dst=gw1-mac actions:drop                 #一计算节点所有子网到其他计算节点其网关的流量

 table=1, priority=1, dl_vlan=vlan1, dl_src=gw1-mac, actions:mod dl_src=dvr-cn1-mac,resubmit(,2)  #所有出计算节点的流量使用DVR MAC

对于入口流量,还得增加一个table 9, 插在table 2&3 (改先跳到table 9) 与table 10之间:

table=9, priority=1, dl_src=dvc-cn1-mac actions=output-> patch-int

table=9, priority=0, action=-resubmit(,10)

真实的流表如下:

$ sudo ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0xb9f4f1698b73b733, duration=4086.225s, table=0, n_packets=0, n_bytes=0, idle_age=4086, priority=10,icmp6,in_port=8,icmp_type=136 actions=resubmit(,24)
 cookie=0xb9f4f1698b73b733, duration=4085.852s, table=0, n_packets=5, n_bytes=210, idle_age=3963, priority=10,arp,in_port=8 actions=resubmit(,24)
 cookie=0xb9f4f1698b73b733, duration=91031.974s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=4,in_port=1,dl_src=fa:16:3f:02:1a:a7 actions=resubmit(,2)
 cookie=0xb9f4f1698b73b733, duration=91031.667s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,in_port=2,dl_src=fa:16:3f:02:1a:a7 actions=resubmit(,1)
 cookie=0xb9f4f1698b73b733, duration=91031.178s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=4,in_port=1,dl_src=fa:16:3f:be:36:1f actions=resubmit(,2)
 cookie=0xb9f4f1698b73b733, duration=91030.686s, table=0, n_packets=87, n_bytes=6438, idle_age=475, hard_age=65534, priority=2,in_port=2,dl_src=fa:16:3f:be:36:1f actions=resubmit(,1)
 cookie=0xb9f4f1698b73b733, duration=91030.203s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=4,in_port=1,dl_src=fa:16:3f:ef:44:e9 actions=resubmit(,2)
 cookie=0xb9f4f1698b73b733, duration=91029.710s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,in_port=2,dl_src=fa:16:3f:ef:44:e9 actions=resubmit(,1)
 cookie=0xb9f4f1698b73b733, duration=91033.507s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,in_port=1 actions=drop
 cookie=0xb9f4f1698b73b733, duration=91039.210s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=NORMAL
 cookie=0xb9f4f1698b73b733, duration=91033.676s, table=0, n_packets=535, n_bytes=51701, idle_age=3963, hard_age=65534, priority=1 actions=NORMAL
 cookie=0xb9f4f1698b73b733, duration=4090.101s, table=1, n_packets=2, n_bytes=196, idle_age=3965, priority=4,dl_vlan=4,dl_dst=fa:16:3e:26:85:02 actions=strip_vlan,mod_dl_src:fa:16:3e:6d:f9:9c,output:8
 cookie=0xb9f4f1698b73b733, duration=91033.992s, table=1, n_packets=84, n_bytes=6144, idle_age=475, hard_age=65534, priority=1 actions=drop
 cookie=0xb9f4f1698b73b733, duration=91033.835s, table=2, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=1 actions=drop
 cookie=0xb9f4f1698b73b733, duration=91034.151s, table=23, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
 cookie=0xb9f4f1698b73b733, duration=4086.430s, table=24, n_packets=0, n_bytes=0, idle_age=4086, priority=2,icmp6,in_port=8,icmp_type=136,nd_target=fe80::f816:3eff:fe26:8502 actions=NORMAL
 cookie=0xb9f4f1698b73b733, duration=4086.043s, table=24, n_packets=5, n_bytes=210, idle_age=3963, priority=2,arp,in_port=8,arp_spa=192.168.22.5 actions=NORMAL
 cookie=0xb9f4f1698b73b733, duration=91038.874s, table=24, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop

$ sudo ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
 cookie=0x8502c5292d4bca36, duration=91092.422s, table=0, n_packets=165, n_bytes=14534, idle_age=4114, hard_age=65534, priority=1,in_port=1 actions=resubmit(,1)
 cookie=0x8502c5292d4bca36, duration=4158.903s, table=0, n_packets=86, n_bytes=6340, idle_age=534, priority=1,in_port=4 actions=resubmit(,3)
 cookie=0x8502c5292d4bca36, duration=4156.707s, table=0, n_packets=104, n_bytes=9459, idle_age=4114, priority=1,in_port=5 actions=resubmit(,3)
 cookie=0x8502c5292d4bca36, duration=91093.596s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
 cookie=0x8502c5292d4bca36, duration=4165.402s, table=1, n_packets=0, n_bytes=0, idle_age=4165, priority=3,arp,dl_vlan=3,arp_tpa=192.168.21.1 actions=drop
 cookie=0x8502c5292d4bca36, duration=4154.134s, table=1, n_packets=1, n_bytes=42, idle_age=4123, priority=3,arp,dl_vlan=4,arp_tpa=192.168.22.1 actions=drop
 cookie=0x8502c5292d4bca36, duration=4165.079s, table=1, n_packets=0, n_bytes=0, idle_age=4165, priority=2,dl_vlan=3,dl_dst=fa:16:3e:f8:fe:97 actions=drop
 cookie=0x8502c5292d4bca36, duration=4153.942s, table=1, n_packets=0, n_bytes=0, idle_age=4153, priority=2,dl_vlan=4,dl_dst=fa:16:3e:6d:f9:9c actions=drop
 cookie=0x8502c5292d4bca36, duration=4164.583s, table=1, n_packets=0, n_bytes=0, idle_age=4164, priority=1,dl_vlan=3,dl_src=fa:16:3e:f8:fe:97 actions=mod_dl_src:fa:16:3f:6b:1f:bb,resubmit(,2)
 cookie=0x8502c5292d4bca36, duration=4153.760s, table=1, n_packets=0, n_bytes=0, idle_age=4153, priority=1,dl_vlan=4,dl_src=fa:16:3e:6d:f9:9c actions=mod_dl_src:fa:16:3f:6b:1f:bb,resubmit(,2)
 cookie=0x8502c5292d4bca36, duration=91092.087s, table=1, n_packets=161, n_bytes=14198, idle_age=4114, hard_age=65534, priority=0 actions=resubmit(,2)
 cookie=0x8502c5292d4bca36, duration=91093.596s, table=2, n_packets=104, n_bytes=8952, idle_age=4114, hard_age=65534, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0x8502c5292d4bca36, duration=91093.596s, table=2, n_packets=60, n_bytes=5540, idle_age=4115, hard_age=65534, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 cookie=0x8502c5292d4bca36, duration=4166.465s, table=3, n_packets=81, n_bytes=6018, idle_age=534, priority=1,tun_id=0x5 actions=mod_vlan_vid:3,resubmit(,9)
 cookie=0x8502c5292d4bca36, duration=4155.086s, table=3, n_packets=109, n_bytes=9781, idle_age=4024, priority=1,tun_id=0x3f4 actions=mod_vlan_vid:4,resubmit(,9)
 cookie=0x8502c5292d4bca36, duration=91093.595s, table=3, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
 cookie=0x8502c5292d4bca36, duration=91093.594s, table=4, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
 cookie=0x8502c5292d4bca36, duration=91093.593s, table=6, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
 cookie=0x8502c5292d4bca36, duration=91090.524s, table=9, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=1,dl_src=fa:16:3f:02:1a:a7 actions=output:1
 cookie=0x8502c5292d4bca36, duration=91089.541s, table=9, n_packets=87, n_bytes=6438, idle_age=534, hard_age=65534, priority=1,dl_src=fa:16:3f:be:36:1f actions=output:1
 cookie=0x8502c5292d4bca36, duration=91088.584s, table=9, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=1,dl_src=fa:16:3f:ef:44:e9 actions=output:1
 cookie=0x8502c5292d4bca36, duration=91092.260s, table=9, n_packets=104, n_bytes=9459, idle_age=4114, hard_age=65534, priority=0 actions=resubmit(,10)
 cookie=0x8502c5292d4bca36, duration=91093.593s, table=10, n_packets=104, n_bytes=9459, idle_age=4114, hard_age=65534, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0x8502c5292d4bca36,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
 cookie=0x8502c5292d4bca36, duration=4158.170s, table=20, n_packets=0, n_bytes=0, idle_age=4158, priority=2,dl_vlan=3,dl_dst=fa:16:3e:4a:60:12 actions=strip_vlan,set_tunnel:0x5,output:4
 cookie=0x8502c5292d4bca36, duration=4155.621s, table=20, n_packets=0, n_bytes=0, idle_age=4155, priority=2,dl_vlan=3,dl_dst=fa:16:3e:f9:88:90 actions=strip_vlan,set_tunnel:0x5,output:5
 cookie=0x8502c5292d4bca36, duration=4155.219s, table=20, n_packets=0, n_bytes=0, idle_age=4155, priority=2,dl_vlan=3,dl_dst=fa:16:3e:62:d3:26 actions=strip_vlan,set_tunnel:0x5,output:5
 cookie=0x8502c5292d4bca36, duration=4150.520s, table=20, n_packets=101, n_bytes=8658, idle_age=4114, priority=2,dl_vlan=4,dl_dst=fa:16:3e:55:1d:42 actions=strip_vlan,set_tunnel:0x3f4,output:5
 cookie=0x8502c5292d4bca36, duration=4150.330s, table=20, n_packets=0, n_bytes=0, idle_age=4150, priority=2,dl_vlan=4,dl_dst=fa:16:3e:3b:59:ef actions=strip_vlan,set_tunnel:0x3f4,output:5
 cookie=0x8502c5292d4bca36, duration=91093.592s, table=20, n_packets=2, n_bytes=196, idle_age=4567, hard_age=65534, priority=0 actions=resubmit(,22)
 cookie=0x8502c5292d4bca36, duration=4158.469s, table=22, n_packets=0, n_bytes=0, idle_age=4158, hard_age=4156, dl_vlan=3 actions=strip_vlan,set_tunnel:0x5,output:4,output:5
 cookie=0x8502c5292d4bca36, duration=4155.574s, table=22, n_packets=13, n_bytes=1554, idle_age=4115, hard_age=4150, dl_vlan=4 actions=strip_vlan,set_tunnel:0x3f4,output:4,output:5
 cookie=0x8502c5292d4bca36, duration=91093.416s, table=22, n_packets=47, n_bytes=3986, idle_age=4148, hard_age=65534, priority=0 actions=drop

特性历史

  • DVR and VRRP are only supported since Juno
  • DVR only supports the use of the vxlan overlay network for Juno
  • l2-population must be disabled with VRRP before Newton
  • l2-population must be enabled with DVR for all releases
  • VRRP must be disabled with DVR before Newton
  • DVR supports augmentation using VRRP since Newton,即支持将中心化的SNAT l3-agent添加VRRP HA支持 - https://review.openstack.org/#/c/143169/
  • DVR对VPNaaS的影响。DVR分legency, dvr, dvr_snat三种模式。dvr只用于计算节点,legency/dvr_snat只用于网络节点。legency模式下只有qrouter-xxx ns(SNAT等设置在此ns里), dvr_snat模式下才有snat_xxx(SNAT设置在此ns里)。由于vpnaas的特性它只能中心化使用(neutron router-create vpn-router --distributed=False --ha=True),所以对于dvr router应该将VPNaaS服务只设置在中心化的snat_xxx ns中,并且将vpn流量从计算节点导到网络节点)。见:https://review.openstack.org/#/c/143203/ 。
  • DVR对FWaaS的影响。legency时代neutron只有一个qrouter-xxx,所以东西南北向的防火墙功能都在qrouter-xxx里;而DVR时代在计算节点多出fip-xxx(此ns中不安装防火墙规则),在网络节点多出snat-xxx处理SNAT,我们确保计算节点上的qrouter-xxx里也有南北向防火墙功能,并确保DVR在东西向流量不出问题。见:https://review.openstack.org/#/c/113359/ 与https://specs.openstack.org/openstack/neutron-specs/specs/juno/neutron-dvr-fwaas.html
  • DVR对LBaaS无影响

参考

https://wiki.openstack.org/wiki/Neutron/DVR/HowTo

https://blueprints.launchpad.net/neutron/+spec/neutron-ovs-dvr

https://wiki.openstack.org/wiki/Neutron/DVR_L2_Agent

https://docs.google.com/document/d/1kMUO1-y4yATQNBlAkdjDu7OwFYt5cfzIzb5NKQjhb08/edit?pli=1#heading=h.ttq3kyub25tn

http://www.cnblogs.com/sammyliu/p/4713562.html

https://kimizhang.wordpress.com/2014/11/25/building-redundant-and-distributed-l3-network-in-juno/

https://docs.openstack.org/newton/networking-guide/deploy-ovs-ha-dvr.html#deploy-ovs-ha-dvr


阅读更多
版权声明:本文为博主原创文章,如需转载,请注明出处! https://blog.csdn.net/quqi99/article/details/20711303
个人分类: OpenStack Networking
相关热词: neutron图
上一篇高可用OpenStack理论分析( by quqi99)
下一篇经常性无法访问某些国内网站的问题(by quqi99)
想对作者说点什么? 我来说一句

没有更多推荐了,返回首页

关闭
关闭