neutron 网络通信原理（非分布式路由）

环境说明：
L2组件：openvswitch
路由：非DVR（分布式路由）模式

1、VM to VM

A:192.168.0.4
B:192.168.0.3
computeA：vm A所在宿主机
computeB：vm B所在宿主机
A ping B

通过arp协议获取目标主机mac

tap852eac19-fe: VM A的虚拟网卡设备
tapf88c6b30-dc：VM B的虚拟网卡设备

[root@compute2 ~]# tcpdump  -i tap852eac19-fe -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap852eac19-fe, link-type EN10MB (Ethernet), capture size 262144 bytes
23:38:00.954276 ARP, Request who-has 192.0.2.3 tell 192.0.2.4, length 28
23:38:00.957103 ARP, Reply 192.0.2.3 is-at fa:16:3e:9b:3e:da, length 28

说明：tap852eac19-fe（A）上监听到arp request请求：
23:51:19.260778 ARP, Request who-has 192.0.2.3 tell 192.0.2.4, length 28

[root@compute1 yum.repos.d]# tcpdump  -i tapf88c6b30-dc -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tapf88c6b30-dc, link-type EN10MB (Ethernet), capture size 262144 bytes
23:51:19.260778 ARP, Request who-has 192.0.2.3 tell 192.0.2.4, length 28
23:51:19.261317 ARP, Reply 192.0.2.3 is-at fa:16:3e:9b:3e:da, length 28
23:51:19.264230 IP 192.0.2.4 > 192.0.2.3: ICMP echo request, id 43265, seq 0, length 64
23:51:19.264533 IP 192.0.2.3 > 192.0.2.4: ICMP echo reply, id 43265, seq 0, length 64

说明：tapf88c6b30-dc（B）上监听到来自A的arp request请求：
23:51:19.260778 ARP, Request who-has 192.0.2.3 tell 192.0.2.4, length 28
B回复A的arp请求：
23:51:19.261317 ARP, Reply 192.0.2.3 is-at fa:16:3e:9b:3e:da, length 28

经过openvswitch br-int流表规则

[root@compute2 ~]# ovs-ofctl dump-flows br-int|more
NXST_FLOW reply (xid=0x4):
 cookie=0x97845993900e0631, duration=250447.688s, table=0, n_packets=521, n_bytes=50145, idle_age=5465, hard_age=65534, priority=0 actions=resubmit(,60)

转交到60号流表

 cookie=0x97845993900e0631, duration=248740.548s, table=60, n_packets=265, n_bytes=23864, idle_age=5653, hard_age=65534, priority=100,in_port=2 actions=load:0x2->NXM_NX_REG5[],load:0x1->NXM_NX_REG6[],resubmit(,71)

tap852eac19-fe对应交换机2号口，匹配以上规则后转交到71号流表

 cookie=0x97845993900e0631, duration=250635.284s, table=71, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=110,ct_state=+trk actions=ct_clear,resubmit(,71)

如果ct_state为trk状态，则去掉该标记，并重新提交到71号流表

cookie=0x97845993900e0631, duration=248740.548s, table=71, n_packets=70, n_bytes=2940, idle_age=5656, hard_age=65534, priority=95,arp,reg5=0x2,in_port=2,dl_src=fa:16:3e:b2:23:f1,arp_spa=192.0.2.4 actions=resubmit(,94)
...
cookie=0x97845993900e0631, duration=250635.328s, table=71, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop

云平台中每台vm有分配好的ip及mac，以上流表判断A主机发起的arp请求的源mac及源ip是否正确，然提交到94号表
如果vm发出的ip或者mac不正确会被drop掉，也就杜绝了vm用户私自改ip或者mac的行为。

 cookie=0x97845993900e0631, duration=250635.274s, table=94, n_packets=263, n_bytes=23684, idle_age=5653, hard_age=65534, priority=1 actions=NORMAL

以上流表放行数据包

综上：arp请求经过br-int交换机后，在判断源mac和ip正确的情况下会继续转给br-tun交换机

经过openvswitch br-tun流表规则

[root@compute2 ~]# ovs-ofctl dump-flows br-tun
 cookie=0x32d8a2805cd98d76, duration=265344.564s, table=0, n_packets=299, n_bytes=27222, priority=1,in_port="patch-int" actions=resubmit(,2)

br-tun 与 br-int通过patch port相连，以上流表将数据流提交至2号表

 cookie=0x32d8a2805cd98d76, duration=265344.562s, table=2, n_packets=242, n_bytes=23524, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0x32d8a2805cd98d76, duration=265344.560s, table=2, n_packets=57, n_bytes=3698, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)

1、如果是广播提交至22号表，当A->B第一次发送ARP请求时即为广播

1 0.000000 fa:16:3e:b2:23:f1 Broadcast ARP 42 Who has 192.0.2.3? Tell 192.0.2.4

 cookie=0x32d8a2805cd98d76, duration=266936.530s, table=22, n_packets=61, n_bytes=3866, priority=1,dl_vlan=1 actions=strip_vlan,load:0x3->NXM_NX_TUN_ID[],output:"vxlan-0ac8895b",output:"vxlan-0ac88961",output:"vxlan-0ac8895e"

以上流表即是将广播报文发送至所有的vxlan口，并使用vxlan的vni地址0x3，0x3是我们创建网络是自动分配的provider:segmentation_id

2、如果是单播提交至20号表

 cookie=0x32d8a2805cd98d76, duration=268075.909s, table=20, n_packets=102, n_bytes=10964, priority=2,dl_vlan=1,dl_dst=fa:16:3e:a8:6c:ec actions=strip_vlan,load:0x3->NXM_NX_TUN_ID[],output:"vxlan-0ac8895b"
 cookie=0x32d8a2805cd98d76, duration=268075.906s, table=20, n_packets=0, n_bytes=0, priority=2,dl_vlan=1,dl_dst=fa:16:3e:56:f4:4e actions=strip_vlan,load:0x3->NXM_NX_TUN_ID[],output:"vxlan-0ac88961"
 cookie=0x32d8a2805cd98d76, duration=268075.902s, table=20, n_packets=15, n_bytes=1302, priority=2,dl_vlan=1,dl_dst=fa:16:3e:8e:25:da actions=strip_vlan,load:0x3->NXM_NX_TUN_ID[],output:"vxlan-0ac8895e"
 cookie=0x32d8a2805cd98d76, duration=268075.900s, table=20, n_packets=159, n_bytes=14350, priority=2,dl_vlan=1,dl_dst=fa:16:3e:9b:3e:da actions=strip_vlan,load:0x3->NXM_NX_TUN_ID[],output:"vxlan-0ac8895e"
 cookie=0x32d8a2805cd98d76, duration=267149.096s, table=20, n_packets=9, n_bytes=618, priority=2,dl_vlan=1,dl_dst=fa:16:3e:80:e5:77 actions=strip_vlan,load:0x3->NXM_NX_TUN_ID[],output:"vxlan-0ac8895b"
 cookie=0x32d8a2805cd98d76, duration=269971.491s, table=20, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,22)

20号流表是各种单播流表，会根据目标mac精确匹配出口

数据包进入compute B的虚拟交换机br-tun流程

 cookie=0x7ed8dc5da3d88808, duration=269042.009s, table=0, n_packets=235, n_bytes=19518, priority=1,in_port="vxlan-0ac88960" actions=resubmit(,4)

vxlan-0ac88960为computeA 与 computeB之间的vxlan隧道，以上流表将报文提交至4号表

 cookie=0x7ed8dc5da3d88808, duration=276504.241s, table=4, n_packets=451, n_bytes=45483, priority=1,tun_id=0x3 actions=mod_vlan_vid:1,resubmit(,10)

将外部vxlan的vni(tun_id=0x3)转换为内部vlan(mod_vlan_vid:1)，并提交至10号流表

 cookie=0x7ed8dc5da3d88808, duration=291960.861s, table=10, n_packets=451, n_bytes=45483, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0x7ed8dc5da3d88808,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:"patch-int"

待分析

cookie=0x740266734426ae73, duration=292713.084s, table=0, n_packets=920, n_bytes=90025, idle_age=2145, hard_age=65534, priority=0 actions=resubmit(,60)
cookie=0x740266734426ae73, duration=292713.084s, table=60, n_packets=109, n_bytes=8658, idle_age=2150, hard_age=65534, priority=3 actions=NORMAL

以上两条流表将arp广播报文广播至br-int上所有的port

综上

vm 至 vm之间是经过源宿主机的br-int -> br-tun 然后再到目标主机的br-tun -> br-int

2、VM至网关

网关mac地址为： fa:16:3e:80:e5:77

[root@compute2 ~]# ovs-ofctl dump-flows br-tun|grep  fa:16:3e:80:e5:77
 cookie=0x32d8a2805cd98d76, duration=269411.131s, table=20, n_packets=19, n_bytes=1542, idle_age=33, hard_age=65534, priority=2,dl_vlan=1,dl_dst=fa:16:3e:80:e5:77 actions=strip_vlan,load:0x3->NXM_NX_TUN_ID[],output:2
 cookie=0x32d8a2805cd98d76, duration=45.651s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, idle_age=45, hard_age=32, priority=1,vlan_tci=0x0001/0x0fff,dl_dst=fa:16:3e:80:e5:77 actions=load:0->NXM_OF_VLAN_TCI[],load:0x3->NXM_NX_TUN_ID[],output:2

匹配第一条规则后，从2号口发出，下面查看下2号口信息

[root@compute2 ~]# ovs-ofctl show br-tun
 2(vxlan-0ac8895b): addr:32:31:9f:98:cd:e3
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
[root@compute2 ~]# ovs-vsctl show
        Port "vxlan-0ac8895b"
            Interface "vxlan-0ac8895b"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.200.137.96", out_key=flow, remote_ip="10.200.137.91"}

2号口为计算节点至网络节点的隧道.

网络节点 br-tun流表

 cookie=0x6b35350f4b4390f1, duration=274641.568s, table=0, n_packets=183, n_bytes=16414, priority=1,in_port="vxlan-0ac88960" actions=resubmit(,4)
 ...
 cookie=0x6b35350f4b4390f1, duration=282494.521s, table=4, n_packets=602, n_bytes=58400, priority=1,tun_id=0x3 actions=mod_vlan_vid:4,resubmit(,10)
  cookie=0x6b35350f4b4390f1, duration=296771.381s, table=10, n_packets=602, n_bytes=58400, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0x6b35350f4b4390f1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:"patch-int"

经过4号，10号流表后，数据包进入br-int，然后达到目标ip（qr-b7a085c1-92）

[root@controller ~]# ip netns
qrouter-b965e86d-5b28-4ef3-b72e-73daa05a821c
qdhcp-997f6e44-56fc-4146-bf7c-fa79313f70fb
qdhcp-f3ab28fd-f8e0-4601-9823-e6b8ab06c2d2
[root@controller ~]# ip netns qrouter-b965e86d-5b28-4ef3-b72e-73daa05a821c exec ip a
Command "qrouter-b965e86d-5b28-4ef3-b72e-73daa05a821c" is unknown, try "ip netns help".
[root@controller ~]# ip netns exec qrouter-b965e86d-5b28-4ef3-b72e-73daa05a821c ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
19: qg-5996ea59-13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:aa:76:6c brd ff:ff:ff:ff:ff:ff
    inet 203.0.113.120/24 brd 203.0.113.255 scope global qg-5996ea59-13
       valid_lft forever preferred_lft forever
    inet 203.0.113.125/32 brd 203.0.113.125 scope global qg-5996ea59-13
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feaa:766c/64 scope link 
       valid_lft forever preferred_lft forever
20: qr-b7a085c1-92: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:80:e5:77 brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.1/24 brd 192.0.2.255 scope global qr-b7a085c1-92
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe80:e577/64 scope link 
       valid_lft forever preferred_lft forever

qr-b7a085c1-92口在qrouter-b965e86d-5b28-4ef3-b72e-73daa05a821c中

3、vm的浮动IP至vm

vm的浮动ip是在qrouter中进行NAT转换

Chain neutron-l3-agent-PREROUTING (1 references)
target     prot opt source               destination         
REDIRECT   tcp  --  0.0.0.0/0            169.254.169.254      tcp dpt:80 redir ports 9697
DNAT       all  --  0.0.0.0/0            203.0.113.125        to:192.0.2.4

203.0.113.125 to:192.0.2.4