VM之间的三层互访

OPENSTACK里面VM，获取IP以及三层互访，需要经过网络节点。这里面演示一台VM是如何使用dnsmasq获取IP的
之前已经看到同一子网的VM之间二层互访的路径了，三层访问都是要经过网络节点的。访问路径如下：
VM---tap3a612207-5d----网桥qbr3a612207-5d----qvb3a612207-5d----qvo3a612207-5d----OVS的br-int----patch-tun----patch-int----OVS的br-tun
到了OVS的br-tun，则通过到网络节点的VXLAN隧道进入到网络节点的OVS br-tun这个网桥
先查看网桥的配置
docker exec openvswitch_vswitchd ovs-vsctl show
ea384205-37b0-413f-bd44-06db00aee861
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
        fail_mode: secure
        datapath_type: system
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tap0e9fc4ce-c9"
            tag: 4095
            Interface "tap0e9fc4ce-c9"
                type: internal
        Port br-int
            Interface br-int
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "tap0d659160-84"
            tag: 1
            Interface "tap0d659160-84"
                type: internal
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
        fail_mode: secure
        datapath_type: system
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vxlan-0a678193"
            Interface "vxlan-0a678193"
                type: vxlan
                options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="10.103.129.149", out_key=flow, remote_ip="10.103.129.147"}
        Port "vxlan-0a678194"
            Interface "vxlan-0a678194"
                type: vxlan
                options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="10.103.129.149", out_key=flow, remote_ip="10.103.129.148"}
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
        fail_mode: secure
        datapath_type: system
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "ens33"
            Interface "ens33"
        Port br-ex
            Interface br-ex
                type: internal

从这里面可以看出VXLAN隧道会接入到Bridge br-tun 这个网桥，然后通过patch-int和patch-tun 接入到br-int这个网桥，并最终接入到tap0d659160-84这个端口。
VM---tap3a612207-5d----网桥qbr3a612207-5d----qvb3a612207-5d----qvo3a612207-5d----OVS的br-int----patch-tun----patch-int----OVS的br-tun------------VXLAN隧道-----------网络节点的vxlan隧道----patch-int----patch-tun----OVS的br-int----tap0d659160-84
查看相应的dnsmasq进程
docker exec neutron_dhcp_agent ps -ef | grep dnsmasq

nobody      140      1  0 Oct13 ?        00:00:00 dnsmasq --no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/4622d8cd-b2f5-4050-89b7-d9051e16c99e/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/4622d8cd-b2f5-4050-89b7-d9051e16c99e/host --addn-hosts=/var/lib/neutron/dhcp/4622d8cd-b2f5-4050-89b7-d9051e16c99e/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/4622d8cd-b2f5-4050-89b7-d9051e16c99e/opts --dhcp-leasefile=/var/lib/neutron/dhcp/4622d8cd-b2f5-4050-89b7-d9051e16c99e/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-dynamic --dhcp-range=set:subnet-29f8d734-695f-48ae-98b0-37f5563a2571,192.168.7.0,static,255.255.255.0,86400s --dhcp-option-force=option:mtu,1450 --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --server=1.1.1.1 --server=8.8.8.8 --server=8.8.4.4 --domain=openstacklocal
nobody      293      1  0 Oct15 ?        00:00:00 dnsmasq --no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/8d55e0d3-f6c1-49f4-834b-74565d57ded1/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/8d55e0d3-f6c1-49f4-834b-74565d57ded1/host --addn-hosts=/var/lib/neutron/dhcp/8d55e0d3-f6c1-49f4-834b-74565d57ded1/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/8d55e0d3-f6c1-49f4-834b-74565d57ded1/opts --dhcp-leasefile=/var/lib/neutron/dhcp/8d55e0d3-f6c1-49f4-834b-74565d57ded1/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-dynamic --dhcp-range=set:subnet-03c95c1a-491f-45d3-9bb3-dd72b2c4b6f3,172.16.1.0,static,255.255.255.0,86400s --dhcp-option-force=option:mtu,1450 --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --server=1.1.1.1 --server=8.8.8.8 --server=8.8.4.4 --domain=openstacklocal
nobody     1065      1  0 Oct21 ?        00:00:00 dnsmasq --no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/ae661cfd-f6df-4c70-90d3-58265c37f62e/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/ae661cfd-f6df-4c70-90d3-58265c37f62e/host --addn-hosts=/var/lib/neutron/dhcp/ae661cfd-f6df-4c70-90d3-58265c37f62e/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/ae661cfd-f6df-4c70-90d3-58265c37f62e/opts --dhcp-leasefile=/var/lib/neutron/dhcp/ae661cfd-f6df-4c70-90d3-58265c37f62e/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-dynamic --dhcp-range=set:subnet-6a084ff3-ecae-4f68-a832-44af0881094f,192.168.8.0,static,255.255.255.0,86400s --dhcp-range=set:subnet-bc90edfa-9f55-4abe-93f7-ae8691f99104,192.168.7.0,static,255.255.255.0,86400s --dhcp-option-force=option:mtu,1450 --dhcp-lease-max=512 --conf-file=/etc/neutron/dnsmasq.conf --server=1.1.1.1 --server=8.8.8.8 --server=8.8.4.4 --domain=openstacklocal
解析一下openstack中dnsmasq的进程参数,参见dnsmasq详解以及在openstack和容器中的使用_dnsmasq no-negcache-CSDN博客
dnsmasq --no-hosts #不加载本地的 /etc/hosts 文件
                --no-resolv #不读取/etc/resolv.conf.，获取dns只从命令行或者dnsmasq配置文件读取
                --strict-order  #严格按照resolv.conf中的顺序进行查找
                --except-interface=lo #不监听lo网卡
                --pid-file=/var/lib/neutron/dhcp/07b37097-c902-4f19-9214-6e23befd476b/pid 保存dnsmasq进程pid号的文件
               --dhcp-hostsfile=/var/lib/neutron/dhcp/07b37097-c902-4f19-9214-6e23befd476b/host 读取该文件保存的dhcp 主机信息
               --addn-hosts=/var/lib/neutron/dhcp/07b37097-c902-4f19-9214-6e23befd476b/addn_hosts  #添加读取额外的 hosts 文件路径
               --dhcp-optsfile=/var/lib/neutron/dhcp/07b37097-c902-4f19-9214-6e23befd476b/opts #从指定的文件中读取DHCP选项信息
               --dhcp-leasefile=/var/lib/neutron/dhcp/07b37097-c902-4f19-9214-6e23befd476b/leases #保存租约的文件
               --dhcp-match=set:ipxe,175 #设置dhcp tag
               --bind-interfaces #绑定接口，开启此项将仅监听指定的接口
               --interface=tapf2df91fa-f7  #要监听的端口
               --dhcp-range=set:tag0,10.10.0.0,static,255.255.0.0,86400s  #1）设置网络的tag,2)dhcp监听的网络范围 3）static表示dnsmasq为指定的网络启用DHCP，而不是动态分配IP地址 4）租约时间
               --dhcp-option-force=option:mtu,1450  为DHCP客户端指定不同的或额外的选项，这里为客户端指定mtu=1450
              --dhcp-lease-max=65536 指定dnsmasq为dhcp提供租约的最大个数，默认为1000
               --conf-file=/etc/neutron/dnsmasq.conf #调用其他的配置文件
               --server=8.8.8.8 --server=8.8.4.4 #指定上游域名服务器
               --domain=openstacklocal #为dhcp服务指定dns域名
这里面我们看到实际运行的dnsmasq是使用了多实例，即每个NETWORK会对应一个dnsmasq进程，我们建立了3个NETWORK，所以有3个dnsmasq进程。通过查看进程启动参数，我们发现实际运行时没有启用--interface参数。那是怎么做到多实例，每个实例监听不同的interface的呢？原来这里面使用了LINUX的NAMESPACE。
ip netns
qdhcp-ae661cfd-f6df-4c70-90d3-58265c37f62e (id: 2)
qdhcp-8d55e0d3-f6c1-49f4-834b-74565d57ded1 (id: 1)
qdhcp-4622d8cd-b2f5-4050-89b7-d9051e16c99e (id: 0)
这里面查到每个NETWORK对应了一个ns。而接口就是归于对应的ns。
我们查看一下qdhcp-4622d8cd-b2f5-4050-89b7-d9051e16c99e这个ns
ip netns exec qdhcp-4622d8cd-b2f5-4050-89b7-d9051e16c99e ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
11: tap0d659160-84: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:ab:3e:5d brd ff:ff:ff:ff:ff:ff
    inet 169.254.169.254/16 brd 169.254.255.255 scope global tap0d659160-84
       valid_lft forever preferred_lft forever
    inet 192.168.7.2/24 brd 192.168.7.255 scope global tap0d659160-84
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feab:3e5d/64 scope link
       valid_lft forever preferred_lft forever

很显然tap0d659160-84在这个qdhcp-4622d8cd-b2f5-4050-89b7-d9051e16c99e里面，而dnsmasq进程实际是在对应的ns启动的，只是使用ps查看进程启动参数时没有显示出来。真正启动命令如下：
ip netns exec qdhcp-4622d8cd-b2f5-4050-89b7-d9051e16c99e dnsmasq --no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/4622d8cd-b2f5-4050-89b7-d9051e16c99e/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/4622d8cd-b2f5-4050-89b7-d9051e16c99e/host --addn-hosts=/var/lib/neutron/dhcp/4622d8cd-b2f5-4050-89b7-d9051e16c99e/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/4622d8cd-b2f5-4050-89b7-d9051e16c99e/opts --dhcp-leasefile=/var/lib/neutron/dhcp/4622d8cd-b2f5-4050-89b7-d9051e16c99e/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-dynamic --dhcp-range=set:subnet-29f8d734-695f-48ae-98b0-37f5563a2571,192.168.7.0,static,255.255.255.0,86400s --dhcp-option-force=option:mtu,1450 --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --server=1.1.1.1 --server=8.8.8.8 --server=8.8.4.4
即前面多了ip netns exec qdhcp-4622d8cd-b2f5-4050-89b7-d9051e16c99e 这个指定了ns.
如果一个network里面有多个子网，则只是会修改dnsmasq启动参数来支持。增加对应的--dhcp-range。例如：
--dhcp-range=set:subnet-6a084ff3-ecae-4f68-a832-44af0881094f,192.168.8.0,static,255.255.255.0,86400s --dhcp-range=set:subnet-bc90edfa-9f55-4abe-93f7-ae8691f99104,192.168.7.0,static,255.255.255.0,86400s
对每个子网分配对应的网络tag，如subnet-6a084ff3-ecae-4f68-a832-44af0881094f。每个tag对应不同的子网及网络参数。启动参数里面的--dhcp-optsfiler指定了对应的文件，里面包含了网络参数，如
docker exec neutron_dhcp_agent cat /var/lib/neutron/dhcp/ae661cfd-f6df-4c70-90d3-58265c37f62e/opts
tag:subnet-6a084ff3-ecae-4f68-a832-44af0881094f,option:classless-static-route,192.168.7.0/24,0.0.0.0,169.254.169.254/32,192.168.8.2,0.0.0.0/0,192.168.8.1
tag:subnet-6a084ff3-ecae-4f68-a832-44af0881094f,249,192.168.7.0/24,0.0.0.0,169.254.169.254/32,192.168.8.2,0.0.0.0/0,192.168.8.1
tag:subnet-6a084ff3-ecae-4f68-a832-44af0881094f,option:router,192.168.8.1
tag:subnet-bc90edfa-9f55-4abe-93f7-ae8691f99104,option:dns-server,202.103.225.68
tag:subnet-bc90edfa-9f55-4abe-93f7-ae8691f99104,option:classless-static-route,192.168.8.0/24,0.0.0.0,169.254.169.254/32,192.168.7.2,0.0.0.0/0,192.168.7.1
tag:subnet-bc90edfa-9f55-4abe-93f7-ae8691f99104,249,192.168.8.0/24,0.0.0.0,169.254.169.254/32,192.168.7.2,0.0.0.0/0,192.168.7.1
tag:subnet-bc90edfa-9f55-4abe-93f7-ae8691f99104,option:router,192.168.7.1

这里要注意的是dnsmasq虽然是使用DHCP分配IP，但在这个场景里面并不是分配动态IP，而是根据虚拟机MAC分配固定IP。然后根据这个固定IP对应到相应的子网，从而再根据子网的tag分配相对应的网络参数。
VM---tap3a612207-5d----网桥qbr3a612207-5d----qvb3a612207-5d----qvo3a612207-5d----OVS的br-int----patch-tun----patch-int----OVS的br-tun------------VXLAN隧道-----------网络节点的vxlan隧道----patch-int----patch-tun----OVS的br-int----tap0d659160-84----NAMESPACE:qdhcp-4622d8cd-b2f5-4050-89b7-d9051e16c99e----dnsmasq分配固定IP及网络参数，比如DNS、网关、静态路由等。