openstack mtu (by quqi99)

需要设置mtu的设备类型

openstack中总共有6种网络设备需要设置mtu:

• 物理交换机和路由, eg: 5000， 手工设置
• 物理网卡, eg: 5000， nova vif根据network.mtu设置（见最后一节）
• 隧道接口, eg: 4958, nova vif根据network.mtu设置（见最后一节）
• 虚机接口, eg: 4958, nova vif根据network.mtu设置（见最后一节）
• 虚机与br-int之间的peer口由l2-agent中的ovs_use_veth, use_veth_interconnection, veth_mtu三个参数控制。

            if self.use_veth_interconnection:
                # enable veth to pass traffic
                int_veth.link.set_up()
                phys_veth.link.set_up()
                if self.veth_mtu:
                    # set up mtu size for veth interfaces
                    int_veth.link.set_mtu(self.veth_mtu)
                    phys_veth.link.set_mtu(self.veth_mtu)
            else:
                # associate patch ports to pass traffic
                self.int_br.set_db_attribute('Interface', int_if_name,
                                             'options', {'peer': phys_if_name})
                br.set_db_attribute('Interface', phys_if_name,
                                    'options', {'peer': int_if_name})

• 虚机里面的虚机网卡, eg: 4958， 如果network中有mtu值，则用于设置dnsmasq。

        mtu = getattr(self.network, 'mtu', 0)
        if mtu > 0:
            cmd.append('--dhcp-option-force=option:mtu,%d' % mtu)

四个和mtu相关的配置

1. Flat/VLAN：Minimum of (global_physnet_mtu or physical_network_mtus)
2. GRE: Minimum of (global_physnet_mtu, path_mtu) subtract 42
3. VXLAN: Minimum of (global_physnet_mtu, path_mtu) subtract 50

一个例子(neutron.conf)：

In the neutron.conf file:
[DEFAULT]
global_physnet_mtu = 9000

In the openvswitch_agent.ini file:
[ovs]
bridge_mappings = provider1:eth1,provider2:eth2,provider3:eth3

In the ml2_conf.ini file:
[ml2]
physical_network_mtus = provider2:4000,provider3:1500
path_mtu = 9000

1. physical_network_mtus，仅用于flat与vlan，用于设置物理网络
2. path_mtu, tenant网络最大可能的mtu，下段代码位于type_tunnel中

    def get_mtu(self, physical_network=None):
        seg_mtu = super(_TunnelTypeDriverBase, self).get_mtu()
        mtu = []
        if seg_mtu > 0:
            mtu.append(seg_mtu)
        if cfg.CONF.ml2.path_mtu > 0:
            mtu.append(cfg.CONF.ml2.path_mtu)
        version = cfg.CONF.ml2.overlay_ip_version
        ip_header_length = p_const.IP_HEADER_LENGTH[version]
        return min(mtu) - ip_header_length if mtu else 0

3. global_physnet_mtu, tenant网络默认的mtu，下列代码中的mtus是来自min(physical_network_mtus, min(path_mtu) - ip_header_length）， get_deployment_physnet_mtu来自global_physnet_mtu，如果没定义physical_network_mtus，则使用默认值global_physnet_mtu，然后用它设置network对象的mtu字段。

        max_mtu = min(mtus) if mtus else p_utils.get_deployment_physnet_mtu()
        net_mtu = network_db.get('mtu')
        # if mtu is not set in database, use the maximum possible
        return net_mtu or max_mtu

4. network_device_mtu, 位于os-vif包中，network中没有mtu值时（neutron net-show |grep mtu），用它来决定虚机接口

    def _get_mtu(self, vif):
        if vif.network and vif.network.mtu:
            return vif.network.mtu
        return self.config.network_device_mtu

nova vif

上面的参数最终算出了network.mtu, nova vif (LibvirtGenericVIFDriver)需要根据这个值去设置上述的物理网卡，隧道接口，虚机接口的mtu值。但是最新代码似乎没有为ovs类型的虚拟接口设置mtu，这应该是一个bug。

其他

1. 手动给ovs port设置mtu的命令如下

ovs-vsctl --columns=mtu_request list interface
ovs-vsctl set Interface tapddcb4e7e-88 mtu_request=1501

2. 检查mtu命令

ping -M do -i 1 -c 2 -s 5000 10.230.56.18

metadata中的mtu问题

有时候　iptables -t mangle -A PREROUTING -i ns-+ -p tcp --dport 80 -j CHECKSUM --checksum-fill
有时候　https://review.opendev.org/c/openstack/neutron/+/656359
有时候用了dpdk口，有时候用的却是bridge口

2021-01-27T12:38:49.585Z|284490|netdev_linux|WARN|error receiving Ethernet packet on tape0eb39f5-20: Network is down
2021-01-27T12:38:49.585Z|284491|dpif_netdev|ERR|error receiving data from tape0eb39f5-20: Network is down

2021-01-27T06:25:06.556Z|493749|dpif(revalidator16)|WARN|netdev@ovs-netdev: failed to put[modify] (No such file or directory) ufid:24048250-ae3b-4079-a476-3a76988ff493 skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0x2),dp_hash(0xba45e006/0xff),in_port(21),packet_type(ns=0,id=0),eth(src=fa:16:3e:5a:4b:4c/00:00:00:00:00:00,dst=fa:16:3e:ea:99:ff/00:00:00:00:00:00),eth_type(0x8100),vlan(vid=1030/0x0,pcp=0/0x0),encap(eth_type(0x0800),ipv4(src=10.243.163.30/0.0.0.0,dst=10.243.163.37/0.0.0.0,proto=6/0,tos=0/0,ttl=64/0,frag=no),tcp(src=13001/0,dst=33246/0),tcp_flags(0/0)), actions:15

$ tshark -r ns-43cbdb54-9b 'ip.addr==169.254.169.254'
237 86.852258 10.243.164.13 → 169.254.169.254 TCP 74 37500 → 80 [SYN] Seq=0 Win=26880 Len=0 MSS=8960 SACK_PERM=1 TSval=1999115463 TSecr=0 WS=128
242 87.868631 10.243.164.13 → 169.254.169.254 TCP 74 [TCP Retransmission] 37500 → 80 [SYN] Seq=0 Win=26880 Len=0 MSS=8960 SACK_PERM=1 TSval=1999116479 TSecr=0 WS=128
248 89.884108 10.243.164.13 → 169.254.169.254 TCP 74 [TCP Retransmission] 37500 → 80 [SYN] Seq=0 Win=26880 Len=0 MSS=8960 SACK_PERM=1 TSval=1999118495 TSecr=0 WS=128
263 94.012172 10.243.164.13 → 169.254.169.254 TCP 74 [TCP Retransmission] 37500 → 80 [SYN] Seq=0 Win=26880 Len=0 MSS=8960 SACK_PERM=1 TSval=1999122623 TSecr=0 WS=128
$ tshark -r ns-4ae73fdc-5f 'ip.addr==169.254.169.254'

Reference

[1] https://blog.csdn.net/quqi99/article/details/15029501