为租户下的虚机提供IPv6 DNS服务(by quqi99)

quqi99

已于 2024-02-21 16:28:45 修改

阅读量2.3k

点赞数

分类专栏： OpenStack Networking 文章标签： designate ml2-dns dns

于 2017-11-03 17:51:31 首次发布

本文链接：https://blog.csdn.net/quqi99/article/details/78437988

版权

OpenStack Networking 专栏收录该内容

79 篇文章 1 订阅

订阅专栏

问题

当虚机运行下列代码时，我们需要考虑为tenant下的VM提供DNS服务:

import dns
import dns.resolver
answers = dns.resolver.query('node1', 'AAAA')
print answers[0].address

思路

需要截获neutron port event把IP/MAC拿到写到DNS的record中去。neutron port表的fixed_ips字段（在neutron ipallocations表里）同时记录了VM的IPv4与IPv6(netaddr.IPNetwork(fixed_ip[‘ip_address’]).version == 6)地址 [1]，neutron dns_integration特性可以从这里面将IPv4与IPv6地址都取出来记录到neutron-dhcp-agent下的dnsmasq中从而实现ml2-dns内置DNS服务。

IPv6

如果只是让OpenStack tenant network支持IPv6的话，很简单，直接用下列命令（下列命令有两个重要属性：ipv6_address_mode 与 ipv6_ra_mode）。当然，如果是OpenStack Over Openstack环境的话，可以让底层provider network也支持IPv6。

neutron subnet-create --ip-version=6 --name=zhhuabj_admin_subnet_v6 --ipv6-address-mode=slaac --ipv6-ra-mode=slaac zhhuabj_admin_net 2001:db8:0:1::/64
neutron router-interface-add zhhuabj_router zhhuabj_admin_subnet_v6

上面命令相当于：

$ cat /var/lib/neutron/ra/5c33033b-a4e1-494d-ab20-e0498b423b6c.radvd.conf
interface qr-10bb0b85-53
{
   AdvSendAdvert on;
   MinRtrAdvInterval 30;
   MaxRtrAdvInterval 100;
   AdvLinkMTU 1458;
   prefix 2001:db8:0:1::/64
   {
        AdvOnLink on;
        AdvAutonomous on;
   }; 
};

$ sudo ip netns exec qdhcp-be442335-ec55-4df8-b68d-dd03fa6edf00 ps -ef|grep radvd
root     16114     1  0 Nov01 ?        00:00:00 radvd -C /var/lib/neutron/ra/5c33033b-a4e1-494d-ab20-e0498b423b6c.radvd.conf -p /var/lib/neutron/external/pids/5c33033b-a4e1-494d-ab20-e0498b423b6c.pid.radvd -m syslog
root     16115 16114  0 Nov01 ?        00:00:00 radvd -C /var/lib/neutron/ra/5c33033b-a4e1-494d-ab20-e0498b423b6c.radvd.conf -p /var/lib/neutron/external/pids/5c33033b-a4e1-494d-ab20-e0498b423b6c.pid.radvd -m syslog
$ sudo ip netns exec qrouter-5c33033b-a4e1-494d-ab20-e0498b423b6c ip addr show qr-10bb0b85-53 |grep inet6 |grep global
    inet6 2001:db8:0:2::1/64 scope global
$ sudo ip netns exec qdhcp-be442335-ec55-4df8-b68d-dd03fa6edf00 ip addr show ns-af35afad-b2 |grep inet6 |grep global
    inet6 2001:db8:0:2:f816:3eff:feef:5190/64 scope global

如果它不work的话，多半两个原因：
1, 底层OpenStack环境 (openstack over openstack)计算节点上的security group应该disable掉, 因为它有类似这种固定的anti-dhcp-spoof for ipv6规则.

-A neutron-openvswi-od7c63bee-9 -p udp -m udp --sport 547 --dport 546 -j DROP                        #Anti-dhcp-spoof for IPv6

2, radvd server端, 至少得有下列防火墙规则:

sudo ip6tables -A INPUT -p icmpv6 -j ACCEPT
sudo ip6tables -A OUTPUT -p icmpv6 -j ACCEPT
sudo ip6tables -A FORWARD -p icmpv6 -j ACCEPT
sudo ip6tables -A INPUT -p udp --dport 546:547 -j ACCEPT 
sudo ufw allow proto udp from fe80::/64 to any port 547

最好测试时先临时采用:

sudo ip6tables -F
sudo ufw disable
sudo sysctl -w net.ipv6.conf.all.forwarding=1

3, radvd进程是否正常启动

附1：若是juju搭建的OpenStack环境要enable IPv6支持的话直接在yaml里添加下列内容即可：

overrides:
  prefer-ipv6: true

附2：使用OpenStack IPv6环境时，直接设置OS_AUTH_URL环境变量指向keystone的IPv6地址即可。

export OS_AUTH_URL=${OS_AUTH_PROTOCOL:-http}://[2001:db8:0:1:f816:3eff:fe3e:5e47]:5000/v2.0

Enable ML2-DNS

该特性有dns_name与dns_domain两个重要的属性，dns_domain可用在network与floatingip中，dns_name可用在port和floatingip中。如创建network时指定dns_name (neutron port-create my-net --dns_name my-port), 这样该dns_name和IP会作为dns record。

检查OpenStack是否支持dns extention API。

neutron ext-list |grep dns
| dns-integration           | DNS Integration

如果不支持，可以修改下列两个文件去支持，dns_domain相当于dnsmasq给不同组织的IP提供DNS服务时的一个区别标志。

vi /etc/neutron/neutron.conf
dns_domain = example.org.
vi /etc/neutron/plugins/ml2/ml2_conf.ini
[ml2]
extension_drivers = port_security,dns

如果OpenStack是由juju创建，直接使用下列命令即可enable上述两个配置。

juju config neutron-api enable-ml2-dns
juju config neutron-api enable-ml2-dns=True

Test ML2-DNS

#sudo ip addr del 2001:db8:0:122::1/64 dev ens3
dig google.com @<DNS-SERVER> -p 53 AAAA
sudo tcpdump -ni ens3 ip6 host fe80::f816:3eff:feb4:8d1f
#  tcpdump -n -i ens3 icmp6 and ip6[40] == 134
sudo dhclient -6 -d ens3

ubuntu@bionic:~$ sudo tcpdump -ni ens3 ip6 host fe80::f816:3eff:feb4:8d1f
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
13:35:28.263222 IP6 fe80::f816:3eff:feb4:8d1f.546 > ff02::1:2.547: dhcp6 solicit

designate架构

在这里插入图片描述
designate的架构如上图:

designate-api, 接收来自远端用户的HTTP/HTTPS请求，通过Keystone验证远端用户的合法性，将HTTP/HTTPS请求传递给Central模块。
designate-sink, 监听来自Nova和Neutron的某些事件，用于自动生成域名资源记录，比如当监听到Nova的compute.instance.create.end事件通知后，自动创建一条对应于刚创建的实例的A记录；当监听到Nuetron的floatingip.update.end事件通知后，自动更新一条相应的A记录。
designate-central, 业务逻辑处理核心。响应API请求以及处理Sink所监听到的来自Nova和Neutron的特定通知事件。同时会存取数据库，对业务逻辑处理所产生的数据进行持久化存储。
designate-mdns, 实现了标准的DNS Notify和Zone Transfer的处理. designate-mdns is the service that sends DNS NOTIFY and answers zone transfer (AXFR) requests. This allows Designate to integrate with any DNS server that supports these very standard methods of communicating. designate-mdns also encapsulates all other forms of DNS protocol that Designate performs. For example, sending SOA queries to check that a change is live.
designate-pool-manager, 连接后端驱动，管理DNS服务器池，与MiniDNS(即designate-mdns)配合同步DNS服务器的域名以及资源记录等数据。

MiniDNS(designate-mdns)：Hidden Master设计
https://blog.csdn.net/andyron/article/details/46053241
Hidden Master是DNS网络安全管理系统设计中所推荐的一种最佳实践。主DNS服务器“隐藏”在内网防火墙背后，负责DNS域名资源的管理并同步变更到从DNS服务器；从DNS服务器部署在DMZ区域，对外提供DNS查询服务。由于主DNS服务器不接受DNS查询，增强了安全性。Designate MiniDNS功能模块就采用了Hidden Master的设计思想。所有托管到Designate中的DNS域都将MiniDNS视为主DNS服务器，而其被委托的DNS服务器都作为从DNS服务器。MiniDNS实现了标准的DNS Notify和Zone Transfer协议，负责同步DNS域名资源记录到从DNS服务器上。
其工作流程如下图:
在这里插入图片描述

首先，用户通过Desingate API创建一个example.com的DNS域；
Designate API将请求传递给Central，Central先将example.com域保存到数据库，接着发送RPC请求给Pool Manager；
Pool Manager收到来自Central的创建域名的请求之后，调用DNS后端驱动，在该域名被委托的服务器池中的所有服务器中创建example.com域。同时在这些服务器中，指定example.com的master服务器是MiniDNS；
Pool Manager完成所有从服务器上example.com域的创建之后，发送RPC请求给MiniDNS。
MiniDNS收到Pool Manager的RPC请求之后，向从服务器发送DNS Notify消息，告诉从服务器example.com有资源更新。
从服务器收到DNS Notify消息后，要求主从数据库启动Zone Transfer，域迁移的方式可以是AXFR，也可以是IXFR。
主服务器从数据库中读取为example.com域自动创建的SOA和NS记录，并将SOA和NS记录传送到从服务器。
后续任何对example.com域的变更操作都会遵循上述过程，由MiniDNS将变更同步到Designate所委派管理example.com域的DNS服务器上。

看一下代码结构, designate支持很多backend(eg: bind), 安装bind服务的机器上可使用rndc命令行工具create/delete zone remotely. The traffic between rndc and bind/named(953/tcp) is authenticated with a key. designate将为每个pool生成下列配置, 这样就可以远程运行rndc命令了(rndc -s 10.5.0.29 -p 953 -k /etc/designate/rndc.key status), 其中5353是designate-mdns监听的端口:

# cat /etc/designate/pools.yaml 
- id: 794ccc2c-d751-44fe-b57f-8894c9f5c842
  name: default
  description: Pool genergated by Juju
  ns_records:
    - hostname: openstack-au-east-2.oc.xxx.com.
      priority: 10
  nameservers:
    - host: 10.5.0.29
      port: 53
  targets:
    - type: bind9
      masters:
        - host: 10.5.0.23
          port: 5354
      options:
        host: 10.5.0.29
        rndc_host: 10.5.0.29
        rndc_key_file: /etc/designate/rndc.key
  also_notifies: []

在designate-bind节点上装有bind服务(运行在953端口, /usr/sbin/named -f -u bind), 需要确保bind能够访问/etc/bind/named.conf和/etc/bind/rndc.key, 并且能够接受从Pool Manager过来的rndc流量:

# cat /etc/bind/named.conf
include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";
include "/etc/bind/named.conf.default-zones";
controls {
  inet 127.0.0.1 allow {localhost;};
  inet 10.5.0.29 allow { 10.5.0.23; };
};
# cat /etc/bind/named.conf.options
options {
        directory "/var/cache/bind";
        dnssec-validation auto;
        auth-nxdomain no;    # conform to RFC1035
        listen-on-v6 { any; };
        allow-new-zones yes;
        request-ixfr no;
        recursion no;
        statistics-file "/var/cache/bind/named.stats";
        zone-statistics yes;
        allow-notify { 10.5.0.23; };
};

rndc命令:

rndc querylog
rndc status
dig -t A openstack-au-east-2.oc.xxx.com@10.5.0.23
dig -t MX openstack-au-east-2.oc.xxx.com@10.5.0.23

bind DNS服务器可作为缓存服务器, 主DNS服务器和辅助DNS服务器, 配置分配如下:

# 缓存服务器, 不负责解析，仅为加速，不需要注册
options {
       forward only;                    
       forwarders {                     
               168.95.1.1;
               139.175.10.20;
       };
};
# 主DNS服务器, 负责解析本地客户端请求
zone "test.com" IN {
       type master;
       file "test.com.zone";
};
# 辅助DNS服务器, 辅助服务器的区域数据都是从主服务器复制而来，其数据都是只读的. 根据序列号大小决定是否复制
zone "test.com" IN {
       type slave;
       masters {ip;};
       file "slaves/test.com.zone";
};

区域传送, 解析库文件同步的过程，即辅助DNS服务器从主DNS服务器或其他的辅助DNS服务器请求数据传输过程。

完全区域传送：传送区域的所有数据，简称AXFR
增量区域传送：传送区域中改变的数据部分，简称IXFR
bind配置中之DNS主从同步，区域安全传送
http://www.it165.net/admin/html/201403/2548.html

附件 - 一个designate问题

可能因为designate不断地升级, 而且是多网卡, 这样导致每个designate-bind unit所指定的master ip (designate unit)不是同一子网. 这样会导致create zone的时候有时候会发两次rndc addzone命令.

- designate/7 only programs designate-bind/3 [172.18.248.94] (it does not know about designate-bind/2). It also uses the .247 subnet for the Master IPs instead of the correct 248 subnet. 
- designate/8 programs both bind9 servers [172.18.248.94, 172.18.248.99] and uses the correct .248 subnet 
- designate/9 only programs designate-bind/3 and also uses the wrong master IPs.

并且因为之前的DB问题, 导致bind cache里有很多stale zone, 这样当create zone的时候, 会说zone已存在.

How to remove stale zones from bind9 dynamic zone configuration

1,check the list of zones in bind9 like so: 
cat /var/cache/bind/*.nzf|grep ^zone|awk '{print $2}'|sed 's/"$/.",/g' #the sed here adds a trailing dot so we can query the database for the same name
2, check if this zone is currently active by querying the database: 
mysql -u root -p designate 
select * from zones where name="xxx.openstack-au-east-2.oc.xxx.com." WHERE deleted_at IS NOT NULL; 
3, rndc delzone xxx.openstack-au-east-2.oc.xx.com

designate recovery service检测到一个zone长期为pending状态时, 再会继续create zone. 这样在日志中会看到一个zone被反复多次create

附件 - Designate环境搭建及测试

ml2dns只能使用neutron.conf配置文件里的域名(如:ml2dns.example.), network里可以配置外部多个顶级不固定的域名(如:neutron net-update private --dns_domain extdns.example.).
ml2dns只为internal fixed ip在dnsmasq里配置域名, designate只为external fip配置域名, dnsmasq里需要配置forwarder到designate-bind中去.

# https://paste.ubuntu.com/p/y5NQwXB95R/
./generate-bundle.sh --name heat -s xenial -r queens --num-compute 1 --heat --designate
juju deploy ./b/heat/openstack.yaml --overlay ./b/heat/o/neutron-gateway.yaml --overlay ./b/heat/o/heat.yaml --overlay ./b/heat/o/neutron-ml2dns.yaml --overlay ./b/heat/o/memcached.yaml --overlay ./b/heat/o/designate.yaml
juju config neutron-api dns-domain=ml2dns.example.
juju config neutron-api reverse-dns-lookup=True
juju config neutron-api enable-ml2-dns=True
juju config designate nameservers=ns1.extdns.example.
./configure
./tools/sec_groups.sh

  applications:
    neutron-api:
      charm: cs:~openstack-charmers-next/neutron-api
      options:
        enable-ml2-dns: True
        dns-domain: ml2dns.example.
        reverse-dns-lookup: True
        ipv4-ptr-zone-prefix-size: 24
    designate:
      charm: cs:~openstack-charmers-next/designate
      options:
        nameservers: ns1.extdns.example.         
    designate-bind:
      charm: cs:~openstack-charmers-next/designate-bind
  relations:
    - [ designate, designate-bind ]
    - [ designate, neutron-api ]


# https://docs.openstack.org/python-designateclient/latest/user/shell-v2.html
sudo apt install -y python-openstackclient python-designateclient

# create the zone for your external dns, the zone name must end in a '.'
DOMAIN_NAME=extdns.example.
openstack zone create --email extdns@example $DOMAIN_NAME

# create an 'A' record for the automaticall created NS record that points to one or more of the designate-bind units
DESIGNATE_BIND_IP=$(juju status designate-bind --format short | awk '/designate-bind/ {print $3}')
openstack recordset create --record $DESIGNATE_BIND_IP --type A $DOMAIN_NAME ns1
dig @$DESIGNATE_BIND_IP -type NS

# tell neutron that for any floating IPs for our tenant network create the the hostname in a given DNS domain
neutron net-update private --dns_domain $DOMAIN_NAME

# tell neutron to use designate bind as it's forwarder by dnsmasq's configuration '--server=10.5.0.16 --domain=ml2dns.example.'
# query path: Project instance -> Neutron dnsmasq -> Designate Bind -> External DNS
# NOTE: must DO NOT SET –dns-nameserver on the tenent subnet  # used in dns_server in ( ovn-nbctl list dhcp-option)
juju config neutron-gateway dns-servers=$DESIGNATE_BIND_IP    

# create a VM for integration test
nova keypair-add --pub-key ~/.ssh/id_rsa.pub mykey
openstack port create --network $(neutron net-show private -f value -c id) --dns-name i1 port_i1
$ openstack port show port_i1 -f value -c dns_assignment
fqdn='i1.ml2dns.example.', hostname='i1', ip_address='192.168.21.18'

openstack server create --wait --image cirros2 --flavor m1.tiny --key-name mykey --port $(openstack port show port_i1 -f value -c id) i1
public_network=$(openstack network show ext_net -f value -c id)
fip=$(openstack floating ip create $public_network -f value -c floating_ip_address)
openstack floating ip set $fip --fixed-ip-address 192.168.21.18 --port $(openstack port list --fixed-ip ip-address=192.168.21.18 -c id -f value)

# Neutron will create forward and reverse records in dnsmasq for the internal DNS.
# NOTE: must DO NOT SET –dns-nameserver on the tenent subnet, By not setting the dns server
#       the project network will default to using the dnsmasq service associated with the network. 
#       This is Neutron handling our project internal DNS records automatically.
openstack subnet unset --dns-nameserver 10.230.64.2 private_subnet
nova boot --hard i1

# test dns inside VM, 192.168.21.2 is the IP inside qdhcp-xxx namespace
$ ssh cirros@10.5.150.2 -- cat /etc/resolv.conf 
search ml2dns.example
nameserver 192.168.21.2

$ ssh cirros@10.5.150.2 -- nslookup i1.ml2dns.example 192.168.21.2
Server:    192.168.21.2
Address 1: 192.168.21.2 host-192-168-21-2.ml2dns.example
Name:      i1.ml2dns.example
Address 1: 192.168.21.18 i1.ml2dns.example

$ juju ssh neutron-gateway/0 -- cat /var/lib/neutron/dhcp/bc66c327-265b-4491-95ab-2d2e14d801d2/host |grep i1
fa:16:3e:09:b7:37,i1.ml2dns.example.,192.168.21.18


# Neutron will inform Designate to create the forward and reverse records for the floating IP in the external DNS
dig @$DESIGNATE_BIND_IP i1.extdns.example
dig @$DESIGNATE_BIND_IP -x $fip

$ dig @$DESIGNATE_BIND_IP i1.extdns.example
...
i1.extdns.example.	3600	IN	A	10.5.150.2
extdns.example.		3600	IN	NS	ns1.extdns.example.
ns1.extdns.example.	3600	IN	A	10.5.0.16
...

# We can also see the external DNS record sets in Designate
openstack recordset list extdns.example.

$ openstack recordset list extdns.example.
+--------------------------------------+---------------------+------+--------------------------------------------------------------------+--------+--------+
| id                                   | name                | type | records                                                            | status | action |
+--------------------------------------+---------------------+------+--------------------------------------------------------------------+--------+--------+
| 92c8f5d6-eeca-4b0d-968b-deed4697b787 | extdns.example.     | NS   | ns1.extdns.example.                                                | ACTIVE | NONE   |
| a0167123-714d-4c34-8542-ad8b5019c318 | extdns.example.     | SOA  | ns1.extdns.example. extdns.example. 1548816241 3505 600 86400 3600 | ACTIVE | NONE   |
| 99f97832-edea-4727-be68-73536d9145d3 | ns1.extdns.example. | A    | 10.5.0.16                                                          | ACTIVE | NONE   |
| 728808e5-e6dd-4b9d-b7fa-ad260a855ac4 | i1.extdns.example.  | A    | 10.5.150.2                                                         | ACTIVE | NONE   |
+--------------------------------------+---------------------+------+--------------------------------------------------------------------+--------+--------+

20220413更新

https://bugs.launchpad.net/charm-ovn-central/+bug/1857026
https://bugs.launchpad.net/neutron/+bug/1947127

20220420更新 - 让ovn同时支持dns与dns_domain_ports扩展

LP bug: https://bugs.launchpad.net/neutron/+bug/1947127
参考：https://docs.openstack.org/neutron/latest/admin/config-dns-int-ext-serv.html#config-dns-int-ext-serv

上面我们使用的是extension_drivers=dns这个扩展，其实还有很多扩展, 对应多个场景．
场景一，Floating IPs are published with associated port DNS attributes
FIP使用dns_domain(from network)和dns_name(from port), 就是neutron早期如我们上面文章的例子．

场景二，Floating IPs are published in the external DNS service, dns_domain和dns_name不从network和port中来，而是由用户自己指定：　
openstack floating ip create --dns-domain example.org. --dns-name my-floatingip 41fa3995-9e4a-4cd9-bb51-3e5424f2ff2a

场景三中的Use case 3c - dns_domain extention (https://docs.openstack.org/neutron/xena/admin/config-dns-int-ext-serv.html)
Use case 3c中有下列限制, 即segment id必须是在vni_ranges之外的:

For use cases 3b and 3c, the externally accessible network must meet the following requirements:
The network may not have attribute router:external set to True.
The network type can be FLAT, VLAN, GRE, VXLAN or GENEVE.
For network types VLAN, GRE, VXLAN or GENEVE, the segmentation ID must be outside the ranges assigned to project networks.

针对Use case 3c的实验1显示当segment id在vni_ranges的范围之内时, 不会为fixed port来创建external dns

$juju config neutron-api vni-ranges
1001:2000
$juju config neutron-api-plugin-ovn geneve-vni-ranges
1001:2000
$ juju ssh neutron-api/0 -- sudo grep -r 'extension_drivers' /etc/neutron/plugins/ml2/ml2_conf.ini
extension_drivers=port_security,dns_domain_ports

openstack router create testrouter
# there is no '--provider-physical-network physnet1' because it's tunnel
openstack network create net1 --provider-network-type geneve --provider-segment 1012
openstack subnet create --subnet-range 192.168.4.0/24 --network net1 --allocation-pool start=192.168.4.87,end=192.168.4.100 --gateway 192.168.4.1 subnet1
openstack router add subnet testrouter subnet1

openstack subnet set --no-dns-nameservers subnet1
openstack network set --dns-domain extdns.example. net1
DESIGNATE_BIND_IP=$(juju status designate-bind --format short | awk '/designate-bind/ {print $3}')
openstack zone create --email extdns@example extdns.example.
openstack recordset create --record $DESIGNATE_BIND_IP --type A extdns.example. ns1
openstack recordset list extdns.example.
juju config neutron-api-plugin-ovn dns-servers=$DESIGNATE_BIND_IP
juju config designate nameservers=ns1.extdns.example.

openstack port create --network net1 --dns-name i1 i1
$ openstack port show i1 |grep dns_assignment
| dns_assignment          | fqdn='i1.openstack.example.', hostname='i1', ip_address='192.168.4.88'      |
openstack recordset list extdns.example. --name i1.extdns.example.
dig @$DESIGNATE_BIND_IP i1.extdns.example.

针对Use case 3c的实验2显示当segment id不在vni_ranges的范围之内时, 就会为fixed port来创建external dns

openstack network create net2 --provider-network-type geneve --provider-segment 55
openstack subnet create --subnet-range 192.168.5.0/24 --network net2 --allocation-pool start=192.168.5.87,end=192.168.5.100 --gateway 192.168.5.1 subnet2
openstack router add subnet testrouter subnet2
openstack subnet set --no-dns-nameservers subnet2
openstack network set --dns-domain extdns.example. net2
openstack port create --network net2 --dns-name i2 i2
$ openstack port show i2 |grep dns_assignment
| dns_assignment          | fqdn='i2.extdns.example.', hostname='i2', ip_address='192.168.5.96'         |
$ openstack recordset list extdns.example. --name i2.extdns.example.
+--------------------------------------+--------------------+------+--------------+--------+--------+
| id                                   | name               | type | records      | status | action |
+--------------------------------------+--------------------+------+--------------+--------+--------+
| 7c62badf-635a-433d-8738-0af377298aa8 | i2.extdns.example. | A    | 192.168.5.96 | ACTIVE | NONE   |
+--------------------------------------+--------------------+------+--------------+--------+--------+
$ dig @$DESIGNATE_BIND_IP i2.extdns.example. |grep 192
i2.extdns.example.      3600    IN      A       192.168.5.96
$ juju ssh ovn-central/1 -- sudo ovn-nbctl list dhcp_options |grep '192.168.5.1'
options             : {classless_static_route="{169.254.169.254/32,192.168.5.87, 0.0.0.0/0,192.168.5.1}", dns_server="{10.5.3.19}", domain_name="\"openstack.example.\"", lease_time="43200", mtu="1492", router="192.168.5.1", server_id="192.168.5.1", server_mac="fa:16:3e:de:4c:b8"}

场景三中的Use case 3b - dns_domain_ports extentions.
正常情况dns_domain是来自network, 但extension_drivers=dns_domain_ports可以让port也能指定dns_domain, If the port is created in an externally accessible network, DNS records will be published for this port

openstack port create --network private --dns-name i1 --dns-domain port-domain.org. port_i1
openstack recordset list port-domain.org.

场景三中的Use case 3a - https://docs.openstack.org/neutron/latest/admin/config-dns-int-ext-serv.html#use-case-3a-the-subnet-dns-publish-fixed-ip-extension
让Port的fixed-IP也可以像FIP一样对外发布DNS(发布到external designate中）, 这需要将subnet_dns_publish_fixed_ip extention中的dns_publish_fixed_ips设置为true．如双栈场景中的IPv6作为fixed IP时添加dns可全球dns路由．

# grep -r 'extension_drivers' /etc/neutron/plugins/ml2/ml2_conf.ini
extension_drivers=port_security,subnet_dns_publish_fixed_ip
openstack network create dualstack
openstack subnet create --network dualstack dualstackv4 --subnet-range 192.0.2.0/24
openstack subnet create --network dualstack dualstackv6 --ip-version 6 --subnet-range 2001:db8:42:42::/64 --dns-publish-fixed-ip
openstack port create i3 --dns-domain extdns.example. --dns-name i3 --network dualstack
$ openstack recordset list extdns.example. --name i3.extdns.example.
+--------------------------------------+--------------------+------+--------------------+--------+--------+
| id                                   | name               | type | records            | status | action |
+--------------------------------------+--------------------+------+--------------------+--------+--------+
| 36f6bd1b-f935-4140-a781-a513e33c1935 | i3.extdns.example. | AAAA | 2001:db8:42:42::76 | ACTIVE | NONE   |
+--------------------------------------+--------------------+------+--------------------+--------+--------+
$ openstack network show dualstack |grep segmentation_id
| provider:segmentation_id  | 1585

注意一点, 像下面将dns_domain_ports与subnet_dns_publish_fixed_ip同时用时会导致’openstack port create i3 …’ hang在那儿, 所以似乎这两个配置不能同时使用.

# grep -r 'extension_drivers' /etc/neutron/plugins/ml2/ml2_conf.ini
extension_drivers=port_security,dns_domain_ports,subnet_dns_publish_fixed_ip

ovn dns

./generate-bundle.sh --name ovn --series focal --release xena --num-compute 1 --ovn --designate --use-stable-charms --run
./tools/vault-unseal-and-authorise.sh
source novarc
./configure
juju config neutron-api enable-ml2-dns=true
juju config neutron-api dns-domain=openstack.example.
juju config neutron-api reverse-dns-lookup=True
juju config neutron-api-plugin-ovn dns-servers="10.5.0.15"
juju config designate nameservers=ns1.ddi1.quqi.com.

# dns-servers应该设置为designate_bind_IP, 同时要取消subnet中关于dns-nameservers的设置, 这样虚机内才会使用designate_bind_ip作为dns
DESIGNATE_BIND_IP=$(juju status designate-bind --format short | awk '/designate-bind/ {print $3}')
juju config neutron-api-plugin-ovn dns-servers=$DESIGNATE_BIND_IP
openstack subnet set --no-dns-nameservers private_subnet
$ juju ssh ovn-central/1 -- sudo ovn-nbctl list dhcp_options |grep options
options             : {classless_static_route="{169.254.169.254/32,192.168.21.2, 0.0.0.0/0,192.168.21.1}", dns_server="{10.5.3.19}", domain_name="\"openstack.example.\"", lease_time="43200", mtu="1492", router="192.168.21.1", server_id="192.168.21.1", server_mac="fa:16:3e:8f:d3:d4"}
openstack subnet show private_subnet |grep -E 'dns|dhcp'
| dns_nameservers      |                                      |
| dns_publish_fixed_ip | None                                 |
| enable_dhcp          | True                                 |

# 在network中设置dns-domain使用prd.cloud.quqi.com.，这样也需要为它在designate中配置zone,
openstack network set --dns-domain prd.cloud.quqi.com. private
$ openstack network show private |grep -i -E 'dns|external'
| dns_domain                | prd.cloud.quqi.com.               |
| router:external           | Internal                             |

openstack zone create --email extdns@example prd.cloud.quqi.com.
openstack recordset create --record $DESIGNATE_BIND_IP --type A prd.cloud.quqi.com. ns1

# 这样以private_subnet来创建虚机时，因为network中倒置了dns-domain就会自动地将fixed_ip也配置使用external dns(bionic-080146.prd.cloud.quqi.com)
openstack server create --wait --image bionic --flavor m1.small --key-name testkey --nic net-id=25651e05-ecec-44a8-9ba8-534f6061d5e9 --min 1 --max 1 bionic-080146

ubuntu@bionic-080146:~$ grep -r 'search' /etc/resolv.conf
search openstack.example
ubuntu@bionic-080146:~$ sudo systemd-resolve --status |grep -E 'DNS Servers|DNS Domain'
         DNS Servers: 10.5.3.19
          DNS Domain: openstack.example
ubuntu@bionic-080146:~$ nslookup bionic-080146.openstack.example 10.5.3.19
;; connection timed out; no servers could be reached
ubuntu@bionic-080146:~$ nslookup bionic-080146.prd.cloud.quqi.com 10.5.3.19
Name:   bionic-080146.prd.cloud.quqi.com
Address: 192.168.21.187

$ openstack recordset list prd.cloud.quqi.com. --name bionic-080146.prd.cloud.quqi.com.
+--------------------------------------+--------------------------------------+------+----------------+--------+--------+
| id                                   | name                                 | type | records        | status | action |
+--------------------------------------+--------------------------------------+------+----------------+--------+--------+
| 9ed3bf26-343f-44c6-bee1-c1e2ebff084e | bionic-080146.prd.cloud.quqi.com. | A    | 192.168.21.187 | ACTIVE | NONE   |
+--------------------------------------+--------------------------------------+------+----------------+--------+--------+

$ dig @$DESIGNATE_BIND_IP bionic-080146.prd.cloud.quqi.com |grep 192
bionic-080146.prd.cloud.quqi.com. 3600 IN A  192.168.21.187

20221025 - SECONDARY zone

designate v2 api(https://docs.openstack.org/designate/latest/user/secondary-zones.html)允许将designate作为一个dns slave rather than a master for a zone, This is accomplished by completing a zone transfer (AXFR) from a DNS server managed outside of Designate.

1, set up a designate as dns slave

./generate-bundle.sh --name dns -s focal -r xena --designate --run
./configure
juju config designate nameservers=ns1.extdns.example.
DOMAIN_NAME=extdns.example.
openstack zone create --email extdns@example $DOMAIN_NAME
DESIGNATE_BIND_IP=$(juju status designate-bind --format short | awk '/designate-bind/ {print $3}')
juju config neutron-gateway dns-servers=$DESIGNATE_BIND_IP
openstack recordset create --record $DESIGNATE_BIND_IP --type A $DOMAIN_NAME ns1
dig @$DESIGNATE_BIND_IP -type NS
neutron net-update private --dns_domain $DOMAIN_NAME
nova keypair-add --pub-key ~/.ssh/id_rsa.pub mykey
openstack port create --network $(neutron net-show private -f value -c id) --dns-name i1 port_i1
openstack server create --wait --image cirros2 --flavor m1.tiny --key-name mykey --port $(openstack port show port_i1 -f value -c id) i1
public_network=$(openstack network show ext_net -f value -c id)
fip=$(openstack floating ip create $public_network -f value -c floating_ip_address)
openstack floating ip set $fip --fixed-ip-address 192.168.21.107 --port $(openstack port list --fixed-ip ip-address=192.168.21.107 -c id -f value)
#openstack subnet unset --dns-nameserver 10.230.64.2 private_subnet
nova boot --hard i1

2, create a dns master

openstack server create --image auto-sync/ubuntu-focal-daily-amd64-server-20221021-disk1.img --flavor m1.large bind --network zhhuabj_admin_net --key mykey
BIND_IP=$(openstack server show bind | grep addresses | sed -e 's/.*=//' -e 's/ .*//')
ssh $BIND_IP
sudo apt install bind9 -y
cat <<EOF | sudo tee /etc/bind/named.conf.options
options {
listen-on port 53 { 0.0.0.0/0; };
allow-query { any; };
allow-transfer { any; };
directory "/var/cache/bind";
dnssec-validation auto;
};
EOF
cat <<EOF | sudo tee /etc/bind/named.conf.local
zone "example.tld" IN {
type master;
file "/etc/bind/example.tld.db";
};
EOF
cat <<"EOF" | sudo tee /etc/bind/example.tld.db
$TTL 5m
@ IN SOA ns.example.tld. email.example.tld. 90 4h 15m 8h 4m
@ IN NS ns.example.tld.
ns IN A 1.1.1.1
test IN A 1.2.3.4
EOF
sudo systemctl restart named

$ dig +noall +authority @$BIND_IP -p 53 example.tld
example.tld.            240     IN      SOA     ns.example.tld. email.example.tld. 90 14400 900 28800 240
$ dig +noall +answer @$BIND_IP -p 53 example.tld axfr
example.tld.            300     IN      SOA     ns.example.tld. email.example.tld. 90 14400 900 28800 240
example.tld.            300     IN      NS      ns.example.tld.
ns.example.tld.         300     IN      A       1.1.1.1
test.example.tld.       300     IN      A       1.2.3.4
example.tld.            300     IN      SOA     ns.example.tld. email.example.tld. 90 14400 900 28800 240
#test AXFR with dnspython:
sudo apt install python3-dnspython -y
ubuntu@bind:~$ python3 -q
>>> import dns.query
>>> import dns.zone
>>> axfr = dns.zone.from_xfr(dns.query.xfr(where='127.0.0.1', port=12753, zone='example.tld', rdtype=dns.rdatatype.AXFR))
>>> axfr = dns.zone.from_xfr(dns.query.xfr(where='127.0.0.1', port=53, zone='example.tld', rdtype=dns.rdatatype.AXFR))
>>> for node in axfr.nodes.keys():
...     print(axfr.nodes[node].to_text(node))
... 
@ 300 IN SOA ns email 90 14400 900 28800 240
@ 300 IN NS ns
ns 300 IN A 1.1.1.1
test 300 IN A 1.2.3.4

3, create SECONDARY zone

dig @$BIND_IP example.tld axfr
openstack zone create example.tld. --type SECONDARY --master $BIND_IP
openstack zone list | grep SECONDARY | awk '{ print $2 }' | xargs -n 1 openstack zone show
openstack zone list --all-projects

4, some outputs

# grep 'example.tld' /var/log/designate/designate-mdns.log |tail -n2
2022-10-25 06:23:09.377 27318 INFO designate.dnsutils [req-fbef1b90-3773-45cb-aaf3-6a464f84b65b 7877a98221744a0d914794e928f5d2fa 82e14f00cc03478fb165c496678242d2 - - -] Doing AXFR for example.tld. from {'zone_id': '44fddde9-3499-4495-920d-81d55a130657', 'host': '10.5.3.184', 'port': 53, 'id': '98d4b9d8-3849-4dd4-8610-b3d85555d5da', 'created_at': datetime.datetime(2022, 10, 25, 6, 19, 53), 'updated_at': datetime.datetime(2022, 10, 25, 6, 23, 9), 'version': 495} 10.5.3.184
2022-10-25 06:23:09.391 27318 DEBUG designate.dnsutils [req-fbef1b90-3773-45cb-aaf3-6a464f84b65b 7877a98221744a0d914794e928f5d2fa 82e14f00cc03478fb165c496678242d2 - - -] AXFR Successful for example.tld. do_axfr /usr/lib/python3/dist-packages/designate/dnsutils.py:364

# grep 'ZoneMasterNotFound' /var/log/designate/designate-mdns.log |tail -n2
2022-10-25 06:24:16.056 27318 ERROR oslo_messaging.rpc.server designate.exceptions_Remote.ZoneMasterNotFound_Remote: Could not find ZoneMaster
2022-10-25 06:24:16.056 27318 ERROR oslo_messaging.rpc.server designate.exceptions.ZoneMasterNotFound: Could not find ZoneMaster

20231212 - OVN DNS

当enable-ml2-dns=true时:

从same subnet里访问dns无法处理EDNS0记录(dig +noall +answer @10.5.3.17 jammy-185130.admin.openstack.stsstack.qa.1ss, 用’+noedns or +tcp’可作为workaround, 或者用workaround: resolvectl flush-caches ; dig +noedns @10.5.1.118 vm-1.example.com)
从another subnet或outside oppenstack to designate-bind则无问题 ( client发出来的arp request不会到VM所在的local ovn-ontroller，而是直接到了desigante, 所以不会遇到这个问题).
当从same subnet时，ovn flow会被VM的udp 53 traffic转到local ovn-controller, 然后ovn flow再决定是直接响应还是转发到真实dns-server - https://docs.openstack.org/charm-guide/latest/admin/networking/ovn/internal-dns.html

dns究竟是不是local ovn-controller来处理的，可以这样判断, 打开日志(sudo ovn-appctl vlog/set pinctrl:file:DBG)之后，/var/log/ovn/ovn-controller.log中看到下列日志代表request转给designate了，若ovn-controller自己处理的dns将不会看到日志。

# ovn -  it never reaches designate-bind
14:24:26.000109 IP (tos 0x0, ttl 64, id 7799, offset 0, flags [none], proto UDP (17), length 81)
172.16.0.76.38961 > juju-585b44-sf00360323-sicredi-2.cloud.sts.domain: [bad udp cksum 0xb997 -> 0xd41d!] 6070+ [1au] A? jammy-210841. ar: . OPT UDPsize=1232 (53)
14:24:26.001746 IP (tos 0x0, ttl 64, id 7799, offset 0, flags [none], proto UDP (17), length 109)
juju-585b44-sf00360323-sicredi-2.cloud.sts.domain > 172.16.0.76.38961: [no cksum] 6070-$ q: A? jammy-210841. 1/0/0 . OPT UDPsize=1232 (81)

# and in ovs:
2023-05-19T14:24:26.001Z|01775|vconn(ovn_pinctrl0)|DBG|unix:/var/run/openvswitch/br-int.mgmt: sent (Success): NXT_RESUME (OF1.5) (xid=0x0): cookie=0xf5e908e total_len=123 reg0=0x293,reg11=0x5,reg12=0x6,reg13=0x9,reg14=0x3,metadata=0x3,in_port=15 (via action) data_len=123 (unbuffered)
continuation.bridge=4fd8df5f-bd1d-4386-b4d3-d4ec00a9f0c4
continuation.actions=unroll_xlate(table=0, cookie=0),resubmit(,30)
continuation.odp_port=3
udp,vlan_tci=0x0000,dl_src=fa:16:3e:6d:69:00,dl_dst=fa:16:3e:ee:be:49,nw_src=172.16.0.76,nw_dst=10.5.2.232,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=38961,tp_dst=53 udp_csum:0
2023-05-19T14:24:26.001Z|01776|pinctrl(ovn_pinctrl0)|DBG|pinctrl received packet-in | opcode=DNS_LOOKUP| OF_Table_ID=0| OF_Cookie_ID=0xf5e908e| in-port=15| src-mac=fa:16:3e:6d:69:00, dst-mac=fa:16:3e:ee:be:49| src-ip=172.16.0.76, dst-ip=10.5.2.232

解决方法就是ovn-controller发现是EDNS查询时就自己不处理直接转给ovn-controller处理 - https://github.com/ovn-org/ovn/issues/192
fix - https://github.com/ovn-org/ovn/commit/4b10571aa89b226c13a8c5551ceb7208d782b580
在22.03.3里 - https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2034675

这样也会导致另外一个bug, 如果在same subnet里用’host IP’时会看到ml2-dns的internal domain name ( dig +tcp 也不会有问题), 而从same subnet里看到的却是desginate的external domain name。这就是因为在same subnet时会经过local ovn-controller，它没有将dns转发给designate. 而在different subnet时不会经过local ovn-controller

其实升级之后会继续遇到一个问题，就是用dig时问题解决了，但用host/nslookup时仍然会出问题。因为在计算节点上tcpdump (sudo ovs-tcpdump -w dns-query-00374248-2.pcap -i tapbed0a828-75 not port 22)就能发现用host/nslookup时是没有"Additional Records"的，但用dig是有"Additional Records"的。这也正常，也就是说dig支持edns0(就是udp能大于512字节）, 而nslookup不支持edns0。而ovn-controller现在是判断"Additional Records"时(如反向查询带了cookie时显然就大于了udb 512bytes)就往desginate里转。而host/nslookup时没有"Additional Records"就不会转了就直接用了ml2-dns来给出了internal dns.
难道是要disable ml2-dns只enable designate吗？显然也不是，其实问题出在这里，需要设置openstack subnet set --no-dns-nameservers private_subnet来让VM里直接用desginate IP作为DNS来直接解析external dns, 在 enable ml2-dns之后，此时还会为port dns_assignment生成internal dns，但实际也为fixed-ip生成了external dns - https://docs.openstack.org/neutron/latest/admin/config-dns-int-ext-serv.html#configuration-of-the-externally-accessible-network-for-use-cases-3b-and-3c , 所以VM直接用desginate IP作为DNS时就会解析为external dns。见：为租户下的虚机提供IPv6 DNS服务 - https://blog.csdn.net/quqi99/article/details/78437988
为什么之前用ovs没问题而升级ovn后就有问题了呢？因为用ovs即使subnet的dns不为空可能neutron-gateway的dnsmasq也会自动往desginate里转.

# tell neutron that for any floating IPs for our tenant network create the the hostname in a given DNS domain
neutron net-update private --dns_domain $DOMAIN_NAME
# tell neutron to use designate bind as it's forwarder by dnsmasq's configuration '--server=10.5.0.16 --domain=ml2dns.example.'
# query path: Project instance -> Neutron dnsmasq -> Designate Bind -> External DNS
# NOTE: must DO NOT SET –dns-nameserver on the tenent subnet
juju config neutron-gateway dns-servers=$DESIGNATE_BIND_IP

为此，我做了一个实验, 这个是正常搭建designate的步骤：

./generate-bundle.sh --name ovn --series jammy --num-compute 1 --ovn --designate --use-stable-charms --run
./tools/vault-unseal-and-authorise.sh
DESIGNATE_BIND_IP=$(juju status designate-bind --format short | awk '/designate-bind/ {print $3}')
juju config neutron-api enable-ml2-dns=true
juju config neutron-api dns-domain=test.ucloud.internal.
juju config neutron-api reverse-dns-lookup=True
#juju config neutron-api-plugin-ovn dns-servers=$DESIGNATE_BIND_IP
juju config neutron-api-plugin-ovn dns-servers=10.5.0.15
juju config designate nameservers=ns1.ucloud.external.
# segmentation_id should not be in the range of vni-ranges if you want to create external dns for a fixed-ip
juju config neutron-api vni-ranges=1001:2000
$ openstack network show private |grep 'provider:segmentation_id'
| provider:segmentation_id  | 5                                    |

# NOTE: 注意，这个创建subnet的命令应该在上面的'juju config neutron-api-plugin-ovn dns-servers'之后运行，否则ovn dhcp-options里的dns-server并不会更新
source ~/novarc && ./configure

 #设置它了优先，不设置它用neutron-api-plugin-ovn中设置的dns-servers (它是设置neutron.conf不会自动更新)
#openstack subnet set --no-dns-nameservers private_subnet   
openstack network set --dns-domain ucloud.external. private             #这个很重要
openstack zone create --email extdns@example ucloud.external.      #这两句也很重要
openstack recordset create --record $DESIGNATE_BIND_IP --type A ucloud.external. ns1

./tools/instance_launch.sh 1 jammy
./tools/float_all.sh
$ openstack server list |grep jammy
| 32a4baa4-0ef1-4795-9489-3f38b8987a94 | jammy-045150 | ACTIVE | private=10.5.151.224, 192.168.21.238 | jammy | m1.small |

# 注意：下面的实验结果实际上是ovn dhcp-options的dns-server为10.5.0.15(不是desginate-bind unit IP)时的输出
# confirm designate has created a external dns(jammy-045150.ucloud.external.) for fixed-ip(192.168.21.238)
$ openstack recordset list ucloud.external.
+--------------------------------------+-------------------------------+------+---------------------------------------------------------------------+--------+--------+
| id                                   | name                          | type | records                                                             | status | action |
+--------------------------------------+-------------------------------+------+---------------------------------------------------------------------+--------+--------+
| cabe0573-4c27-4cc8-b286-2e7001c5f2f2 | ucloud.external.              | SOA  | ns1.ucloud.external. extdns.example. 1706158325 3530 600 86400 3600 | ACTIVE | NONE   |
| cbc46250-bd07-4ccc-9a26-23d121d1b2e1 | ucloud.external.              | NS   | ns1.ucloud.external.                                                | ACTIVE | NONE   |
| fbb32547-0e37-4948-8a71-2755661e20ea | ns1.ucloud.external.          | A    | 10.5.4.21                                                           | ACTIVE | NONE   |
| b27d3b0f-e77c-454a-9aee-918b87a6786e | jammy-045150.ucloud.external. | A    | 192.168.21.238                                                      | ACTIVE | NONE   |
+--------------------------------------+-------------------------------+------+---------------------------------------------------------------------+--------+--------+

# confirm ovn/ml2-dns has created a external dns for fixed-ip as well
$ openstack port list |grep '192.168.21.238' |awk '{print $2}' |xargs -i openstack port show {} |grep dns
| dns_assignment          | fqdn='jammy-045150.ucloud.external.', hostname='jammy-045150', ip_address='192.168.21.238' |
| dns_domain              |                                                                                            |
| dns_name                | jammy-045150                                                                               |
# ovn-sbctl list DNS
_uuid               : 0ebe1041-68fa-4686-87ac-d5a8408bebab
datapaths           : [948e65f1-809b-4424-9214-392560f89fcc]
external_ids        : {dns_id="16b9abb8-86d0-4384-aa83-b8effee33f6b"}
records             : {"238.21.168.192.in-addr.arpa"=jammy-045150.ucloud.external, jammy-045150="192.168.21.238", jammy-045150.ucloud.external="192.168.21.238"}

$ ssh -i ~/testkey.priv ubuntu@10.5.151.224 -- cat /etc/resolv.conf |tail -n3
nameserver 127.0.0.53
options edns0 trust-ad
search test.ucloud.dc3.internal

$ ssh -i ~/testkey.priv ubuntu@10.5.151.224 -- dig -x 192.168.21.238 |grep 192
; <<>> DiG 9.18.1-1ubuntu1.3-Ubuntu <<>> -x 192.168.21.238
;238.21.168.192.in-addr.arpa.   IN      PTR
238.21.168.192.in-addr.arpa. 3525 IN    PTR     jammy-045150.ucloud.external.
$ ssh -i ~/testkey.priv ubuntu@10.5.151.224 -- host 192.168.21.238
238.21.168.192.in-addr.arpa domain name pointer jammy-045150.ucloud.external.
$ ssh -i ~/testkey.priv ubuntu@10.5.151.224 -- nslookup 192.168.21.238 |head -n1
238.21.168.192.in-addr.arpa     name = jammy-045150.ucloud.external.

$ ssh -i ~/testkey.priv ubuntu@10.5.151.224 -- ping jammy-045150.ucloud.external -c1
64 bytes from jammy-045150.ucloud.external (192.168.21.238): icmp_seq=1 ttl=64 time=0.016 ms
$ ssh -i ~/testkey.priv ubuntu@10.5.151.224 -- ping jammy-045150.ucloud.internal -c1
ping: jammy-045150.ucloud.internal: Name or service not known

但客户的环境似乎并没有使用desgiante, 它们的下列参数指向一个似乎可以解析external dns的自定义的dns server(反正不是desginate dns), 如果subnet中的dns_nameservers是空的话就会用下列的neutron-api-plugin-ovn中指定的dns-servers.

juju config neutron-api-plugin-ovn dns-servers=$DESIGNATE_BIND_IP
openstack subnet set --dns-nameserver $DESIGNATE_BIND_IP private_subnet
juju config neutron-api enable-ml2-dns=false
注意，上面的'juju config neutron-api-plugin-ovn dns-servers=$DESIGNATE_BIND_IP'并无法修改dhcp-options中的dns_server所以不得不修改subnet中的dns-nameserver来变通。（若设置了subnet里的dns-server优先用它，如果subnet里的dns-server为空则用neutron-api-plugin-ovn设置在/etc/neutron/neutron.conf里的dns-server但这里不会修改已经创建的subnet)
root@juju-a949eb-ovn-12:/home/ubuntu# ovn-nbctl list dhcp-options |grep dns_server
options             : {classless_static_route="{169.254.169.254/32,192.168.21.3, 0.0.0.0/0,192.168.21.1}", dns_server="{10.5.4.21}", lease_time="43200", mtu="1492", router="192.168.21.1", server_id="192.168.21.1", server_mac="fa:16:3e:b1:fc:b5"}

nova delete jammy-045150
./tools/instance_launch.sh 1 jammy
$ openstack server list |grep 192
| be5893ea-86db-4e80-ae25-84168b4a28fd | jammy-064631 | ACTIVE | private=10.5.151.224, 192.168.21.196 | jammy | m1.small |
# ovn-sbctl list DNS |grep records
records             : {"128.21.168.192.in-addr.arpa"=jammy-054146.ucloud.external, jammy-054146="192.168.21.128", jammy-054146.ucloud.external="192.168.21.128"}
$ ssh -i ~/testkey.priv ubuntu@10.5.151.224 -- sudo resolvectl status |grep 'Current DNS' -A3
Current DNS Server: 10.5.4.21
       DNS Servers: 10.5.4.21
$ ssh -i ~/testkey.priv ubuntu@10.5.151.224 -- cat /etc/resolv.conf |tail -n3
nameserver 127.0.0.53
options edns0 trust-ad
search .
$ ssh -i ~/testkey.priv ubuntu@10.5.151.224 -- dig -x 192.168.21.196 |grep 192
; <<>> DiG 9.18.1-1ubuntu1.3-Ubuntu <<>> -x 192.168.21.196
;196.21.168.192.in-addr.arpa.   IN      PTR
196.21.168.192.in-addr.arpa. 0  IN      PTR     jammy-064631.
196.21.168.192.in-addr.arpa. 0  IN      PTR     jammy-064631.local.

上面由于同时disable ml2-dns所以internal与external dns都没有了，看样子，还是得必须enable ml2-dns。这样都是正常的。

openstack subnet set --dns-nameserver $DESIGNATE_BIND_IP private_subnet
juju config neutron-api enable-ml2-dns=true

运行"openstack subnet set --dns-nameserver 8.8.8.8 private_subnet"后’ovn-nbctl list dhcp-options’中的会变成8.8.8.8
但是运行"openstack subnet set --no-dns-nameservers private_subnet"后它又会变成neturon.conf(juju config neutron-api-plugin-ovn dns-servers)中的默认值。
直接运行"juju config neutron-api-plugin-ovn dns-servers=8.8.4.4"不会更改ovn中的dns-server值，但会修改neutron.conf里的值，所以在重启neutron-server服务后再同时运行"openstack subnet set --dns-nameserver 8.8.8.8 private_subnet && openstack subnet set --no-dns-nameservers private_subnet"才会修改为8.8.4.4

整个过程总结如下：
1, ml2-dns必须打开，打开之后下列设置会让VM的resolve.conf中配置dns_server=10.5.0.15 (或者用juju config neutron-api-plugin-ovn dns-servers来指定默认的，若subnet里不设置会用默认的)

openstack subnet set --dns-nameserver 10.5.0.15 private_subnet
# ovn-nbctl list dhcp-options |grep dns_server
options             : {classless_static_route="{169.254.169.254/32,192.168.21.3, 0.0.0.0/0,192.168.21.1}", dns_server="{10.5.0.15}", domain_name="\"test.ucloud.internal.\"", lease_time="43200", mtu="1492", router="192.168.21.1", server_id="192.168.21.1", server_mac="fa:16:3e:b1:fc:b5"}

2, 之前默认是ml2-dns只为 fixed-ip生成internal dns, 而designate只为FIP生成external dns, 现在在配置"extension_drivers=dns_domain_ports"且保证segmentation_id不在vni-ranges的范围之后desginate也可以为fixed-ip生成external ip (也得在network上设置dns-domain=ucloud.external.)

juju config neutron-api vni-ranges=1001:2000
$ openstack network show private |grep 'provider:segmentation_id'
| provider:segmentation_id  | 5                                    |
openstack network set --dns-domain ucloud.external. private
openstack zone create --email extdns@example ucloud.external.
openstack recordset create --record $DESIGNATE_BIND_IP --type A ucloud.external. ns1

3, 照上步配置之后，OVN SB就会为fixed-IP配置external dns, 然后虚机所在的节点上的local ovn-controller上就会应答 external dns, 最终转给desgiante来处理

root@juju-a949eb-ovn-14:/home/ubuntu# ovn-sbctl list DNS |grep records
records             : {jammy-055444.ucloud.external="192.168.21.218", jammy-071924="192.168.21.210", jammy-071924.ucloud.external="192.168.21.210"}
会在计算节点上生成如下流表, 从192.168.21.210到GW(192.168.21.1)且是dhcp request流量（tp_src=68,tp_dst=67）转到table 28处理, 最终转给desgiante 处理
root@juju-a949eb-ovn-11:~# ovs-ofctl -O OpenFlow13 dump-flows br-int | grep 192.168.21.210 |grep -E '=67|=68'
 cookie=0xc011820d, duration=964.444s, table=27, n_packets=0, n_bytes=0, priority=100,udp,reg14=0x3,metadata=0x3,dl_src=fa:16:3e:92:77:94,nw_src=192.168.21.210,nw_dst=192.168.21.1,tp_src=68,tp_dst=67 actions=controller(userdata=00.00.00.02.00.00.00.00.00.01.de.10.00.00.00.63.c0.a8.15.d2.79.0e.20.a9.fe.a9.fe.c0.a8.15.03.00.c0.a8.15.01.06.04.0a.05.00.0f.0f.15.74.65.73.74.2e.75.63.6c.6f.75.64.2e.69.6e.74.65.72.6e.61.6c.2e.33.04.00.00.a8.c0.1a.02.05.d4.01.04.ff.ff.ff.00.03.04.c0.a8.15.01.36.04.c0.a8.15.01,pause),resubmit(,28)
 cookie=0xc011820d, duration=964.444s, table=27, n_packets=0, n_bytes=0, priority=100,udp,reg14=0x3,metadata=0x3,dl_src=fa:16:3e:92:77:94,nw_src=192.168.21.210,nw_dst=255.255.255.255,tp_src=68,tp_dst=67 actions=controller(userdata=00.00.00.02.00.00.00.00.00.01.de.10.00.00.00.63.c0.a8.15.d2.79.0e.20.a9.fe.a9.fe.c0.a8.15.03.00.c0.a8.15.01.06.04.0a.05.00.0f.0f.15.74.65.73.74.2e.75.63.6c.6f.75.64.2e.69.6e.74.65.72.6e.61.6c.2e.33.04.00.00.a8.c0.1a.02.05.d4.01.04.ff.ff.ff.00.03.04.c0.a8.15.01.36.04.c0.a8.15.01,pause),resubmit(,28)

数据路径是：

$ ssh -i ~/testkey.priv ubuntu@10.5.151.224 -- cat /etc/resolv.conf |tail -n3
nameserver 127.0.0.53
options edns0 trust-ad
search test.ucloud.internal
$ ssh -i ~/testkey.priv ubuntu@10.5.151.224 -- sudo resolvectl status |grep 'Current DNS' -A2
Current DNS Server: 10.5.0.15
      DNS Servers: 10.5.0.15
       DNS Domain: test.ucloud.internal
$ ssh -i ~/testkey.priv ubuntu@10.5.151.224 -- dig -x 192.168.21.210 |grep 192
; <<>> DiG 9.18.1-1ubuntu1.3-Ubuntu <<>> -x 192.168.21.210
;210.21.168.192.in-addr.arpa.   IN      PTR
210.21.168.192.in-addr.arpa. 3361 IN    PTR     jammy-071924.ucloud.external.
$ ssh -i ~/testkey.priv ubuntu@10.5.151.224 -- nslookup 192.168.21.210
210.21.168.192.in-addr.arpa     name = jammy-071924.ucloud.external.

$ openstack recordset list ucloud.external.
+--------------------------------------+-------------------------------+------+---------------------------------------------------------------------+--------+--------+
| id                                   | name                          | type | records                                                             | status | action |
+--------------------------------------+-------------------------------+------+---------------------------------------------------------------------+--------+--------+
| cabe0573-4c27-4cc8-b286-2e7001c5f2f2 | ucloud.external.              | SOA  | ns1.ucloud.external. extdns.example. 1706167177 3530 600 86400 3600 | ACTIVE | NONE   |
| cbc46250-bd07-4ccc-9a26-23d121d1b2e1 | ucloud.external.              | NS   | ns1.ucloud.external.                                                | ACTIVE | NONE   |
| fbb32547-0e37-4948-8a71-2755661e20ea | ns1.ucloud.external.          | A    | 10.5.4.21                                                           | ACTIVE | NONE   |
| 29cc0527-2c15-4eb4-b809-94d84b49aa5a | jammy-071924.ucloud.external. | A    | 192.168.21.210                                                      | ACTIVE | NONE   |
+--------------------------------------+-------------------------------+------+---------------------------------------------------------------------+--------+--------+

20240126

OVS: VM -> dnsmasq(ml2-dns, on gateway node, dnsmasq's upstream server) -> bind9(managed by designate-bind) -> upstream dns
OVN: VM -> ovn-controller(ml2-dns, on compute node, dhcp-options) -> bind9(managed by designate-bind) -> upstream dns

./generate-bundle.sh --name ovn --series jammy --num-compute 1 --ovn --designate --use-stable-charms --run
./tools/vault-unseal-and-authorise.sh
DESIGNATE_BIND_IP=$(juju status designate-bind --format short | awk '/designate-bind/ {print $3}')
juju config neutron-api enable-ml2-dns=true
juju config neutron-api dns-domain=test.ucloud.internal.
juju config neutron-api reverse-dns-lookup=True
juju config neutron-api-plugin-ovn dns-servers=$DESIGNATE_BIND_IP
juju config designate nameservers=ns1.ucloud.external.
juju config neutron-api vni-ranges=1001:2000
source ~/novarc && ./configure
openstack subnet set --dns-nameserver $DESIGNATE_BIND_IP private_subnet
openstack network set --dns-domain ucloud.external. private
openstack zone create --email extdns@example ucloud.external.
openstack recordset create --record $DESIGNATE_BIND_IP --type A ucloud.external. ns1
./tools/instance_launch.sh 1 jammy
./tools/float_all.sh

wget https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-2111.qcow2
wget https://mirrors.163.com/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-DVD-2009.iso
openstack image create --public --container-format=bare --disk-format=qcow2 --file=/home/ubuntu/images/CentOS-7-x86_64-GenericCloud-2111.qcow2 centos7 --property architecture=x86_64
openstack server create --wait --image centos7 --flavor m1.small --key-name testkey --network private --min 1 --max 1 centos7
openstack server add floating ip 0f7ec082-fa64-4094-96aa-c4be18f22220 10.5.150.143
scp -i ~/testkey.priv /home/ubuntu/sos/CentOS-7-x86_64-DVD-2009.iso centos@10.5.150.143:~
ssh -i ~/testkey.priv 10.5.150.143 -l centos

#Inside of CentOS, to install host/nslookup/dig
sudo su
mount /home/centos/CentOS-7-x86_64-DVD-2009.iso /media/CentOS/
cd /etc/yum.repos.d ;mkdir bak; mv *.repo bak;cp -p bak/CentOS-Media.repo .
cat CentOS-Media.repo|grep -v ^#|grep -v ^$
[c7-media]
name=CentOS-$releasever - Media
baseurl=file:///media/CentOS/
file:///media/cdrom/
file:///media/cdrecorder/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
yum clean all
yum makecache
yum update
yum install bind-utils -y

# 创建一个外部dns服务器， 并支持反向记录
sudo apt install bind9 -y
cat <<EOF | sudo tee /etc/bind/named.conf.options
options {
    listen-on port 53 { 0.0.0.0/0; };
    allow-query { any; };
    allow-transfer { any; };
    directory "/var/cache/bind";
    dnssec-validation auto;
    forwarders {
        10.5.0.15;
    };
};
EOF
cat <<EOF | sudo tee /etc/bind/named.conf.local
zone "ucloud.external" IN {
    type master;
    file "/etc/bind/ucloud.external.db";
};
zone "21.168.192.in-addr.arpa" {
    type master;
    file "/etc/bind/ucloud.external.db";
};
EOF
cat <<"EOF" | sudo tee /etc/bind/ucloud.external.db
$TTL 5m
@ IN SOA ns.ucloud.external. email.ucloud.external. 90 4h 15m 8h 4m
@ IN NS ns.ucloud.external.
ns IN A 10.5.0.15
jammy-090702 IN A 192.168.21.4
4 IN PTR jammy-090702.ucloud.external.
EOF
sudo systemctl restart named
sudo resolvectl flush-caches && nslookup jammy-090702.ucloud.external 10.230.65.104
sudo rndc flush && sudo resolvectl flush-caches && dig -x 192.168.21.4 @10.230.65.104

#在ubuntu上的host命令上面必须接dns server，否则ovs-tcpdump抓不到东西
sudo resolvectl flush-caches && host 192.168.21.192 10.5.2.223
sudo resolvectl flush-caches && dig -x 192.168.21.192 @10.5.2.223

#计算节点抓VM的流量, 然后用wireshark GUI查看时，鼠标移动时左侧会出现向左向右的箭头，这代表发出和返回的dns request与dns reply, 这个闭合代表一个包。
sudo ovs-tcpdump -w dns.pcap -i tapxxx port 53

其实问题应该与ovn-controller的截取无关，如果不截取应该是由"sudo ovn-sbctl list DNS"来决定转发，关键是这里面的就用的是internal dns, 而’openstack port show’里的dns_assignment

总结

正常搭建ovn dns环境的命令如下：

./generate-bundle.sh --name ovn --series jammy --num-compute 1 --ovn --designate --use-stable-charms --run
./tools/vault-unseal-and-authorise.sh
juju config neutron-api enable-ml2-dns=true
juju config neutron-api dns-domain=test.ucloud.internal.
juju config neutron-api reverse-dns-lookup=True
DESIGNATE_BIND_IP=$(juju status designate-bind --format short | awk '/designate-bind/ {print $3}')
#juju config neutron-api-plugin-ovn dns-servers=$DESIGNATE_BIND_IP
juju config designate nameservers=ns1.ucloud.external.
juju config neutron-api vni-ranges=1001:2000
source ~/novarc && ./configure
openstack subnet set --dns-nameserver $DESIGNATE_BIND_IP private_subnet
openstack network set --dns-domain ucloud.external. private
openstack zone create --email extdns@example ucloud.external.
openstack recordset create --record $DESIGNATE_BIND_IP --type A ucloud.external. ns1
./tools/instance_launch.sh 1 jammy
./tools/float_all.sh

这里面有ml2-dns与designate两处，在每个计算节点上的ovn-controller根据是否有edns0的additional records （https://github.com/ovn-org/ovn/commit/4b10571aa89b226c13a8c5551ceb7208d782b580）来决定是否将udp 53过来的dns request转发给designate-bind/external-dns. 'dig -x’会带有edns0，而host/nslookup可能不带有ends0.
那些没有被转给designate-bind/external-dns的流量需ovn-controller本地处理，而本地是根据’sudo ovn-sbctl list dns |grep records’里的流表来处理的。
正常情况下，如上面的配置已经配置了network里的dns-name (openstack network set --dns-domain ucloud.external. private), 若network里不配置dns-name，那就是用neutron配置文件里默认的internal 的dns（juju config neutron-api dns-domain=test.ucloud.internal.）, 那就会出现下列一种情况，此时，在同一个子网内，用’dig -x’就会返回external dns, 若用host/nslookup则可能返回internal dns, 若跨子网则不会出现结果不一致的情况。

$ openstack port show 5ba0c744-d402-4c2c-854f-635d1dfd57d1 |grep dns
| dns_assignment          | fqdn='cirros2-081030.test.ucloud.internal.', hostname='cirros2-081030', ip_address='192.168.21.72' |
| dns_domain              |                                                                                                    |
| dns_name                | cirros2-081030                                                                                     |

ubuntu@juju-53cf31-ovn-12:~$ sudo ovn-sbctl list dns |grep records
records             : {"4.21.168.192.in-addr.arpa"=jammy-090702.ucloud.external, "72.21.168.192.in-addr.arpa"=cirros2-081030.test.ucloud.internal, cirros2-081030="192.168.21.72", cirros2-081030.test.ucloud.internal="192.168.21.72", jammy-090702="192.168.21.4", jammy-090702.ucloud.external="192.168.21.4"}

类似的情况还有下面subnet里的dns-servers, 若不在subnet里配置它就会用neutron.conf里默认的（juju config neutron-api-plugin-ovn dns-servers）。正常情况下，这里都应该指定$DESIGNATE_BIND_IP, 当然，你不想用designate-bind用直接用external dns, 那这里就直接填external dns IP.

juju config neutron-api-plugin-ovn dns-servers=$DESIGNATE_BIND_IP
openstack subnet set --dns-nameserver $DESIGNATE_BIND_IP private_subnet

Reference

[1] https://github.com/openstack/neutron/blob/stable/pike/neutron/plugins/ml2/extensions/dns_integration.py#L285
[2] https://docs.openstack.org/mitaka/networking-guide/config-dns-int.html
[3] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/8/html-single/dns-as-a-service_guide/index
[4] https://openstackdevops.wordpress.com/2018/01/27/designate-and-neutron-dns-integration/

quqi99

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
打赏
0
评论
为租户下的虚机提供IPv6 DNS服务(by quqi99)

版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明 (http://blog.csdn.net/quqi99)问题当虚机运行下列代码时，我们需要考虑为tenant下的VM提供DNS服务:import dnsimport dns.resolveranswers = dns.resolver.query('node1', 'AAAA')print answers............
复制链接

扫一扫