Openshift v3.11 DNS配置及理解

前言

OpenShift在安装的时候依赖于NetworkManager服务完成dnsmasq自动配置，请确保该服务保护启动状态，并保证在网卡配置文件中不要添加 NM_CONTROLLED=no参数,否则会导致一系列网络问题。如果一定需要开启此参数，请做好心里准备，你要知道整个dns访问配置流程，需要手动维护下列配置文件：
/etc/resolv.conf
/etc/dnsmasq.d/origin-dns.conf
/etc/dnsmasq.d/origin-upstream-dns.conf
/etc/origin/node/resolv.conf

三种DNS访问类型

1. Pod访问外网
2. 集群内部服务发现
3. 外部服务访问集群内部

Pod访问外网dns查询流程：pod – dnsmasq – 上游dns(多个地址倒序查询)
Pod访问集群内服务dns查询流程：pod – dnsmasq – skydns（127.0.0.1:53) – 如果没有本地缓存 – openshift api
SkyDns：用于解析service域名 ，调用 OpenShift API 服务来获取主机名、IP地址等信息，然后封装成标准 DNS 记录并返回给查询客户端。

在node节点查看dns信息，可以看到dnsmasq启了3个进程，openshift启动了1个进程为skydns
#netstat -atunlp|grep 53
tcp 0 0 10.10.6.136:53 0.0.0.0:* LISTEN 22595/dnsmasq
tcp 0 0 172.17.0.1:53 0.0.0.0:* LISTEN 22595/dnsmasq
tcp 0 0 10.131.0.1:53 0.0.0.0:* LISTEN 22595/dnsmasq
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 8542/openshift

其中通过dnsmasq启动的53端口监听的网络与route -n查看到的网段对应
10.10.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
10.128.0.0 0.0.0.0 255.252.0.0 U 0 0 0 tun0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.30.0.0 0.0.0.0 255.255.0.0 U 0 0 0 tun0
在node节点查看skydns信息，通过openshift进程启动。

#netstat -atunlp|grep 53
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 8542/openshift
node节点上的skydns从缓存或API获取数据返回客户端查询
所有节点都安装SkyDns，组成一个分布式集群，其中master节点额外启动8053端口

所有DNS配置文件详细信息

Pod DNS:
#cat /etc/resolv.conf
nameserver 10.10.6.136
search nginx-10-10-163-88.svc.cluster.local svc.cluster.local cluster.local localdomain
options ndots:5
注解：
nameserver是宿主机ip地址，所有dns请求会发送到宿主机53端口

宿主机DNS：
#cat /etc/resolv.conf
#nameserver updated by /etc/NetworkManager/dispatcher.d/99-origin-dns.sh
#Generated by NetworkManager
search cluster.local
nameserver 10.10.6.136

安装启动NetworkManager会触发/etc/NetworkManager/dispatcher.d/99-origin-dns.sh脚本，修改 /etc/resolv.conf、设置搜索域，以及将宿主机的默认 IP 作为 nameserver、创建 /etc/origin/node/resolv.conf、生成/etc/dnsmasq.d/origin-dns.conf,/etc/dnsmasq.d/origin-upstream-dns.conf两个配置文件、启动dnsmasq服务，其中origin-upstream-dns.conf 定义了上游DNS服务器，该地址通过DHCP获取，我这里没有生成成功，手动配置并重启dnsmasq

#cat origin-dns.conf
no-resolv
domain-needed
no-negcache
max-cache-ttl=1
enable-dbus
dns-forward-max=10000
cache-size=10000
bind-dynamic
min-port=1024
except-interface=lo
#End of config

#cat origin-upstream-dns.conf
server=10.10.255.1
server=10.10.255.2
server=10.10.153.132

#systemctl restart dnsmasq
dnsmasq查询顺序是自下而上,即先查询10.10.153.132

dnsmasq日志开启日志调试及原理验证

vi /etc/dnsmasq.conf
#For debugging purposes, log each DNS query as it passes through
#dnsmasq.
log-queries

#systemctl restart dnsmasq
dnsmasq查看日志
#journalctl -f -u dnsmasq
Jan 14 11:10:51 openshift2 dnsmasq[14678]: setting upstream servers from DBus
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 10.10.255.2#53
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 10.10.255.1#53
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 10.10.153.132#53
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 127.0.0.1#53 for domain cluster.local
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[AAAA] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: forwarded hawkular-metrics.openshift-infra.svc.cluster.local to 127.0.0.1
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[A] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: forwarded hawkular-metrics.openshift-infra.svc.cluster.local to 127.0.0.1
Jan 14 11:11:05 openshift2 dnsmasq[14678]: reply hawkular-metrics.openshift-infra.svc.cluster.local is 172.30.87.204
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[AAAA] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: forwarded hawkular-metrics.openshift-infra.svc.cluster.local to 127.0.0.1
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[A] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: cached hawkular-metrics.openshift-infra.svc.cluster.local is 172.30.87.204
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[AAAA] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: forwarded hawkular-metrics.openshift-infra.svc.cluster.local to 127.0.0.1
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[A] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4

根据日志得知，dnsmasq把请求都转发到127.0.0.1:53上的skydns中（封装在openshift进程）node节点上的skydns从缓存或API获取数据返回客户端查询，可以看到示例中是走的cached

openshift中dns配置信息：
master节点
vim /etc/origin/master/master-config.yaml
dnsConfig:
bindAddress: 0.0.0.0:8053
bindNetwork: tcp4

node节点
vim /etc/origin/node/node-config.yaml
dnsBindAddress: 127.0.0.1:53
dnsDomain: cluster.local
dnsIP: 0.0.0.0
dnsNameservers: null
dnsRecursiveResolvConf: /etc/origin/node/resolv.conf

无论是在宿主机还是在pod内，均可以访问service，因为首先都要经过宿主机dns服务器127.0.0.1:53
#ping hawkular-metrics.openshift-infra.svc.cluster.local
PING hawkular-metrics.openshift-infra.svc.cluster.local (172.30.87.204) 56(84) bytes of data.

参考文档:
http://www.cnblogs.com/sammyliu/p/10056035.html
https://www.cnblogs.com/ericnie/p/10216775.html
https://blog.cloudtechgroup.cn/Blog/2018/07/23/ocp-2018-07-23/