云主机中的docker DNS问题导致的容器重启问题

大囚长

已于 2022-08-18 16:09:23 修改

阅读量612

点赞数

分类专栏：操作系统 Linux 运维人生文章标签： docker 容器运维

于 2022-08-17 19:26:20 首次发布

本文链接：https://blog.csdn.net/Jailman/article/details/126392584

版权

运维人生同时被 3 个专栏收录

266 篇文章 17 订阅

订阅专栏

Linux

74 篇文章 6 订阅

订阅专栏

操作系统

35 篇文章 4 订阅

订阅专栏

以下情况在你iptables被冲掉的情况下也可能发生，不一定是DNS问题
docker-compose在aws云ubuntu20部署nakama和cockroachdb服务

version: '3'
services:
  cockroachdb:
    image: cockroachdb/cockroach:v19.2.5
    command: start --insecure --store=attrs=ssd,path=/var/lib/cockroach/
    restart: always
    volumes:
      - data:/var/lib/cockroach
    expose:
      - "8080"
      - "26257"
    ports:
      - "26257:26257"
      - "8080:8080"
  nakama:
    image: heroiclabs/nakama:2.15.0
    entrypoint:
      - "/bin/sh"
      - "-ecx"
      - >
          /nakama/nakama migrate up --database.address root@cockroachdb:26257 &&
          exec /nakama/nakama --name nakama1 --database.address root@cockroachdb:26257
    restart: always
    links:
      - "cockroachdb:db"
    depends_on:
      - cockroachdb
    volumes:
      - ./nakama/data:/nakama/data
    expose:
      - "7349"
      - "7350"
      - "7351"
    ports:
      - "7349:7349"
      - "7350:7350"
      - "7351:7351"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:7350/"]
      interval: 10s
      timeout: 5s
      retries: 5
volumes:
  data: {}

容器应用不停重启

# docker ps
CONTAINER ID   IMAGE                            COMMAND                  CREATED         STATUS                                  PORTS                                                                                      NAMES
c52f44c45710   heroiclabs/nakama:2.15.0         "/bin/sh -ecx '/naka…"   4 seconds ago   Restarting (1) Less than a second ago                                                                                              fish-docker_nakama_1
af9aea602658   cockroachdb/cockroach:v21.2.14   "/cockroach/cockroac…"   4 seconds ago   Up Less than a second                   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:26257->26257/tcp, :::26257->26257/tcp   fish-docker_cockroachdb_1

查看网络也没有正常运行

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:b3:c8:df:43:5e brd ff:ff:ff:ff:ff:ff
    inet 172.31.2.101/20 brd 172.31.15.255 scope global dynamic ens5
       valid_lft 3458sec preferred_lft 3458sec
    inet6 fe80::8b3:c8ff:fedf:435e/64 scope link 
       valid_lft forever preferred_lft forever
8: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:65:7d:11:d0 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
9: br-ee6c48b23d19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:81:23:4c:1a brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-ee6c48b23d19
       valid_lft forever preferred_lft forever
    inet6 fe80::42:81ff:fe23:4c1a/64 scope link 
       valid_lft forever preferred_lft forever
17: veth7bd9b7a@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ee6c48b23d19 state UP group default 
    link/ether 3a:9a:0b:d8:53:58 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::389a:bff:fed8:5358/64 scope link 
       valid_lft forever preferred_lft forever

查看日志

# docker logs c52f44c45710
{"level":"info","ts":"2022-08-17T11:01:42.734Z","caller":"migrate/migrate.go:139","msg":"Database connection","dsn":"postgresql://root@cockroachdb:26257?sslmode=disable"}
{"level":"fatal","ts":"2022-08-17T11:01:42.738Z","caller":"migrate/migrate.go:147","msg":"Error pinging database","error":"dial tcp: lookup cockroachdb: Temporary failure in name resolution"}
+ /nakama/nakama migrate up --database.address root@cockroachdb:26257
{"level":"info","ts":"2022-08-17T11:01:49.585Z","caller":"migrate/migrate.go:139","msg":"Database connection","dsn":"postgresql://root@cockroachdb:26257?sslmode=disable"}
{"level":"fatal","ts":"2022-08-17T11:01:49.587Z","caller":"migrate/migrate.go:147","msg":"Error pinging database","error":"dial tcp: lookup cockroachdb: Temporary failure in name resolution"}

显然是DNS问题
修改配置文件重启

# vim /etc/docker/daemon.json
{
	"dns": ["8.8.8.8","114.114.114.114"]
}
# systemctl restart docker

无效，继续解决DNS问题，修改resolv.conf配置文件

# vim /etc/resolv.conf
nameserver 8.8.8.8

删除重启docker-compose

docker-compose stop
docker-compose rm
docker rmi `docker images -q`
docker-compose up -d

再次查看，正常

# docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS                    PORTS                                                                                      NAMES
69e84460ceee   heroiclabs/nakama:2.15.0        "/bin/sh -ecx '/naka…"   43 seconds ago   Up 42 seconds (healthy)   0.0.0.0:7349-7351->7349-7351/tcp, :::7349-7351->7349-7351/tcp                              fish-docker_nakama_1
f55f3d459ccc   cockroachdb/cockroach:v19.2.5   "/cockroach/cockroac…"   44 seconds ago   Up 43 seconds             0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:26257->26257/tcp, :::26257->26257/tcp   fish-docker_cockroachdb_1

云主机的resolv.conf文件会被定期更新，配置会被冲掉

查看状态

# resolvectl status
Global
       LLMNR setting: no                  
MulticastDNS setting: no                  
  DNSOverTLS setting: no                  
      DNSSEC setting: no                  
    DNSSEC supported: no                  
          DNSSEC NTA: 10.in-addr.arpa     
                      16.172.in-addr.arpa 
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa 
                      18.172.in-addr.arpa 
                      19.172.in-addr.arpa 
                      20.172.in-addr.arpa 
                      21.172.in-addr.arpa 
                      22.172.in-addr.arpa 
                      23.172.in-addr.arpa 
                      24.172.in-addr.arpa 
                      25.172.in-addr.arpa 
                      26.172.in-addr.arpa 
                      27.172.in-addr.arpa 
                      28.172.in-addr.arpa 
                      29.172.in-addr.arpa 
                      30.172.in-addr.arpa 
                      31.172.in-addr.arpa 
                      corp                
                      d.f.ip6.arpa        
                      home                
                      internal            
                      intranet            
                      lan                 
                      local               
                      private             
                      test                

Link 9 (br-ee6c48b23d19)
      Current Scopes: none
DefaultRoute setting: no  
       LLMNR setting: yes 
MulticastDNS setting: no  
  DNSOverTLS setting: no  
      DNSSEC setting: no  
    DNSSEC supported: no  

Link 8 (docker0)
      Current Scopes: none
DefaultRoute setting: no  
       LLMNR setting: yes 
MulticastDNS setting: no  
  DNSOverTLS setting: no  
      DNSSEC setting: no  
    DNSSEC supported: no  

Link 259 (vethcea37b2)
      Current Scopes: none
DefaultRoute setting: no  
       LLMNR setting: yes 
MulticastDNS setting: no  
  DNSOverTLS setting: no  
      DNSSEC setting: no  
    DNSSEC supported: no  

Link 2 (ens5)
      Current Scopes: DNS                       
DefaultRoute setting: yes                       
       LLMNR setting: yes                       
MulticastDNS setting: no                        
  DNSOverTLS setting: no                        
      DNSSEC setting: no                        
    DNSSEC supported: no                        
  Current DNS Server: 172.31.0.2                
         DNS Servers: 172.31.0.2                
          DNS Domain: ap-east-1.compute.internal

Link 257 (veth02d1f2b)
      Current Scopes: none
DefaultRoute setting: no  
       LLMNR setting: yes 
MulticastDNS setting: no  
  DNSOverTLS setting: no  
      DNSSEC setting: no  
    DNSSEC supported: no

查看resolv.conf文件

# ll /etc/resolv.conf 
lrwxrwxrwx 1 root root 39 Nov 30  2021 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

修改systemd-resolvd配置

# vim /etc/systemd/resolved.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details

[Resolve]
DNS=8.8.8.8 114.114.114.114
#DNS=
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#DNSOverTLS=no
#Cache=no-negative
#DNSStubListener=yes
#ReadEtcHosts=yes

重启

systemctl restart systemd-resolved

或者修改resolv.conf文件

# mv /etc/resolv.conf /etc/resolv.conf.bak
# ln -s /run/systemd/resolve/resolv.conf /etc/
# vim /run/systemd/resolve/resolv.conf
nameserver 8.8.8.8
nameserver 114.114.114.114
nameserver 172.31.0.2
search ap-east-1.compute.internal

大囚长

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
云主机中的docker DNS问题导致的容器重启问题

docker-compose在aws云ubuntu20部署nakama和cockroachdb服务。无效，继续解决DNS问题，修改resolv.conf配置文件。云主机的resolv.conf文件会被定期更新，配置会被冲掉。删除重启docker-compose。查看网络也没有正常运行。.........
复制链接

扫一扫

专栏目录