基于Consul实现Docker Vxlan网络浅析

Consul实现Docker Vxlan

​ 首先,我们将构建一个Overlay网络在三个Docker hosts之间,二个节点运行Docker,另一个节点运行Consul,Docker将会使用Consul来存储Overlay网络metadadata
在这里插入图片描述

启动Consul服务

Consul节点启动服务

# 启动Consul服务
[root@consul ~]# nohup consul agent -server -dev -ui -client 0.0.0.0 &
[1] 14532
[root@consul ~]# nohup: ignoring input and appending output to ‘nohup.out’

# agent: 启动一个agent
# dev: 创建一个开发者模式的consul
# server: 以server模式启动consul
# ui: 提供一个可视化的UI界面
# client: 设置bind地址

登录Consul

在这里插入图片描述

启动自定义dockerd进程

  • --cluster-store=参数指向docker daemon所使用key value service的地址(本例中即consul的服务地址)
  • --cluster-advertise= 参数决定了所使用网卡以及docker daemon端口信息
  • -H参数分别指定了docker demon服务的地址和协议
# 关闭Docker
	systemctl disable docker --now

# 二个节点启动自定义Docker
[root@docker1 ~]# nohup dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --cluster-store consul://192.168.0.13:8500 --cluster-advertise ens33:2376 & 
[1] 15048
[root@docker1 ~]# nohup: ignoring input and appending output to ‘nohup.out’

[root@docker2 ~]# nohup dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --cluster-store consul://192.168.0.13:8500 --cluster-advertise ens33:2376 & 
[1] 2620
[root@docker2 ~]# nohup: ignoring input and appending output to ‘nohup.out’

待二个dockerd进程启动后,可以发现consul已经拿到相关数据了

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-zc1o5pVB-1655559888399)(image/image-20220509131523544.png)]

流量转发图

由于后续分析十分枯燥,这里放上流量转发图,供大家参考
在这里插入图片描述

创建Overlay网络

  • 在Docker1创建一个Overlay网络,通过下面可见Docker1的网络信息通过Consul传递给Docker2了
# 创建一个Overlay网络
[root@docker1 ~]# docker network create --driver overlay --subnet 10.70.2.0/24 demonset
743aa849b76afd5a33298083cd4c54548207ddd4cb95ae276c00009936bdadd8

# 查看Docker1网络
[root@docker1 ~]# docker network ls
NETWORK ID     NAME              DRIVER    SCOPE
59bdb4c8ea2b   bridge            bridge    local
1be3297604d1   demonset          overlay   global
254821e742a1   docker_gwbridge   bridge    local
969166ca1bf5   host              host      local
e7736b1206fe   none              null      local



# 查看Docker2网络
[root@docker2 ~]# docker network ls
NETWORK ID     NAME              DRIVER    SCOPE
186f412ffc45   bridge            bridge    local
1be3297604d1   demonset          overlay   global
4afd26fa8e1b   docker_gwbridge   bridge    local
e0e57d329138   host              host      local
02ebc9feeb3b   none              null      local

启动二个容器验证网络

  • 可见通过Consul共享二个Docker之间的数据后,这二个容器实现了夸主机ping,且是在他们自己的一个网段10.70.2.0/24内通信
  • 此时流量转发大致如下图所示
image-20220509133625036
# Docker1启动一个容器
[root@docker1 ~]# docker run -d --ip 10.70.2.111 --net demonset --name busybox1 busybox:1.28 sleep 100000
a00e2a1828c5147091f52e1cbc5ff87d9148369b5ecb202c5209244774fd3e43


# Docker2启动一个容器
[root@docker2 ~]# docker run -d --ip 10.70.2.112 --net demonset --name busybox2 busybox:1.28 sleep 100000
1c6902b826a7da0d22c59b877294d72d696b02df96fccff0ec78a0de08a06f37

# Docker2 ping Docker1测试
[root@docker2 ~]# docker exec busybox2 ping -c 2 10.70.2.111
PING 10.70.2.111 (10.70.2.111): 56 data bytes
64 bytes from 10.70.2.111: seq=0 ttl=64 time=0.548 ms
64 bytes from 10.70.2.111: seq=1 ttl=64 time=0.256 ms

--- 10.70.2.111 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.256/0.402/0.548 ms

网络容器网络浅析

由于实验做了很久,当时没有完整记录分析过程,这里只记录了一部分,只提供给大家一个分析思路

docker1节点

# 查看busybox1网卡信息
[root@docker1 ~]# docker exec busybox1 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
7: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue 
    link/ether 02:42:0a:46:02:6f brd ff:ff:ff:ff:ff:ff
    inet 10.70.2.111/24 brd 10.70.2.255 scope global eth0
       valid_lft forever preferred_lft forever
9: eth1@if10: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1
       valid_lft forever preferred_lft forever


# 查看busybox1路由信息
[root@docker1 ~]# docker exec busybox1 ip route show
default via 172.18.0.1 dev eth1 
10.70.2.0/24 dev eth0 scope link  src 10.70.2.111 
172.18.0.0/16 dev eth1 scope link  src 172.18.0.2 


# 查看docker1宿主机网络信息
[root@docker1 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:4c:30:4e brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.11/24 brd 192.168.0.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::624c:c1db:e3b4:9165/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::2e8f:238e:c202:4027/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:22:fd:10:94 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global docker_gwbridge
       valid_lft forever preferred_lft forever
    inet6 fe80::42:22ff:fefd:1094/64 scope link 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:e6:92:c4:d8 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
10: veth0475f3f@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default 
    link/ether 2a:8e:ee:f7:af:f3 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::288e:eeff:fef7:aff3/64 scope link 
       valid_lft forever preferred_lft forever

# 查看docker1 路由信息
[root@docker1 ~]# ip route show
default via 192.168.0.2 dev ens33 proto static metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.1 
192.168.0.0/24 dev ens33 proto kernel scope link src 192.168.0.11 metric 100 


# 确定docker_gwbridge 下行设备
[root@docker1 ~]# brctl show
bridge name	bridge id		STP enabled	interfaces
docker0		8000.0242e692c4d8	no		
docker_gwbridge		8000.024222fd1094	no		veth0475f3f

# 查看docker_gwbridge设备信息
[root@docker1 ~]# docker inspect docker_gwbridge
[
    {
        "Name": "docker_gwbridge",
        ..................
        "Driver": "bridge",
        "IPAM": {
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Options": {
            "com.docker.network.bridge.enable_icc": "false",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.name": "docker_gwbridge"
        },
    }
]
# 此网络使用驱动桥(与标准 docker 桥 docker0 使用的相同)
# 它使用子网 172.18.0.0/16,与 eth1 一致
# enable_icc 设置为 false,这意味着我们不能使用这个桥接器进行容器间通信
# enable_ip_masquerade 设置为 true,这意味着来自容器的流量将被 NAT 以访问外部网络(我们之前在成功 ping 8.8.8.8 时看到过)

docker2节点

# 查看busybox2网卡信息
[root@docker2 ~]# docker exec busybox2 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
7: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue 
    link/ether 02:42:0a:46:02:70 brd ff:ff:ff:ff:ff:ff
    inet 10.70.2.112/24 brd 10.70.2.255 scope global eth0
       valid_lft forever preferred_lft forever
9: eth1@if10: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1
       valid_lft forever preferred_lft forever


# 查看busybox2路由信息
[root@docker2 ~]# docker exec busybox2 ip route show
default via 172.18.0.1 dev eth1 
10.70.2.0/24 dev eth0 scope link  src 10.70.2.112 
172.18.0.0/16 dev eth1 scope link  src 172.18.0.2

# 查看docker2宿主机网络信息
[root@docker2 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:19:a1:51 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.12/24 brd 192.168.0.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::624c:c1db:e3b4:9165/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::2e8f:238e:c202:4027/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::919c:e504:f9af:83db/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:1b:fe:6a:6d brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:97:2e:c3:e1 brd ff:ff:ff:ff:ff:ff
10: veth1b1e9f6@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default 
    link/ether 4e:5b:1e:f6:37:62 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::4c5b:1eff:fef6:3762/64 scope link 
       valid_lft forever preferred_lft forever



# 查看docker2路由信息
[root@docker2 ~]# ip route show
default via 192.168.0.2 dev ens33 proto static metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.1 
192.168.0.0/24 dev ens33 proto kernel scope link src 192.168.0.12 metric 100 


# 确定docker_gwbridge 下行设备
[root@docker2 ~]# brctl show
bridge name	bridge id		STP enabled	interfaces
docker0		8000.02421bfe6a6d	no		
docker_gwbridge		8000.0242972ec3e1	no		veth1b1e9f6

查找eth0@if7

  • 基于上面的分析我们可以发现一个问题:veth设备都是成对出现,但是busybox容器内出现了一个eth0@if7,我们在宿主机内找不到index=7的网卡,那么这张网卡必定存在其他的命名空间内
  • 基于下面信息,我们可以得到一个信息:docker1,docker2之间共享了一个network namespace 1-c8669d70d4,br0设备下面连接了二个设备:vxlan0@if5和veth0@if6
# 查看docker1 容器使用了哪些命令空间
[root@docker1 ~]# ls /var/run/docker/netns
1-743aa849b7  9c002f20bb33

[root@docker1 ~]# nsenter --net=/var/run/docker/netns/1-743aa849b7 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 1e:08:06:d4:40:6b brd ff:ff:ff:ff:ff:ff
    inet 10.70.2.1/24 brd 10.70.2.255 scope global br0
       valid_lft forever preferred_lft forever
5: vxlan0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br0 state UNKNOWN group default 
    link/ether 26:de:18:1e:16:c2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: veth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br0 state UP group default 
    link/ether 1e:08:06:d4:40:6b brd ff:ff:ff:ff:ff:ff link-netnsid 1

# 查看路由信息
[root@docker1 ~]# nsenter --net=/var/run/docker/netns/1-c8669d70d4 ip route show
10.70.2.0/24 dev br0 proto kernel scope link src 10.70.2.1

# 查看br0设备连接情况
[root@docker1 ~]# nsenter --net=/var/run/docker/netns/1-c8669d70d4 brctl show br0
bridge name	bridge id		STP enabled	interfaces
br0		8000.1e0806d4406b	no		vxlan0
									veth0

[root@docker1 ~]# nsenter --net=/var/run/docker/netns/3fe21b67885e ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
6: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 02:42:0a:46:02:6f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.70.2.111/24 brd 10.70.2.255 scope global eth0
       valid_lft forever preferred_lft forever
9: eth1@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1
       valid_lft forever preferred_lft forever
# 查看docker2 容器使用了哪些命令空间
[root@docker2 ~]# ls /var/run/docker/netns
1-c8669d70d4  349cb1aac200

[root@docker2 ~]# nsenter --net=/var/run/docker/netns/1-c8669d70d4 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 0e:ad:33:09:ad:c6 brd ff:ff:ff:ff:ff:ff
    inet 10.70.2.1/24 brd 10.70.2.255 scope global br0
       valid_lft forever preferred_lft forever
5: vxlan0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br0 state UNKNOWN group default 
    link/ether 0e:ad:33:09:ad:c6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: veth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br0 state UP group default 
    link/ether 82:4d:95:fe:51:6b brd ff:ff:ff:ff:ff:ff link-netnsid 1

# 查看路由信息
[root@docker2 ~]# nsenter --net=/var/run/docker/netns/1-c8669d70d4 ip route show
10.70.2.0/24 dev br0 proto kernel scope link src 10.70.2.1

# 查看br0设备连接情况
[root@docker2 ~]# nsenter --net=/var/run/docker/netns/1-c8669d70d4 brctl show br0
bridge name	bridge id		STP enabled	interfaces
br0		8000.0ead3309adc6	no		vxlan0
									veth0
									
[root@docker2 ~]# nsenter --net=/var/run/docker/netns/349cb1aac200 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
6: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 02:42:0a:46:02:70 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.70.2.112/24 brd 10.70.2.255 scope global eth0
       valid_lft forever preferred_lft forever
9: eth1@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1
       valid_lft forever preferred_lft forever

参考链接:https://cizixs.com/2017/09/28/linux-vxlan/
参考链接:https://chengqian90.com/Linux%E5%86%85%E6%A0%B8/Linux%E5%AF%B9vxlan%E7%9A%84%E6%94%AF%E6%8C%81.html
参考链接:https://blog.revolve.team/2017/08/20/deep-dive-3-into-docker-overlay-networks-part-3/

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

旺仔_牛奶

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值