【运维】K8S集群部署系列之普通Node节点搭建（七）

最新推荐文章于 2023-12-12 23:27:21 发布
最新推荐文章于 2023-12-12 23:27:21 发布
阅读量1.3k 收藏
点赞数
分类专栏： k8s 文章标签： docker flannel k8s Node kubernetes docker网络配置
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_39974140/article/details/100033814
版权

K8S 普通Node节点之Docker环境搭建

文章目录

系统环境

NodeIPCentOSkernelcpumemory
node1192.168.159.4CentOS Linux release 7.4.1708 (Core)3.10.0-693.el7.x86_64Intel® Core™ i5-7500 CPU @ 3.40GHz * 12G
node2192.168.159.5CentOS Linux release 7.4.1708 (Core)3.10.0-693.el7.x86_64Intel® Core™ i5-7500 CPU @ 3.40GHz * 12G

Node 软件环境

NodeIPdockerflannel
node1192.168.159.419.03.10.11.0
node2192.168.159.519.03.10.11.0

Docker 安装

官方文档

Docker配置文件详解

docker 下载

官方下载地址

wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.1.tgz
tar -zxvf  docker-19.03.1.tgz
cp -f docker/* /usr/local/bin/

docker 配置文件

cat > /opt/k8s/node/etc/docker.conf << "EOF"
DOCKER_NETWORK_OPTIONS="-H unix:///var/run/docker.sock \
-H 0.0.0.0:2375"
"EOF"

docker 服务文件

cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Engine Service
Documentation=https://docs.docker.com
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin"
EnvironmentFile=-/opt/k8s/node/etc/docker.conf
ExecStart=/usr/local/bin/dockerd $DOCKER_NETWORK_OPTIONS 
ExecReload=/bin/kill -s HUP $MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target
EOF

docker 服务启动

启动
systemctl daemon-reload && systemctl start docker
查看

查看docker版本信息

[root@node1 k8s]# docker version
Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        74b1e89e8a
 Built:             Thu Jul 25 21:17:37 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89e8a
  Built:            Thu Jul 25 21:27:55 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

查看本地镜像

[root@node1 docker]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

查看容器

[root@node1 docker]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
验证
  • 从仓库搜索镜像
        [root@node1 pki]# docker search centos
   NAME                               DESCRIPTION                                     STARS               OFFICIAL            AUTOMATED
   centos                             The official build of CentOS.                   5524                [OK]                
   ansible/centos7-ansible            Ansible on Centos7                              122                                     [OK]
   jdeathe/centos-ssh                 CentOS-6 6.10 x86_64 / CentOS-7 7.6.1810 x86…   111                                     [OK]
   consol/centos-xfce-vnc             Centos container with "headless" VNC session…   99                                      [OK]
   centos/mysql-57-centos7            MySQL 5.7 SQL database server                   62                                      
   imagine10255/centos6-lnmp-php56    centos6-lnmp-php56                              57                                      [OK]
   tutum/centos                       Simple CentOS docker image with SSH access      44                                      
   centos/postgresql-96-centos7       PostgreSQL is an advanced Object-Relational …   39                                      
   kinogmt/centos-ssh                 CentOS with SSH                                 29                                      [OK]
   pivotaldata/centos-gpdb-dev        CentOS image for GPDB development. Tag names…   10                                      
   guyton/centos6                     From official centos6 container with full up…   9                                       [OK]
   drecom/centos-ruby                 centos ruby                                     6                                       [OK]
   pivotaldata/centos                 Base centos, freshened up a little with a Do…   3                                       
   mamohr/centos-java                 Oracle Java 8 Docker image based on Centos 7    3                                       [OK]
   darksheer/centos                   Base Centos Image -- Updated hourly             3                                       [OK]
   pivotaldata/centos-gcc-toolchain   CentOS with a toolchain, but unaffiliated wi…   2                                       
   pivotaldata/centos-mingw           Using the mingw toolchain to cross-compile t…   2                                       
   ovirtguestagent/centos7-atomic     The oVirt Guest Agent for Centos 7 Atomic Ho…   2                                       
   miko2u/centos6                     CentOS6 日本語環境                                   2                                       [OK]
   mcnaughton/centos-base             centos base image                               1                                       [OK]
   indigo/centos-maven                Vanilla CentOS 7 with Oracle Java Developmen…   1                                       [OK]
   blacklabelops/centos               CentOS Base Image! Built and Updates Daily!     1                                       [OK]
   smartentry/centos                  centos with smartentry                          0                                       [OK]
   pivotaldata/centos7-dev            CentosOS 7 image for GPDB development           0                                       
   pivotaldata/centos6.8-dev          CentosOS 6.8 image for GPDB development         0
  • 拉取centos镜像
    [root@node1 pki]# docker pull centos
Using default tag: latest
latest: Pulling from library/centos
d8d02d457314: Pull complete 
Digest: sha256:307835c385f656ec2e2fec602cf093224173c51119bbebd602c53c3653a3d6eb
Status: Downloaded newer image for centos:latest
docker.io/library/centos:latest

    [root@node1 pki]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
centos              latest              67fa590cfc1c        2 days ago          202MB

  • 创建test容器

    [root@node1 pki]# docker run -i -t --name test centos /bin/bash
WARNING: IPv4 forwarding is disabled. Networking will not work.
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"write /proc/self/attr/keycreate: permission denied\"": unknown.

    [root@node1 pki]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
5ba3f8995eff        centos              "/bin/bash"         7 minutes ago       Created

  • 问题解决1
    WARNING: IPv4 forwarding is disabled. Networking will not work.
    方案1、开启IPv4转发，允许容器访问外部网络

    sed -i '$a\net.ipv4.ip_forward=1' /usr/lib/sysctl.d/00-system.conf
systemctl daemon-reload && systemctl restart network

    方案二、在配置DOCKER_NETWORK_OPTIONS中添加启动参数--ip-forward=true，则重启docker服务时就会自动设定系统的 ip_forward 参数为 1

  • 问题解决2

    docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"write /proc/self/attr/keycreate: permission denied\"": unknown.

    查看SELINUX状态

    [root@node1 pki]# getenforce
Enforcing

    临时关闭SELINUX

    setenforce 0

    查看SELINUX状态

    [root@node1 pki]# getenforce
Permissive

    启动容器

    [root@node1 pki]# docker start test
test

    [root@node1 pki]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
5ba3f8995eff        centos              "/bin/bash"         14 minutes ago      Up 14 seconds

    永久关闭SELINUX

    sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
reboot
systemctl start docker
docker start test

    查看SELINUX状态

    [root@node1 pki]# getenforce
Disabled
Docker常用命令介绍

  • 进入test容器命令行

    docker attach test

  • 退出容器
    退出并关闭容器
    exit 或者 Ctrl+C && Ctrl + D
    退出命令行
    Ctrl + P && Ctrl +Q

  • 停止容器
    docker stop test

  • 删除test容器
    docker rm test

  • 保存镜像到本地
    docker save -o centos.tar centos:latest

  • 删除镜像
    docker rmi centos:latest

  • 载入本地镜像
    docker load --input centos.tar

Docker跨主机集群网络配置

Node节点网卡查看

查看node1网桥，新增docker0

[root@node1 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:c5ff:fed3:c41d  prefixlen 64  scopeid 0x20<link>
        ether 02:42:c5:d3:c4:1d  txqueuelen 0  (Ethernet)
        RX packets 2901  bytes 119827 (117.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3839  bytes 15022256 (14.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.159.4  netmask 255.255.255.0  broadcast 192.168.159.255
        inet6 fe80::69a7:b207:afb9:9db4  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::2b45:985c:3f4e:908b  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:34:3a:20  txqueuelen 1000  (Ethernet)
        RX packets 65971  bytes 19511878 (18.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 56734  bytes 3494791 (3.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 32  bytes 2592 (2.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 32  bytes 2592 (2.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth9d114c0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::80e1:39ff:fe8e:240e  prefixlen 64  scopeid 0x20<link>
        ether 82:e1:39:8e:24:0e  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 656 (656.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

查看node2网桥，新增docker0

[root@node2 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:1aff:fe5e:9ea4  prefixlen 64  scopeid 0x20<link>
        ether 02:42:1a:5e:9e:a4  txqueuelen 0  (Ethernet)
        RX packets 13  bytes 980 (980.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 21  bytes 2014 (1.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.159.5  netmask 255.255.255.0  broadcast 192.168.159.255
        inet6 fe80::69a7:b207:afb9:9db4  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:24:97:9a  txqueuelen 1000  (Ethernet)
        RX packets 150770  bytes 220073564 (209.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8739  bytes 781644 (763.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 64  bytes 5568 (5.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 64  bytes 5568 (5.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth7fa5fd1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::6c8e:4dff:fed4:5761  prefixlen 64  scopeid 0x20<link>
        ether 6e:8e:4d:d4:57:61  txqueuelen 0  (Ethernet)
        RX packets 13  bytes 1162 (1.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 21  bytes 2014 (1.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

查看node1的docker网络

[root@node1 ~]# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
899aadb08971        bridge              bridge              local
2b6bb260df1e        host                host                local
f227670e6a5b        none                null                local

bridge，类似VMWare的Nat模式，（容器eth0 --> veth* --> docker0 --> 主机ens33），从docker0子网中分配一个IP给容器，并设置docker0为容器默认网关；
host，类似VMWare的桥接模式，容器没有独立网络，使用宿主机的IP和端口；
none， 网络没有配置，适用于容器不需要访问网络或者网络独立的overlay模式。

查看node2的docker网络

[root@node2 ~]# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
16f0b5194fdb        bridge              bridge              local
593efbd057ff        host                host                local
e9e9bbc0bef6        none                null                local
测试跨主机容器互访
在node1节点启动centos镜像容器node1

  • 启动
    docker run -i -t --name node1 centos /bin/bash

  • 查看容器hosts

    [root@0402a3fb29e6 /]# cat /etc/hosts
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.17.0.2	0402a3fb29e6
在node2节点启动centos镜像容器node2

  • 启动
    docker run -i -t --name node2 centos

  • 查看容器hosts

    [root@bc6194e6391e /]# cat /etc/hosts
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.17.0.2	bc6194e6391e
主机网络互访

  • 主机node1访问主机node2

    [root@node1 ~]# ping node2
PING node2 (192.168.159.5) 56(84) bytes of data.
64 bytes from node2 (192.168.159.5): icmp_seq=1 ttl=64 time=0.252 ms
64 bytes from node2 (192.168.159.5): icmp_seq=2 ttl=64 time=0.463 ms
64 bytes from node2 (192.168.159.5): icmp_seq=3 ttl=64 time=0.374 ms
64 bytes from node2 (192.168.159.5): icmp_seq=4 ttl=64 time=0.242 ms
64 bytes from node2 (192.168.159.5): icmp_seq=5 ttl=64 time=0.495 ms

  • 主机node2访问主机node1

    [root@node2 ~]# ping node1
PING node1 (192.168.159.4) 56(84) bytes of data.
64 bytes from node1 (192.168.159.4): icmp_seq=1 ttl=64 time=0.557 ms
64 bytes from node1 (192.168.159.4): icmp_seq=2 ttl=64 time=0.441 ms
64 bytes from node1 (192.168.159.4): icmp_seq=3 ttl=64 time=0.413 ms
64 bytes from node1 (192.168.159.4): icmp_seq=4 ttl=64 time=0.284 ms
64 bytes from node1 (192.168.159.4): icmp_seq=5 ttl=64 time=0.301 ms
容器访问主机

  • 容器node1访问主机node1

    [root@0402a3fb29e6 /]# ping 192.168.159.4
PING 192.168.159.4 (192.168.159.4) 56(84) bytes of data.
64 bytes from 192.168.159.4: icmp_seq=1 ttl=64 time=0.261 ms
64 bytes from 192.168.159.4: icmp_seq=2 ttl=64 time=0.250 ms
64 bytes from 192.168.159.4: icmp_seq=3 ttl=64 time=0.129 ms
64 bytes from 192.168.159.4: icmp_seq=4 ttl=64 time=0.126 ms
64 bytes from 192.168.159.4: icmp_seq=5 ttl=64 time=0.129 ms

  • 容器node1访问主机node2

    [root@0402a3fb29e6 /]# ping 192.168.159.5
PING 192.168.159.5 (192.168.159.5) 56(84) bytes of data.
64 bytes from 192.168.159.5: icmp_seq=1 ttl=63 time=1.28 ms
64 bytes from 192.168.159.5: icmp_seq=2 ttl=63 time=0.310 ms
64 bytes from 192.168.159.5: icmp_seq=3 ttl=63 time=0.724 ms
64 bytes from 192.168.159.5: icmp_seq=4 ttl=63 time=0.523 ms
64 bytes from 192.168.159.5: icmp_seq=5 ttl=63 time=0.564 ms

  • 容器node2访问主机node2

    [root@bc6194e6391e /]# ping 192.168.159.5
PING 192.168.159.5 (192.168.159.5) 56(84) bytes of data.
64 bytes from 192.168.159.5: icmp_seq=1 ttl=64 time=0.283 ms
64 bytes from 192.168.159.5: icmp_seq=2 ttl=64 time=0.210 ms
64 bytes from 192.168.159.5: icmp_seq=3 ttl=64 time=0.129 ms
64 bytes from 192.168.159.5: icmp_seq=4 ttl=64 time=0.122 ms
64 bytes from 192.168.159.5: icmp_seq=5 ttl=64 time=0.258 ms

  • 容器node2访问主机node1

    [root@bc6194e6391e /]# ping 192.168.159.4
PING 192.168.159.4 (192.168.159.4) 56(84) bytes of data.
64 bytes from 192.168.159.4: icmp_seq=1 ttl=63 time=0.571 ms
64 bytes from 192.168.159.4: icmp_seq=2 ttl=63 time=0.356 ms
64 bytes from 192.168.159.4: icmp_seq=3 ttl=63 time=0.609 ms
64 bytes from 192.168.159.4: icmp_seq=4 ttl=63 time=0.607 ms
64 bytes from 192.168.159.4: icmp_seq=5 ttl=63 time=0.417 ms
容器互访

ping -c 3 172.17.0.2

此时容器node1和容器node2在各自宿主机上没有添加对应的路由表，且两个容器IP地址相同，因此无法访问目标容器。
通过tcpdump抓包分析，数据包没有到达宿主机的docker0、veth*和ens33网桥。

  • tcpdump 命令
    tcpdump -i docker0 -n icmp
使用直接路由方式实现跨主机容器互访
修改node1主机的Docker0的网络地址

  • 修改/etc/docker/daemon.json

    docker守护进出默认配置文件，通过参数--config-file设定，默认为/etc/docker/daemon.json。
    此处指定容器node1网桥IP为172.17.1.1/24，可以在DOCKER_NETWORK_OPTIONS中设置参数--bip=172.17.1.1/24，但是二者不能同时设置，否则docker无法启动。

    [root@node1 ~]# cat /etc/docker/daemon.json 
{
"bip": "172.17.1.1/24"
}

  • 重启docker
    systemctl daemon-reload && sytemctl restart docker

  • 查看node1主机网络

    [root@node1 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.1.1  netmask 255.255.255.0  broadcast 172.17.1.255
        inet6 fe80::42:23ff:fe43:87a4  prefixlen 64  scopeid 0x20<link>
        ether 02:42:23:43:87:a4  txqueuelen 0  (Ethernet)
        RX packets 2068  bytes 97452 (95.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2847  bytes 17131048 (16.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.159.4  netmask 255.255.255.0  broadcast 192.168.159.255
        inet6 fe80::69a7:b207:afb9:9db4  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:34:3a:20  txqueuelen 1000  (Ethernet)
        RX packets 474960  bytes 52578160 (50.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 461729  bytes 28211425 (26.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethf5fadcf: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::404e:afff:feec:c983  prefixlen 64  scopeid 0x20<link>
        ether 42:4e:af:ec:c9:83  txqueuelen 0  (Ethernet)
        RX packets 187  bytes 17934 (17.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15  bytes 950 (950.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

  • 查看容器node1网络

    [root@0402a3fb29e6 /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.1.2  netmask 255.255.255.0  broadcast 172.17.1.255
        ether 02:42:ac:11:01:02  txqueuelen 0  (Ethernet)
        RX packets 8  bytes 656 (656.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
修改node2主机的Docker0的网络地址

  • 修改/etc/docker/daemon.json

    指定容器node2网桥IP为172.17.0.1/24

    [root@node1 ~]# cat /etc/docker/daemon.json 
{
"bip": "172.17.0.1/24"
}

  • 重启docker
    systemctl daemon-reload && sytemctl restart docker

  • 查看node2网络

    [root@node2 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.255.0  broadcast 172.17.0.255
        inet6 fe80::42:a6ff:feb7:78bf  prefixlen 64  scopeid 0x20<link>
        ether 02:42:a6:b7:78:bf  txqueuelen 0  (Ethernet)
        RX packets 4121  bytes 170163 (166.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4782  bytes 17294541 (16.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.159.5  netmask 255.255.255.0  broadcast 192.168.159.255
        inet6 fe80::69a7:b207:afb9:9db4  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::2b45:985c:3f4e:908b  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::c846:2166:96b5:5f86  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:24:97:9a  txqueuelen 1000  (Ethernet)
        RX packets 18895  bytes 18246491 (17.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8037  bytes 815344 (796.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 68  bytes 5904 (5.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 68  bytes 5904 (5.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth09bb648: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::9069:b6ff:fed1:8797  prefixlen 64  scopeid 0x20<link>
        ether 92:69:b6:d1:87:97  txqueuelen 0  (Ethernet)
        RX packets 4  bytes 280 (280.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12  bytes 936 (936.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
容器node1访问容器node2

无法访问

[root@0402a3fb29e6 /]# ping -c 3 172.17.0.2
PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.

--- 172.17.0.2 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2000ms

tcpdump抓取主机ens33网卡

[root@node1 ~]# tcpdump -i ens33 -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes
17:12:19.636080 IP 192.168.159.4 > 172.17.0.2: ICMP echo request, id 37, seq 1, length 64
17:12:20.640152 IP 192.168.159.4 > 172.17.0.2: ICMP echo request, id 37, seq 2, length 64
17:12:21.640212 IP 192.168.159.4 > 172.17.0.2: ICMP echo request, id 37, seq 3, length 64
容器node2访问容器node1

无法访问

[root@bc6194e6391e /]# ping -c 3 172.17.1.2
PING 172.17.1.2 (172.17.1.2) 56(84) bytes of data.
From 172.17.0.2 icmp_seq=1 Destination Host Unreachable
From 172.17.0.2 icmp_seq=2 Destination Host Unreachable
From 172.17.0.2 icmp_seq=3 Destination Host Unreachable

--- 172.17.1.2 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 1999ms
pipe 3
添加静态路由

  • node1主机
    ip route add 172.17.0.0/24 via 192.168.159.5

    [root@node1 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.159.2   0.0.0.0         UG    100    0        0 ens33
172.17.0.0      192.168.159.5   255.255.255.0   UG    0      0        0 ens33
172.17.1.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0
192.168.159.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33

    添加永久路由方式

    cat > /etc/sysconfig/static-routes << EOF
any net 172.17.0.0 netmask 255.255.255.0 gw 192.168.159.5 dev ens33
EOF

  • node2主机
    ip route add 172.17.1.0/24 via 192.168.159.4

    [root@node2 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.159.2   0.0.0.0         UG    100    0        0 ens33
172.17.0.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0
172.17.1.0      192.168.159.4   255.255.255.0   UG    0      0        0 ens33
192.168.159.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33

    添加永久路由方式

    cat > /etc/sysconfig/static-routes << EOF
any net 172.17.1.0 netmask 255.255.255.0 gw 192.168.159.4 dev ens33
EOF
容器node1访问容器node2

  • 容器node1访问容器node2

    [root@0402a3fb29e6 /]# ping 172.17.0.2 -c 3
PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.
64 bytes from 172.17.0.2: icmp_seq=1 ttl=62 time=1.34 ms
64 bytes from 172.17.0.2: icmp_seq=2 ttl=62 time=0.934 ms
64 bytes from 172.17.0.2: icmp_seq=3 ttl=62 time=0.425 ms

--- 172.17.0.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.425/0.902/1.347/0.377 ms

  • 容器node2抓包eth0

    [root@bc6194e6391e /]# tcpdump -i eth0 -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:49:31.700980 IP 192.168.159.4 > 172.17.0.2: ICMP echo request, id 46, seq 1, length 64
09:49:31.701198 IP 172.17.0.2 > 192.168.159.4: ICMP echo reply, id 46, seq 1, length 64
09:49:32.703187 IP 192.168.159.4 > 172.17.0.2: ICMP echo request, id 46, seq 2, length 64
09:49:32.703231 IP 172.17.0.2 > 192.168.159.4: ICMP echo reply, id 46, seq 2, length 64
09:49:33.704675 IP 192.168.159.4 > 172.17.0.2: ICMP echo request, id 46, seq 3, length 64
09:49:33.704695 IP 172.17.0.2 > 192.168.159.4: ICMP echo reply, id 46, seq 3, length 64

  • 容器node2访问容器node1

    [root@bc6194e6391e /]# ping 172.17.1.2 -c 3
PING 172.17.1.2 (172.17.1.2) 56(84) bytes of data.
64 bytes from 172.17.1.2: icmp_seq=1 ttl=62 time=1.69 ms
64 bytes from 172.17.1.2: icmp_seq=2 ttl=62 time=0.987 ms
64 bytes from 172.17.1.2: icmp_seq=3 ttl=62 time=1.15 ms

--- 172.17.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.987/1.279/1.695/0.304 ms

  • 容器node1抓包eth0

    [root@0402a3fb29e6 /]# tcpdump -i eth0 -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:52:51.691610 IP 192.168.159.5 > 172.17.1.2: ICMP echo request, id 21, seq 1, length 64
09:52:51.691738 IP 172.17.1.2 > 192.168.159.5: ICMP echo reply, id 21, seq 1, length 64
09:52:52.693258 IP 192.168.159.5 > 172.17.1.2: ICMP echo request, id 21, seq 2, length 64
09:52:52.693361 IP 172.17.1.2 > 192.168.159.5: ICMP echo reply, id 21, seq 2, length 64
09:52:53.695265 IP 192.168.159.5 > 172.17.1.2: ICMP echo request, id 21, seq 3, length 64
09:52:53.695350 IP 172.17.1.2 > 192.168.159.5: ICMP echo reply, id 21, seq 3, length 64

通过配置静态路由的方式达到了跨主机docker容器的网络互访，但是在大规模集群中手动配置过于低效，下面将进一步进行优化。

使用fannel配置docker集群网络
环境准备

  • 停止docker服务
    systemctl stop docker

  • 启动etcd

    flaanel使用etcd作为数据库，存放网络配置、已分配 的subnet、host的IP等信息，
    所以要先保证etcd已正确启动，前边章节我们已经介绍了ETCD集群的配置，在这里直接使用。

    [root@node1 pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem cluster-health
member 46899d42c87d524e is healthy: got healthy result from https://192.168.159.4:2379
member a3ec213779ea2c81 is healthy: got healthy result from https://192.168.159.3:2379
cluster is healthy

  • 清除路由
    sed -i '$d' /etc/sysconfig/static-routes && systemctl restart network

    [root@node1 etc]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.159.2   0.0.0.0         UG    100    0        0 ens33
192.168.159.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33

    [root@node2 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.159.2   0.0.0.0         UG    100    0        0 ens33
192.168.159.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33
添加flannel网络信息到etcd

  • Flannel网络配置

    flannel-etcd-config.json

    {
    "Network": "172.17.0.0/8",
    "SubnetLen": "24",
    "SubnetMin": "172.17.0.0",
    "SudbnetMax": "172.17.255.0",
    "Backend": {
        "Type": "vxlan",
        "VNI": 1,      
        "Port": 8472         
    }
}
    • 字段释义
      Backend文档地址
        Network: 指定flannel地址池,必须至少能容纳四个子网；
  SubnetLen: 宿主机docker0的子网掩码长度,注意为int类型数字；
  SubnetMin: 分配给子网的起始IP，默认为地址池第一个子网IP；
  SudbnetMax: 分配给子网的最后一个IP，默认为地址池最后一个子网IP；
  Backend: 使用后端类型及其后端特征:
    Type: 后端类型，默认udp，推荐vxlan，性能最优host-gw但不支持云环境;
    VNI: 要使用的VXLAN标识符(VNI)。在Linux上，默认值为1。在Windows上应该大于或等于4096;
    Port: UDP端口，用于发送封装的数据包。在Linux上，内核默认值为8472，但在Windows上必须是4789。

  • 添加配置到etcd

    [root@node1 pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem set /k8s/network/config < /opt/k8s/node/etc/flannel-etcd-config.json 
{
    "Network": "172.17.0.0",
    "SubnetLen": 24,
    "SubnetMin": "172.17.0.0",
    "SudbnetMax": "172.17.255.0",
    "Backend": {
        "Type": "vxlan",
        "VNI": 1,      
        "Port": 8472         
    }
}

  • 确认是否配置到etcd

    [root@node1 pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem get /k8s/network/config
{
    "Network": "172.17.0.0/8",
    "SubnetLen": 24,
    "SubnetMin": "172.17.0.0",
    "SudbnetMax": "172.17.255.0",
    "Backend": {
        "Type": "vxlan",
        "VNI": 1,      
        "Port": 8472         
    }
}
flannel安装

以下操作在node1和node2同时执行

官方文档
参考博客

flannel下载

官方下载地址

wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
tar -zxvf flannel-v0.11.0-linux-amd64.tar.gz -C flannel
cd flannel
cp -f flanneld mk-docker-opts.sh /usr/local/bin
flannel配置文件
cat > /opt/k8s/node/etc/flanneld.conf << EOF
FLANNELD_INSECURE_OPTS="--etcd-cafile=/opt/etcd/pki/ca.pem \
--etcd-certfile=/opt/etcd/pki/etcdctl.pem \
--etcd-keyfile=/opt/etcd/pki/etcdctl-key.pem \
--etcd-endpoints=https://192.168.159.3:2379,https://192.168.159.4:2379 \
--etcd-prefix=/k8s/network \
--kube-subnet-mgr=false \
--subnet-file=/run/flannel/subnet.env \
--iface=ens33 \
--iptables-resync=5 \
--ip-masq=true"
EOF

  • 参数说明

    参数名说明
    --etcd-cafileetcd根证书
    --etcd-certfileetcd客户端证书
    --etcd-keyfileetcd客户端证书私钥
    --etcd-endpointsetcd服务地址
    --etcd-prefixetcd配置的flannel网络的key
    --kube-subnet-mgrfalse网络配置从etcd中读取，true网络配置从--net-config-path指定的文件中读取
    --subnet-file生成子网信息保存在该文件
    --iface指定跨主机通信接口，默认为主机的默认路由地址，可以多次指定，按顺序查找
    --iptables-resynciptables刷新周期
    --ip-masq为flannel外的网络设置IP伪装
flannel服务文件
cat > /usr/lib/systemd/system/flanneld.service << EOF
[Unit]
Description=Flanneld
Documentation=https://github.com/coreos/flannel
After=network.target
After=network-online.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/opt/k8s/node/etc/flanneld.conf
ExecStart=/usr/local/bin/flanneld $FLANNELD_INSECURE_OPTS
ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

  • 字段说明

    字段名说明
    Typenotify启动成功后返回READY=1
    Beforeflannel在docker前启动
    ExecStartPostExecStart执行成功后执行的操作，此处生成docker的子网信息
flannel服务启动

  • 启动

    systemctl daemon-reload && systemctl start flanneld

  • 验证

    • 服务状态

      [root@node1 pki]# systemctl status flanneld
● flanneld.service - Flanneld
   Loaded: loaded (/usr/lib/systemd/system/flanneld.service; disabled; vendor preset: disabled)
   Active: active (running) since 二 2019-08-27 15:11:05 CST; 30min ago
     Docs: https://github.com/coreos/flannel
  Process: 2447 ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env (code=exited, status=0/SUCCESS)
 Main PID: 2432 (flanneld)
    Tasks: 8
   Memory: 8.4M
   CGroup: /system.slice/flanneld.service
           └─2432 /usr/local/bin/flanneld --etcd-cafile=/opt/etcd/pki/ca.pem --etcd-certfile=/opt/etcd/pki/etcdctl.pem --etcd-keyfile=/opt/etcd/pki/etcdctl-key.pem --etcd-endpoints=https://192.168.159.3..

    • etcd存储信息

      [root@node1 pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem ls /k8s/network/subnets
/k8s/network/subnets/172.17.83.0-24
/k8s/network/subnets/172.17.46.0-24

[root@node1 pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem get /k8s/network/subnets/172.17.83.0-24
{"PublicIP":"192.168.159.4","BackendType":"vxlan","BackendData":{"VtepMAC":"5a:f0:7a:bf:de:79"}}

[root@node1 pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem get /k8s/network/subnets/172.17.46.0-24
{"PublicIP":"192.168.159.5","BackendType":"vxlan","BackendData":{"VtepMAC":"fe:32:4e:da:3f:21"}}

    • node1本地生成的docker子网信息

      [root@node1 pki]# cat /run/flannel/subnet.env 
DOCKER_OPT_BIP="--bip=172.17.83.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.17.83.1/24 --ip-masq=false --mtu=1450"

    • node2本地生成的docker子网信息

      [root@node2 pki]# cat /run/flannel/subnet.env 
DOCKER_OPT_BIP="--bip=172.17.46.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.17.46.1/24 --ip-masq=false --mtu=1450"

    • node1主机网卡

      [root@node1 pki]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.159.4  netmask 255.255.255.0  broadcast 192.168.159.255
        inet6 fe80::69a7:b207:afb9:9db4  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:34:3a:20  txqueuelen 1000  (Ethernet)
        RX packets 430580  bytes 54653587 (52.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 431431  bytes 56850150 (54.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.83.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::58f0:7aff:febf:de79  prefixlen 64  scopeid 0x20<link>
        ether 5a:f0:7a:bf:de:79  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 2749  bytes 333239 (325.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2749  bytes 333239 (325.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

    • node2主机网卡

      [root@node2 ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.159.5  netmask 255.255.255.0  broadcast 192.168.159.255
        inet6 fe80::69a7:b207:afb9:9db4  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::2b45:985c:3f4e:908b  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:24:97:9a  txqueuelen 1000  (Ethernet)
        RX packets 1355  bytes 128086 (125.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1421  bytes 157618 (153.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.46.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::f828:21ff:fef6:2bd4  prefixlen 64  scopeid 0x20<link>
        ether fa:28:21:f6:2b:d4  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 64  bytes 5568 (5.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 64  bytes 5568 (5.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
重新配置docker服务
服务文件配置

修改EnvironmentFile等于flannel为docker生成的网络配置/run/flannel/subnet.env

cat > docker.service << EOF 
[Unit]
Description=Docker Engine Service
Documentation=https://docs.docker.com
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin"
EnvironmentFile=-/run/flannel/subnet.env
ExecStart=/usr/local/bin/dockerd $DOCKER_NETWORK_OPTIONS 
ExecReload=/bin/kill -s HUP $MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target
EOF
服务试启动
[root@node1 etc]# systemctl daemon-reload && systemctl start docker
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

通过journalctl -xe注意到日志中报如下错误：

unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: bip: (from flag: 172.17.83.1/24, from file: 172.17.1.1/24)

问题解决：
前边通过直接路由方式设置跨主机容器网络时候说过/etc/docker/daemon.json文件与参数--bip不能同时设置，在变量DOCKER_NETWORK_OPTIONS中已经包含了参数--bip，因此需要把/etc/docker/daemon.json文件删除。

rm -f /etc/docker/daemon.json
systemctl daemon-reload && systemctl start docker
网络查看

node1主机网络

[root@node1 etc]# ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.83.1  netmask 255.255.255.0  broadcast 172.17.83.255
        inet6 fe80::42:e2ff:feaf:525d  prefixlen 64  scopeid 0x20<link>
        ether 02:42:e2:af:52:5d  txqueuelen 0  (Ethernet)
        RX packets 156  bytes 9898 (9.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 67  bytes 80756 (78.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.159.4  netmask 255.255.255.0  broadcast 192.168.159.255
        inet6 fe80::69a7:b207:afb9:9db4  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:34:3a:20  txqueuelen 1000  (Ethernet)
        RX packets 530560  bytes 67349085 (64.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 533082  bytes 70220830 (66.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.83.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::58f0:7aff:febf:de79  prefixlen 64  scopeid 0x20<link>
        ether 5a:f0:7a:bf:de:79  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 2841  bytes 338023 (330.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2841  bytes 338023 (330.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

node2主机网络

[root@node2 etc]# ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.46.1  netmask 255.255.255.0  broadcast 172.17.46.255
        inet6 fe80::42:d4ff:fe47:66f4  prefixlen 64  scopeid 0x20<link>
        ether 02:42:d4:47:66:f4  txqueuelen 0  (Ethernet)
        RX packets 200  bytes 12034 (11.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 40  bytes 3394 (3.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.159.5  netmask 255.255.255.0  broadcast 192.168.159.255
        inet6 fe80::69a7:b207:afb9:9db4  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::2b45:985c:3f4e:908b  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:24:97:9a  txqueuelen 1000  (Ethernet)
        RX packets 2201  bytes 199369 (194.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2147  bytes 256110 (250.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.46.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::f828:21ff:fef6:2bd4  prefixlen 64  scopeid 0x20<link>
        ether fa:28:21:f6:2b:d4  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 64  bytes 5568 (5.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 64  bytes 5568 (5.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
测试跨主机容器互访

  • 启动node1容器
    启动容器
    docker start node1

    查看宿主机网络

    [root@node1 etc]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.83.1  netmask 255.255.255.0  broadcast 172.17.83.255
        inet6 fe80::42:e2ff:feaf:525d  prefixlen 64  scopeid 0x20<link>
        ether 02:42:e2:af:52:5d  txqueuelen 0  (Ethernet)
        RX packets 301  bytes 21598 (21.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 95  bytes 82756 (80.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.159.4  netmask 255.255.255.0  broadcast 192.168.159.255
        inet6 fe80::69a7:b207:afb9:9db4  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:34:3a:20  txqueuelen 1000  (Ethernet)
        RX packets 613016  bytes 77826826 (74.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 616778  bytes 81210151 (77.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.83.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::58f0:7aff:febf:de79  prefixlen 64  scopeid 0x20<link>
        ether 5a:f0:7a:bf:de:79  txqueuelen 0  (Ethernet)
        RX packets 22  bytes 1536 (1.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 139  bytes 11712 (11.4 KiB)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 2917  bytes 341975 (333.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2917  bytes 341975 (333.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth9c659ff: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::80a2:90ff:fe94:c603  prefixlen 64  scopeid 0x20<link>
        ether 82:a2:90:94:c6:03  txqueuelen 0  (Ethernet)
        RX packets 145  bytes 13730 (13.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 36  bytes 2656 (2.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
      
[root@node1 etc]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.159.2   0.0.0.0         UG    100    0        0 ens33
172.17.46.0     172.17.46.0     255.255.255.0   UG    0      0        0 flannel.1
172.17.83.0     0.0.0.0         255.255.255.0   U     0      0        0 docker0
192.168.159.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33

    进入node1容器
    docker attach node1

    查看node1容器网络

    [root@0402a3fb29e6 /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.83.2  netmask 255.255.255.0  broadcast 172.17.83.255
        ether 02:42:ac:11:53:02  txqueuelen 0  (Ethernet)
        RX packets 36  bytes 2656 (2.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 145  bytes 13730 (13.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 50  bytes 4200 (4.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 50  bytes 4200 (4.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

  • 启动node2容器
    启动容器
    docker start node2

    查看宿主机网络

    [root@node2 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.46.1  netmask 255.255.255.0  broadcast 172.17.46.255
        inet6 fe80::42:d4ff:fe47:66f4  prefixlen 64  scopeid 0x20<link>
        ether 02:42:d4:47:66:f4  txqueuelen 0  (Ethernet)
        RX packets 411  bytes 26542 (25.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 79  bytes 6604 (6.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.159.5  netmask 255.255.255.0  broadcast 192.168.159.255
        inet6 fe80::69a7:b207:afb9:9db4  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::2b45:985c:3f4e:908b  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:24:97:9a  txqueuelen 1000  (Ethernet)
        RX packets 3649  bytes 336880 (328.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3511  bytes 407431 (397.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.46.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::f828:21ff:fef6:2bd4  prefixlen 64  scopeid 0x20<link>
        ether fa:28:21:f6:2b:d4  txqueuelen 0  (Ethernet)
        RX packets 18  bytes 1548 (1.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 190  bytes 13560 (13.2 KiB)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 64  bytes 5568 (5.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 64  bytes 5568 (5.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth71ae48a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::ccb2:8eff:fe58:e9f9  prefixlen 64  scopeid 0x20<link>
        ether ce:b2:8e:58:e9:f9  txqueuelen 0  (Ethernet)
        RX packets 211  bytes 17462 (17.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 47  bytes 3866 (3.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@node2 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.159.2   0.0.0.0         UG    100    0        0 ens33
172.17.46.0     0.0.0.0         255.255.255.0   U     0      0        0 docker0
172.17.83.0     172.17.83.0     255.255.255.0   UG    0      0        0 flannel.1
192.168.159.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33

    进入node2容器
    docker attach node2

    查看node2容器网络

    [root@bc6194e6391e /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.46.2  netmask 255.255.255.0  broadcast 172.17.46.255
        ether 02:42:ac:11:2e:02  txqueuelen 0  (Ethernet)
        RX packets 47  bytes 3866 (3.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 211  bytes 17462 (17.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

  • 防火墙设置

    由于flannel采用了UDP的8472端口进行封装数据包发送，因此需要开放该端口

    firewall-cmd --zone=public --add-port=8472/udp --permanent
firewall-cmd reload

    [root@node2 ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens33
  sources: 
  services: ssh dhcpv6-client
  ports: 2380/tcp 2379/tcp 8472/udp
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

  • 容器node1网络测试

    # node1主机访问node1容器
[root@node1 etc]# ping 172.17.83.2 -c 3
PING 172.17.83.2 (172.17.83.2) 56(84) bytes of data.
64 bytes from 172.17.83.2: icmp_seq=1 ttl=64 time=0.103 ms
64 bytes from 172.17.83.2: icmp_seq=2 ttl=64 time=0.116 ms
64 bytes from 172.17.83.2: icmp_seq=3 ttl=64 time=0.171 ms

--- 172.17.83.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.103/0.130/0.171/0.029 ms

# node2主机访问node1容器
[root@node2 ~]# ping 172.17.83.2 -c 3
PING 172.17.83.2 (172.17.83.2) 56(84) bytes of data.
64 bytes from 172.17.83.2: icmp_seq=1 ttl=63 time=0.823 ms
64 bytes from 172.17.83.2: icmp_seq=2 ttl=63 time=0.699 ms
64 bytes from 172.17.83.2: icmp_seq=3 ttl=63 time=0.748 ms

--- 172.17.83.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.699/0.756/0.823/0.060 ms

# node1容器访问node1主机
[root@0402a3fb29e6 /]# ping 192.168.159.4 -c 3
PING 192.168.159.4 (192.168.159.4) 56(84) bytes of data.
64 bytes from 192.168.159.4: icmp_seq=1 ttl=64 time=0.140 ms
64 bytes from 192.168.159.4: icmp_seq=2 ttl=64 time=0.120 ms
64 bytes from 192.168.159.4: icmp_seq=3 ttl=64 time=0.132 ms

--- 192.168.159.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.120/0.130/0.140/0.015 ms

# node1容器访问node2主机
[root@0402a3fb29e6 /]# ping 192.168.159.5 -c 3
PING 192.168.159.5 (192.168.159.5) 56(84) bytes of data.
64 bytes from 192.168.159.5: icmp_seq=1 ttl=63 time=1.08 ms
64 bytes from 192.168.159.5: icmp_seq=2 ttl=63 time=0.670 ms
64 bytes from 192.168.159.5: icmp_seq=3 ttl=63 time=0.711 ms

--- 192.168.159.5 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.670/0.821/1.083/0.187 ms

# node1容器访问node2容器  
[root@0402a3fb29e6 /]# ping 172.17.46.2 -c 3
PING 172.17.46.2 (172.17.46.2) 56(84) bytes of data.
64 bytes from 172.17.46.2: icmp_seq=1 ttl=62 time=1.55 ms
64 bytes from 172.17.46.2: icmp_seq=2 ttl=62 time=0.862 ms
64 bytes from 172.17.46.2: icmp_seq=3 ttl=62 time=0.757 ms

--- 172.17.46.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.757/1.058/1.556/0.355 ms

# node1容器访问百度  
[root@0402a3fb29e6 /]# ping www.baidu.com -c 3
PING www.a.shifen.com (14.215.177.39) 56(84) bytes of data.
64 bytes from 14.215.177.39 (14.215.177.39): icmp_seq=1 ttl=127 time=36.0 ms
64 bytes from 14.215.177.39 (14.215.177.39): icmp_seq=2 ttl=127 time=35.9 ms
64 bytes from 14.215.177.39 (14.215.177.39): icmp_seq=3 ttl=127 time=35.8 ms

--- www.a.shifen.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 35.882/35.951/36.009/0.163 ms

  • 容器node2网络测试

    略

至此Docker跨主机集群网络配置完毕，下一步接着探讨Node节点的其他服务安装。

Let's Golang
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
k8s使用时无法ping通服务器From IP地址 icmp_seq=1 Destination Host Unreachable
分享式获得也是一种学习的态度
09-20 1566
每个人都有惰性，但不断学习是好好生活的根本，共勉！
K8S集群中Node节点资源不足导致Pod无法运行的故障排查思路
江晓龙的博客
08-25 3685
每一台Node节点中默认限制最多运行110个Pod资源，当一个应用程序有成百上千的Pod资源时，如果不扩容Node节点或者修改最大Pod数量限制，那么就会导致部分Pod资源无法正常运行，因为节点已经没有资源可以被调度了。2）从Pod的详细信息中可以得知，3个节点中，没有一个节点符合Pod的资源设置，其中有2个节点内内存不足和一个节点是Master，污点不能被容忍。当Pod资源数量太多，已经达到了Node节点默认的110个Pod时，新创建的Pod资源会一直处于Pending状态，并且大量的Pod会频繁重启。
Kubernetes部署搭建及测试
m0_59770129的博客
11-21 505
Kubernetes集群的搭建部署
kubernetes学习-集群搭建
tootsy_you的博客
04-02 1032
文章目录1.环境准备1.1注意事项2.操作步骤2.1 初始化操作（三台机器都需要做）2.1.1 系统初始化2.1.2 检查网络连通性2.1.3 将桥接的 IPv4 流量传递到 iptables 的链2.1.4 安装docker2.1.5 安装nfs共享存储2.1.6 配置k8s源2.1.7 k8s安装 1.环境准备 此次搭建准备三台服务器，系统皆为CentOS Linux release 7.6.1810 (Core) 节点名称 节点地址 k8s-Master 192.168.204.128
解决kubenetes中pod一直处于ContainerCreating
weixin_42599091的博客
07-09 1282
在k8s的master服务器中运行 kubectl apply -f nginx-deployment.yaml 创建一个nginx的服务 然后发现pod一直处于ContainerCreating状态 然后我们查看该pod的详细信息 发现报错信息 /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt：no such file or directory 这个问题在于 node 节点的服务器上缺少python-rhsm...
使用kubeadm搭建k8s高可用集群
qq_38419276的博客
10-04 582
环境准备 系统使用的Ubuntu18.04 主机IP 主机名 docker版本 172.31.1.10 k8s-master1 19.03.15 172.31.1.11 k8s-master2 19.03.15 172.31.1.12 k8s-master3 19.03.15 172.31.1.13 harbor 19.03.15 172.31.1.14 haproxy1 172.31.1.15 haproxy2 172.31.1.16 k8s-node1 19.
ansible一键部署k8s1.16集群新增node节点-二进制-参考
06-28
ansible一键部署k8s1.16集群新增node节点—二进制—参考
node节点加入k8s集群
weixin_43707076的博客
08-01 425
node节点加入k8s集群 依赖此集群：https://blog.csdn.net/weixin_43707076/article/details/119282577 初始化操作 参考：https://blog.csdn.net/weixin_43707076/article/details/119282577 变化： vi /etc/docker/daemon.json { "registry-mirrors":["https://rsbud4vc.mirror.aliyuncs.com","https
k8s集群部署hello应用实例
10-20
在本教程中，我们将深入探讨如何在Kubernetes（k8s）集群上部署一个名为"hello"的应用程序实例。Kubernetes是一种流行的容器编排系统，它允许开发者和运维人员高效地管理和扩展微服务应用程序。 首先，理解...
K8s（二）：集群部署-----＞超详细
最新发布
✨ 欢迎来到【Seal ^_^ 的CSDN博客】！这里是我记录技术心得、分享经验的地方。✨
12-12 3万+
🔴 K8s（二）：集群部署-----＞超详细
k8s容器跨节点ping不通
superzlc的专栏
02-11 2194
用的flannel，遇到的情况是各节点有两个网卡enp0s3和enp0s8，需要加参数 --iface=enp0s8 让flannel.1关联到enp0s8上。 这个不正确，不仅容器跨节点ping不通，也可能会容器内nslookup查域名不正常，通过ingress集群外访问service不正常。 ...
k8s排查无法访问服务
过于丰富，无法简介
08-01 4215
如果网络没有为“发夹模式”流量生成正确配置，通常当kube-proxy以iptables模式运行，并且Pod与桥接网络连接时，就会发生这种情况。大致结论是，应该由cni插件来根据这个值来做对应的操作，所以veth该文件内容设置1（本人/sys/devices/virtual/net/docker0/brif/路径下没有文件所以无法设置）上面是通过cni设置hairpin-mode参数，这里再通过flannel设置参数（本人系统文件本来就是这个配置，所以无效）访问，gg，啥也不是。..............
【运维】K8S集群部署系列之普通Node节点搭建（八）
weixin_39974140的博客
09-10 9899
系统环境 Node IP CentOS kernel cpu memory node1 192.168.159.4 CentOS Linux release 7.4.1708 (Core) 3.10.0-693.el7.x86_64 Intel® Core™ i5-7500 CPU @ 3.40GHz * 1 2G node2 192.168.159.5 CentOS Linu...
从零搭建一个自动化运维体系
热门推荐
高效运维
03-23 4万+
作者简介：胥峰，著有畅销书《Linux运维最佳实践》、译著《DevOps：软件架构师行动指南》，资深运维专家，有 11 年运维经验，在业界颇具威望和影响力。2006 年毕业于南京大学，曾就职于盛大游戏等大型知名互联网公司，现就职于Garena Singapore 。拥有工信部认证高级信息系统项目管理师资格。对自动化运维体系的需求，是随着业务的增长、对运维效率和质量的要求不断提高而产生的。前言：在很
K8s入门-K8s节点notReady状态解决
weixin_43169156的博客
04-01 6615
K8s节点notReady状态解决 挂掉原因：我想要通过externalIP来发布一个service，同时要删除旧的pod，删除命令执行后，节点就不可用了。 错误操作复现 创建externalIP类型的service 将已有的deployments/demo的节点置为0（这一步有大问题） 删除已有的pod节点（直接卡死，之后node节点全部断掉了） [root@k8s-master01 ~]# kubectl apply -f services-externalip-demo.yaml service
一直显示Destination Host Unreachable的解决方法之一
fighting_dream的博客
04-28 3万+
** 一直显示Destination Host Unreachable的解决方法之一 新手小白，
K8s集群部署Docker-Harbor镜像：Node认证详解
"这篇文档主要介绍了如何在Kubernetes (k8s) 集群中部署基于Docker-Harbor的镜像，并在Node节点上进行必要的认证过程。" 在Kubernetes环境中部署应用通常涉及创建Deployment、Service以及Ingress资源。在本案例中，...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值