目录
1、实现并总结容器跨主机的通信过程
默认配置下宿主机安装好docker后会有一个生成一个docker0的网卡,正常情况下使用docker run拉起来的容器都会在同一个网段172.17.0.0中,通过以下实验验证容器在跨主机通信时网络走向
实验机器的网卡
root@ubuntu2204-server1:~#ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.208 netmask 255.255.255.0 broadcast 10.0.0.255
inet6 fe80::20c:29ff:fe67:6cdd prefixlen 64 scopeid 0x20<link>
veth8fda2d4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::4446:c2ff:fef3:e196 prefixlen 64 scopeid 0x20<link>
ether 46:46:c2:f3:e1:96 txqueuelen 0 (Ethernet)
root@ubuntu2204-server2:~# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.206 netmask 255.255.255.0 broadcast 10.0.0.255
inet6 fe80::20c:29ff:fe5a:367d prefixlen 64 scopeid 0x20<link>
veth6367947: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::3c28:55ff:fef3:6947 prefixlen 64 scopeid 0x20<link>
ether 3e:28:55:f3:69:47 txqueuelen 0 (Ethernet)
在宿主机1上抓包,开另一个窗口进入nginx容器对宿主机2发起请求
root@ubuntu2204-server1:~# tcpdump -nn -vvv -i veth8fda2d4 -vvv -nn ! port 22 and ! arp and ! port 53 -w 1-veth8fda2d4.pcap
root@ubuntu2204-server1:~# tcpdump -nn -vvv -i docker0 -vvv -nn ! port 22 and ! arp and ! port 53 -w 2-docker0.pcap
root@ubuntu2204-server1:~# tcpdump -nn -vvv -i ens33 -vvv -nn ! port 22 and ! arp and ! port 53 -w 3-ens33.pcap
/ # curl http://10.0.0.206:8080
目的主机2抓包
root@ubuntu2204-server2:~# tcpdump -nn -vvv -i ens33 -vvv -nn ! port 22 and ! arp and ! port 53 -w 4-ens33.pcap
root@ubuntu2204-server2:~# tcpdump -nn -vvv -i docker0 -vvv -nn ! port 22 and ! arp and ! port 53 -w 5-docker0.pcap
root@ubuntu2204-server2:~# tcpdump -nn -vvv -i veth6367947 -vvv -nn ! port 22 and ! arp and ! port 53 -w 6-veth6367947.pcap
容器网络端口信息如下:
root@ubuntu2204-server1
root@ubuntu2204-server1:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:67:6c:dd brd ff:ff:ff:ff:ff:ff
altname enp2s1
inet 10.0.0.208/24 metric 100 brd 10.0.0.255 scope global dynamic ens33
valid_lft 1666sec preferred_lft 1666sec
inet6 fe80::20c:29ff:fe67:6cdd/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:9d:58:cb:00 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:9dff:fe58:cb00/64 scope link
valid_lft forever preferred_lft forever
5: veth8fda2d4@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
link/ether 46:46:c2:f3:e1:96 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::4446:c2ff:fef3:e196/64 scope link
valid_lft forever preferred_lft forever
root@ubuntu2204-server1:~#
root@ubuntu2204-server1:~# docker exec -it b167216dfafc sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
4: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
/ #
root@ubuntu2204-server2
root@ubuntu2204-server2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:5a:36:7d brd ff:ff:ff:ff:ff:ff
altname enp2s1
inet 10.0.0.206/24 metric 100 brd 10.0.0.255 scope global dynamic ens33
valid_lft 1585sec preferred_lft 1585sec
inet6 fe80::20c:29ff:fe5a:367d/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:93:70:6c:07 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:93ff:fe70:6c07/64 scope link
valid_lft forever preferred_lft forever
5: veth6367947@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
link/ether 3e:28:55:f3:69:47 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::3c28:55ff:fef3:6947/64 scope link
valid_lft forever preferred_lft forever
root@ubuntu2204-server2:~#
root@ubuntu2204-server2:~# docker exec -it d14468ad6c45 sh
/usr/local/tomcat # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
4: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forev
使用抓包分析工具确定网络流量走向
第1个抓包:1-veth8fda2d4.pcap
src:宿主机1容器:02:42:ac:11:00:02
dst:宿主机1docker0:02:42:9d:58:cb:00
第2个抓包:2-docker0.pcap9
src:宿主机1容器:02:42:ac:11:00:02
dst:宿主机1docker0:02:42:9d:58:cb:00
第3个抓包:3-ens33.pcap
src:宿主机1ens33:00:0c:29:67:6c:dd
dst:宿主机2ens33:00:0c:29:5a:36:7d
第4个抓包:4-ens33.pcap
src:宿主机1ens33:00:0c:29:67:6c:dd
dst:宿主机2ens33:00:0c:29:5a:36:7d
第5个抓包:5-docker0.pcap
src:宿主机2docker0:02:42:93:70:6c:07
dst:宿主机2容器:02:42:ac:11:00:02
第6个抓包:6-veth6367947.pcap
src:宿主机2docker0:02:42:93:70:6c:07
dst:宿主机2容器:02:42:ac:11:00:02
可以看到,流量的走向是:
宿主机1容器——宿主机1docker0——宿主机1ens33——宿主机2ens33——宿主机2docker0——宿主机2容器
docker跨主机的容器通信是逐层转发出去的(同个宿主机内则,通过docker0)
2、Dockerfile的常见指令
FROM alpine #指定使用的镜像
LABEL zezehu #指定维护人信息
ADD [--chown root:root ]src dest #添加宿主机本地文件到镜像中去,会自动解压tar.gz压缩包,不会解压zip
COPY [--chown root:root ]src dest #添加宿主机本地文件到镜像中去,不会解压任何文件
ENV NAME="huzewei" #设置环境变量
USER<user>[:<group>] or USER<UID>[:<GID>] #设置执行命令的用户
RUN apt install -y vim unzip && cd /etc/nginx #run后面带执行的命令,无法执行交互式命令
VOLUME [/data/nginx] #挂载卷,定义volume
WORKDIR /data/data1 #用与定义当前的工作目录,会影响后续的命令
EXPOSE <port> #声明端口映射
CMD ["可执行脚本或者程序","参数1","参数2"]
ENTRYPOINT ["可执行脚本或者程序","-b","-c"]
CMD与ENTRYPOINT结合使用,把CMD的命令当作参数给ENTRYPOINT后面的脚本
ENTERYPOINT ["top","-b"]
CMD ["-c"]
等于
ENTRYPOINT ["top","-b","-c"]
注:经常变动包文件/配置/放后面
3、使用Dockerfile构建Nginx镜像
- 安装docker #略
- 下载Nginx包文件, #https://hub.docker.com/search?q=nginx
root@docker1:/data/nginx# ll
total 1116
drwxr-xr-x 2 root root 4096 Jan 22 09:28 ./
drwxr-xr-x 5 root root 4096 Jan 12 08:00 ../
-rw-r--r-- 1 root root 265 Aug 25 16:54 build-command.sh
-rw-r--r-- 1 root root 869 Jan 12 08:05 Dockerfile
-rw-r--r-- 1 root root 38751 Aug 25 16:54 frontend.tar.gz
-rw-r--r-- 1 root root 1073322 Aug 25 16:55 nginx-1.22.0.tar.gz
-rw-r--r-- 1 root root 2812 Jan 12 08:46 nginx.conf
-rw-r--r-- 1 root root 1139 Aug 25 16:54 sources.list
- 编辑Dockerfile文件
oot@docker1:/data/nginx# vim Dockerfile
root@docker1:/data/nginx# cat Dockerfile
FROM ubuntu:22.04
MAINTAINER "huzewei"
ADD sources.list /etc/apt/sources.list
RUN apt update && apt install -y iproute2 ntpdate tcpdump telnet traceroute nfs-kernel-server nfs-common lrzsz tree openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev ntpdate tcpdump telnet traceroute gcc openssh-server lrzsz tree openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev ntpdate tcpdump telnet traceroute iotop unzip zip make
ADD nginx-1.22.0.tar.gz /usr/local/src/
RUN cd /usr/local/src/nginx-1.22.0 && ./configure --prefix=/apps/nginx && make && make install && ln -sv /apps/nginx/sbin/nginx /usr/bin
RUN groupadd -g 2088 nginx && useradd -g nginx -s /usr/sbin/nologin -u 2088 nginx && chown -R nginx.nginx /apps/nginx
ADD nginx.conf /apps/nginx/conf/
ADD frontend.tar.gz /apps/nginx/html/
EXPOSE 80 443
#ENTRYPOINT ["nginx"]
CMD ["nginx","-g","daemon off;"]
root@docker1:/data/nginx#
- 构建镜像并上创仓库
root@docker1:/data/nginx# vi build-command.sh
root@docker1:/data/nginx#cat build-command.sh
#!/bin/bash
docker build -t zezehu.cloud/myserver/nginx:20220122 .
docker push zezehu.cloud/myserver/nginx:20220122
#/usr/local/bin/nerdctl build -t harbor.magedu.net/magedu/nginx-base:1.22.0 .
#/usr/local/bin/nerdctl push harbor.magedu.net/magedu/nginx-base:1.22.0
root@docker1:/data/nginx# sh build-command.sh
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
Install the buildx component to build images with BuildKit:
https://docs.docker.com/go/buildx/
Sending build context to Docker daemon 1.123MB
Step 1/11 : FROM ubuntu:22.04
22.04: Pulling from library/ubuntu
29202e855b20: Pull complete
Digest: sha256:e6173d4dc55e76b87c4af8db8821b1feae4146dd47341e4d431118c7dd060a74
Status: Downloaded newer image for ubuntu:22.04
---> e34e831650c1
Step 2/11 : MAINTAINER "huzewei"
---> Running in 09bbfeef352f
Removing intermediate container 09bbfeef352f
---> 06eb6b6d5e67
Step 3/11 : ADD sources.list /etc/apt/sources.list
---> 5d5fa7e0b2a5
Step 4/11 : RUN apt update && apt install -y iproute2 ntpdate tcpdump telnet traceroute nfs-kernel-server nfs-common lrzsz tree openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev ntpdate tcpdump telnet traceroute gcc openssh-server lrzsz tree openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev ntpdate tcpdump telnet traceroute iotop unzip zip make
---> Running in 017365c09752
root@docker1:/data/nginx# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
zezehu.cloud/myserver/nginx 20240122 209a8378c051 38 seconds ago 453MB
zezehu.cloud/pub-images/tomcat-base v8.5.43 66094737053a 6 days ago 1.2GB
harbor.linuxarchitect.io/pub-images/tomcat-base v8.5.43 66094737053a 6 days ago 1.2GB
zezehu.cloud/pub-images/jdk-base v8.212 5198cb715eb5 6 days ago 1.17GB
zezehu.cloud/baseimages/magedu-centos-base 7.9.2009 37bb3db69be0 6 days ago 766MB
ubuntu 22.04 e34e831650c1 10 days ago 77.9MB
tomcat latest f1fd4dbf53f6 12 days ago 454MB
mysql latest 73246731c4b0 4 weeks ago 619MB
alpine 3.16.2 9c6f07244728 17 months ago 5.54MB
zezehu.cloud/myserver/alpine 3.16.2 9c6f07244728 17 months ago 5.54MB
centos 7.9.2009 eeb6ee3f44bd 2 years ago 204MB
root@docker1:/data/nginx#
- 创建并运行nginx容器
root@docker1:/data/nginx# docker run -it zezehu.cloud/myserver/nginx:20240122 bash
root@a37e8420d5a0:/# cd /apps/
root@a37e8420d5a0:/apps# ls
nginx
root@a37e8420d5a0:/apps#
4、部署单机harbor并实现镜像的上传与下载
- 创建新数据盘 /data/harbor 用于harbor仓库
root@harbor1:~# df -hT
Filesystem Type Size Used Avail Use% Mounted on
tmpfs tmpfs 388M 1.5M 387M 1% /run
/dev/sda2 ext4 20G 5.8G 13G 32% /
tmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda3 ext4 2.0G 129M 1.7G 8% /boot
/dev/sda5 ext4 3.9G 56K 3.7G 1% /home
tmpfs tmpfs 388M 4.0K 388M 1% /run/user/0
/dev/sdb1 ext4 1007G 28K 956G 1% /data/harbor
-
安装docker #略
-
下载harbor包文件 #https://github.com/goharbor/harbor
- 修改配置文件harbor.yml
root@harbor1:~# mkdir /apps
root@harbor1:~# cd /apps/
root@harbor1:/apps# tar xvf harbor-offline-installer-v2.6.1.tgz
root@harbor1:/apps# cd harbor/
#修改harbor配置文件
root@harbor1:/apps# cp harbor.yml.tmpl harbor.yml
root@harbor1:/apps/harbor# vim harbor.yml
#注释掉https
# https related config
#https:
# https port for harbor, default is 443
# port: 443
# The path of cert and key files for nginx
# certificate: /your/certificate/path
# private_key: /your/private/key/path
#修改域名
hostname: zezehu.cloud
#修改密码
harbor_admin_password: 12345678
- 部署harbor
#安装脚本
root@harbor1:/app/harbor# ./install.sh --with-trivy
[Step 0]: checking if docker is installed ...
Note: docker version: 24.0.2
[Step 1]: checking docker-compose is installed ...
Note: docker-compose version: 1.28.6
[Step 2]: loading Harbor images ...
#docker主机配置域名解析和docker免认证
root@docker1:~# vim /etc/hosts
192.168.10.15 zezehu.cloud
root@docker1:~# vim /etc/docker/daemon.json
"insecure-registries": ["zezehu.cloud","www.zezehu.cloud",,"192.168.10.15"], #加上这条配置
root@docker1:~# systemctl restart docker #重启docker
root@docker1:~# docker login zezehu.cloud #docker登录harbor
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
root@docker1:~# docker push zezehu.cloud/myserver/alpine:3.16.2 #上传镜像
The push refers to repository [zezehu.cloud/myserver/alpine]
994393dc58e7: Pushed
3.16.2: digest: sha256:1304f174557314a7ed9eddb4eab12fed12cb0cd9809e4c28f29af86979a3c870 size: 528
#docker登录harbor成功后会在家目录生成以下密钥文件
root@docker1:~# cat ~/.docker/config.json
{
"auths": {
"zezehu.cloud": {
"auth": "YWRtaW46MTIzNDU2Nzg="
}
}
}
root@docker1:~#
5.基于systemd实现容器的CPU及内存的使用限制
- 实现原理:docker/kubelet使用systemd去调用cgroups对容器进行资源限制
root@docker1:/etc/docker# vim daemon.json
"exec-opts": ["native.cgroupdriver=systemd"],
- docker实现资源限制参数:
-m
--cpus
- 启动容器
root@docker1:/etc/docker# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root@docker1:/etc/docker# docker run -it --rm -m 256m --cpus 2 centos:7.9.2009
[root@9037eb7916e0 /]# free -h
total used free shared buff/cache available
Mem: 3.8G 423M 358M 752K 3.0G 3.1G
Swap: 4.0G 2.5M 4.0G
[root@9037eb7916e0 /]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
Stepping: 1
CPU MHz: 2394.453
BogoMIPS: 4788.90
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 35840K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid rdseed adx smap xsaveopt arat md_clear flush_l1d arch_capabilities
[root@9037eb7916e0 /]#
由于容器⾥⾯的/proc⽂件系统是宿主机的,在容器内使用free、lscpu等命令查看资源是不准确,使用LXCFS可以解决该问题
扩展:
1.总结镜像的分层构建流程
原理:
- 一个镜像是有一层或者多层合并而成,每一层称为是一个layer
-
镜像可以基于其它镜像进行重新构建,被引用的镜像称为父镜像
在使用Dockerfile构建镜像时,通常会把经常变动包文件/配置放后面,在有变更时重新构建镜像可以节约时间。
镜像分层构建流程中,通常会把操作系统、JDK、python、go、nginx等基础架构打成镜像,后续业务再根据上述基础镜像构建不同的业务镜像,此类做法有以下优势:
- 镜像复用,节省资源
- 减少重新构建镜像变更操作时间
- 方便管理
2.总结基于lxcfs对容器的内存及CPU的资源限制——略
#安装apt install lxcfs