Dockerfile镜像构建实战

镜像制作规范

先制作最通用的一层，后面的每一个镜像都依赖于之前的父镜像。

分层构建镜像并部署tomcat高可用

tomcat1、nginx1、haproxy1和keepalived1部署在10.0.0.51节点
tomcat2、nginx2、haproxy2和keepalived2部署在10.0.0.52节点
镜像构建主要在51节点执行。

1. 创建基础centos-base镜像

1.1 创建镜像分类目录结构

$ mkdir /opt/dockerfile/{web/{nginx,tomcat,jdk,apache},system/{centos,ubuntu,redhat}} -pv

1.2 编写centos-base镜像dockerfile。

$ cd /opt/dockerfile/system/centos
$ mkdir 7.9 && cd 7.9
$ vim Dockerfile
FROM centos:7.9.2009
LABEL author=LiangDong
RUN rpm -ivh http://mirrors.aliyun.com/epel/epel-release-latest-7.noarch.rpm \
        && yum update -y \
        && yum install -y vim wget tree lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop \
        && groupadd -g 2021 maple \
       && useradd -u 2021 -g 2021 maple

1.3 将构建命令写到sh里，方便后续打镜像使用。

$ cat build-command.sh 
#!/bin/bash
docker build -t harbor.maple.com/oss/centos-base:v7.9.2009 .

1.4 构建镜像并测试

$ bash build-command.sh
$ docker run -it --rm  harbor.maple.com/oss/centos-base:v7.9.2009 bash
# 查看是否创建了用户，是否有vim等命令

2.从centos-base镜像基础上构建jdk8镜像

2.1 上传jdk8u291包

$ cd /opt/dockerfile/web/jdk
$ mkdir jdk8u291 && cd jdk8u291
# 上传jdk8u291包
$ ls
jdk-8u291-linux-x64.tar.gz

2.2 编写jdk Dockerfile

$ vim Dockerfile 
FROM harbor.maple.com/oss/centos-base:v7.9.2009
MAINTAINER "LiangDong 395539184@qq.com"
ADD jdk-8u291-linux-x64.tar.gz /usr/local/src/
RUN ln -s /usr/local/src/jdk1.8.0_291 /usr/local/jdk \
    && echo "export JAVA_HOME=/usr/local/jdk" > /etc/profile.d/jdk.sh \
    && echo "export JRE_HOME=\$JAVA_HOME/jre" >> /etc/profile.d/jdk.sh \
    && echo "export CLASSPATH=\$JAVA_HOME/lib/:\$JRE_HOME/lib/" >> /etc/profile.d/jdk.sh \
    && echo "export PATH=\$PATH:\$JAVA_HOME/bin" >> /etc/profile.d/jdk.sh \
    && source /etc/profile.d/jdk.sh \
    && rm -rf /etc/localtime \
    && ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

2.3 构建并测试运行

$ ls
build-command.sh  Dockerfile  jdk-8u291-linux-x64.tar.gz
$ bash build-command.sh
$ docker run -it --rm  harbor.maple.com/oss/centos-base-jdk:v1.8.291 bash
# 测试date是否是和本地时区时间一致
# 测试root用户是否有java环境变量，可以执行java命令
# 测试maple用户是否有java环境变量，可以执行java命令

3.从jdk8镜像基础上构建tomcat8镜像

3.1 上传tomcat包

$ cd /opt/dockerfile/web/tomcat/8.5.64
$ ls
apache-tomcat-8.5.64.tar.gz  build-command.sh  Dockerfile

3.2 创建tomcat的Dockerfile文件

$ cat Dockerfile 
FROM harbor.maple.com/oss/centos-base-jdk:v1.8.291
LABEL author="LiangDong 395539184@qq.com"
ADD apache-tomcat-8.5.64.tar.gz /apps/
RUN ln -s /apps/apache-tomcat-8.5.64 /apps/tomcat

3.3 构建并测试

$ bash build-command.sh
$ docker run -it --rm -p 8080:8080 harbor.maple.com/oss/tomcat:v8.5.64 bash
# /apps/tomcat/bin/catalina.sh start 
# 页面访问8080测试

4. 基于tomcat8镜像构建业务镜像1

4.1 准备配置文件和测试页面文件

$ cd /opt/dockerfile/web/tomcat/myapp1
$ mkdir myapp1
# 准备页面测试文件
$ echo "<h1>tomcat myapp1</h1>" > myapp1/index.jsp  
$  tar cvf myapp1.tar.gz myapp1/
$  rm -rf myapp1
# 准备tomcat配置文件，修改tomcat项目目录
$ docker run -it --rm -p 8080:8080 harbor.maple.com/oss/tomcat:v8.5.64 bash
# 另一个终端执行复制
docker cp 9070a1b76dce:/apps/tomcat/conf/server.xml .

# 修改tomcat项目部署目录为/data/tomcat/webapps
# 修改不自动解压，不自动部署，容易造攻击。
vim server.xml
<Host name="localhost"  appBase="/data/tomcat/webapps" unpackWARs="false" autoDeploy="false">

4.2 准备tomcat运行脚本，使tomcat服务不是容器PID为1的进程。

$ vim run_tomcat.sh
#!/bin/bash
echo "nameserver 223.5.5.5" > /etc/resolv.conf
# 避免以root身份运行tomcat
su - maple -c "/apps/tomcat/bin/catalina.sh start"
su - maple -c "tail -f /etc/hosts"

$ chmod +x run_tomcat.sh #需要有执行权限，不然容器内无法执行该脚本

4.3 编写myapp1的Dockerfile文件

$ vim Dockerfile 
FROM harbor.maple.com/oss/tomcat:v8.5.64
ENV TOMCAT_HOME /apps/tomcat
COPY server.xml $TOMCAT_HOME/conf/server.xml
COPY run_tomcat.sh $TOMCAT_HOME/bin/
ADD myapp1.tar.gz /data/tomcat/webapps/
RUN chown -R maple.maple $TOMCAT_HOME/ \
    && chown -R maple.maple /data/tomcat
CMD ["/bin/bash","-c","$TOMCAT_HOME/bin/run_tomcat.sh"]

注意：

	# 下面这个chown命令修改失败，但是修改/data/tomcat还是没问题，很奇怪
	chown -R maple.maple $TOMCAT_HOME
	#添加一个/可以修改权限成功
	chown -R maple.maple $TOMCAT_HOME/

4.4 构建并测试

$ bash build-command.sh 
$ docker run -it --rm -p 8080:8080 harbor.maple.com/oss/tomcat-myapp1:v1

5. 创建业务镜像2

$ cd /opt/dockerfile/web/tomcat
$ cp myapp1 -r myapp2
$ cd myapp2
# 修改构建命令
$ vim build-command.sh
docker build -t harbor.maple.com/oss/tomcat-myapp2:v1 .
# 修改测试页面内容
$ tar xf myapp1.tar.gz
$ vim myapp1/index.jsp
<h1>tomcat myapp2</h1>
$ tar cf myapp1.tar.gz myapp1
$ rm -rf myapp1

# 构建
$ bash build-command.sh 
# 运行测试
$ docker run -it --rm -p 8080:8080 harbor.maple.com/oss/tomcat-myapp2:v1

导入到另一台docker主机

$ docker save harbor.maple.com/oss/tomcat-myapp2 > myapp2.tar
$ scp myapp2.tar 10.0.0.52:/home/maple/

另一台主机加载并运行

$ docker load -i myapp2.tar 
$ docker run -it --rm -p 8080:8080 harbor.maple.com/oss/tomcat-myapp2:v1

6、创建nginx镜像

6.1 先构建一个测试nginx，获取nginx.conf配置文件

$ cd /opt/dockerfile/web/nginx
$ ls
Dockerfile  index.html  nginx-1.20.1.tar.gz 

$ vim Dockerfile
FROM harbor.maple.com/oss/centos-base:v7.9.2009
MAINTAINER liangsircc@gmail.com
ADD nginx-1.20.1.tar.gz /usr/local/src
RUN yum install -y make vim wget tree lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop \
    && cd /usr/local/src/nginx-1.20.1 \
    && ./configure --prefix=/usr/local/nginx --with-http_sub_module \
    && make \
    && make install
COPY index.html /usr/local/nginx/html/index.html
RUN useradd -s /sbin/nologin nginx \
    && ln -s /usr/local/nginx/sbin/nginx /usr/sbin/nginx
CMD ["nginx","-g","daemon off;"]

$ docker cp 6b8e2d2a5f19:/usr/local/nginx/conf/nginx.conf .
$ ls
Dockerfile  index.html  nginx-1.20.1.tar.gz  nginx.conf

6.2 修改配置文件

$ vim nginx.conf
user  maple maple;
http {
    up_stream myapp1 {
        server 10.0.0.51:8080;
        server 10.0.0.52:8080;
    }
    server {
         location / {
            root   html;
            index  index.html index.htm;
        }
        location /myapp1 {
            proxy_pass http://myapp1;
        }
    }
}

6.3 修改Dockerfile文件增加nginx配置文件

$ vim Dockerfile 
FROM harbor.maple.com/oss/centos-base:v7.9.2009
MAINTAINER liangsircc@gmail.com
ADD nginx-1.20.1.tar.gz /usr/local/src
RUN yum install -y make vim wget tree lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop \
    && cd /usr/local/src/nginx-1.20.1 \
    && ./configure --prefix=/usr/local/nginx --with-http_sub_module \
    && make \
    && make install
COPY index.html /usr/local/nginx/html/index.html
ADD nginx.conf /usr/local/nginx/conf/
RUN ln -s /usr/local/nginx/sbin/nginx /usr/sbin/nginx
CMD ["nginx","-g","daemon off;"]

6.4 构建运行测试

$ bash build-command.sh
$ docker run  -p 8888:80 -d  harbor.maple.com/oss/nginx:v1

可以看到work进程是普通用户启动的

7. 拷贝镜像到另一台主机运行nginx2

$ docker save harbor.maple.com/oss/nginx:v1 > nginx.tar
$ scp nginx.tar 10.0.0.52:/home/maple/

#另一节点加载运行
$ docker load -i nginx.tar 
$ docker run  -p 8888:80 -d  harbor.maple.com/oss/nginx:v1

测试

8. 配置haproxy镜像1

8.1 先准备配置文件

$ cd /opt/dockerfile/web/haproxy
$ ls
build-command.sh  Dockerfile  haproxy-2.4.1.tar.gz  haproxy.cfg  run_haproxy.sh

8.2 准备haproxy配置文件

$ cat haproxy.cfg 
global
        chroot /usr/local/haproxy
        stats socket /var/lib/haproxy/haproxy.sock mode 600 level admin
        uid 99
        gid 99
        daemon
        nbproc 1
        pidfile /usr/local/haproxy/run/haproxy.pid
        log 127.0.0.1 local3 info
        defaults
        option http-keep-alive
        option forwardfor
        mode http
        timeout connect 300000ms
        timeout client 300000ms
        timeout server 300000ms
listen stats
        mode http
        bind 0.0.0.0:9999
        stats enable
        log global
        stats uri /haproxy-status
        stats auth haadmin:123456
listen web_port
        bind 0.0.0.0:80
        mode http
        log global
        balance roundrobin
        server web1 10.0.0.51:8888 check inter 3000 fall 2 rise 5
        server web2 10.0.0.52:8888 check inter 3000 fall 2 rise 5

8.3 准备run_haproxy.sh脚本

$ cat run_haproxy.sh 
#!/bin/bash

/usr/local/haproxy/sbin/haproxy -f /etc/haproxy/haproxy.cfg

tail -f /etc/hosts

$ chmod +x run_haproxy.sh

8.4 dockerfile文件

$ cat Dockerfile 
FROM harbor.maple.com/oss/centos-base:v7.9.2009
ADD haproxy-2.4.1.tar.gz /usr/local/src/
ADD haproxy.cfg /etc/haproxy/
ADD run_haproxy.sh /usr/bin/
RUN  yum install -y yum install gcc gcc-c++ glibc glibc-devel pcre pcre-devel openssl openssl-devel systemd-devel net-tools vim iotop bc zip unzip zlib-devel lrzsz tree screen lsof tcpdump wget ntpdate \
        && cd /usr/local/src/haproxy-2.4.1 \
        && make ARCH=x86_64 TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 \
           USE_SYSTEMD=1 USE_CPU_AFFINITY=1 PREFIX=/usr/local/haproxy \
        && make install PREFIX=/usr/local/haproxy \
        && cp haproxy /usr/sbin/ \
        && mkdir /usr/local/haproxy/run \
        && mkdir /var/lib/haproxy
EXPOSE 80 9999
CMD ["/usr/bin/run_haproxy.sh"]

8.5 构建运行测试

$ docker run -d -p 80:80 -p 9999:9999 harbor.maple.com/oss/haproxy:v1

9. 配置haproxy镜像2

$ docker save harbor.maple.com/oss/haproxy:v1 > haproxy.tar
$ scp haproxy.tar 10.0.0.52:/home/maple/
#另一台主机加载并启动容器
$ docker load -i haproxy.tar
$ docker run -d -p 80:80 -p 9999:9999 harbor.maple.com/oss/haproxy:v1  

10、配置keepalived

两种方式

• 宿主机运行keepalived
• 容器运行keepalived，需要配置–network=host

10.1 宿主机运行keepalived

10.1.1 安装

$ sudo apt install -y keepalived

#创建配置文件
$ cat /etc/keepalived/keepalived.conf 
global_defs {
   notification_email {
     liangsircc@gmail.com
   }
   notification_email_from 395539184@qq.com
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka1.maple.com
   vrrp_skip_check_adv_addr
  # vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
vrrp_script check_haproxy {
        script "/etc/keepalived/check_haproxy.sh"
        interval 1
        weight -40
        fall 3
        rise 2
        timeout 2
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 66
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass admin
    }
    virtual_ipaddress {
        10.0.0.50 dev eth0 label eth0:1
    }
    unicast_src_ip 10.0.0.51
    unicast_peer {
       10.0.0.52
    }
    track_script {
        check_haproxy
    }
}

#创建检测脚本
$ sudo vim /etc/keepalived/check_haproxy.sh
#!/bin/bash
if ss -tln | grep -E "\<80\>" &> /dev/null ;then
        exit 0;
else 
        exit 1;
fi

#启动
sudo systemctl start keepalived.service

10.1.2 另一台主机

$ apt install -y keepalived
$ cat /etc/keepalived/keepalived.conf 
global_defs {
   notification_email {
     liangsircc@gmail.com
   }
   notification_email_from 395539184@qq.com
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka1.maple.com
   vrrp_skip_check_adv_addr
  # vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 66
    priority 70
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass admin
    }
    virtual_ipaddress {
        10.0.0.50 dev eth0 label eth0:1
    }
    unicast_src_ip 10.0.0.52
    unicast_peer {
       10.0.0.51
    }
}

#启动
$ sudo systemctl start keepalived.service 

不用修改haproxy，
因为haproxy配置监听的是0.0.0.0 包含了vip地址

测试停止haproxy容器
docker stop 07e3d40467c0

观测vip是否切到另一主机 10.0.0.52
网站是否正常访问

10.2 容器运行keepalived

10.2.1 准备包和配置文件 第一台主机

$ sudo systemctl stop keepalived.service #先停掉宿主机的keepalived
$ cd /opt/dockerfile/web/keepalived
$ sudo cp /etc/keepalived/* .

# 构建命令
$ cat build-command.sh
#!/bin/bash
docker build -t harbor.maple.com/oss/keepalived:v1 .

#检测脚本
$ cat check_haproxy.sh 
#!/bin/bash
if ss -tln | grep -E "\<80\>" &> /dev/null ;then
        exit 0;
else 
        exit 1;
fi

$ ls
build-command.sh  check_haproxy.sh  Dockerfile  keepalived.conf

10.2.2 构建dockerfile

$ cat Dockerfile 
FROM harbor.maple.com/oss/centos-base:v7.9.2009
RUN yum install -y keepalived
ADD check_haproxy.sh /etc/keepalived/
ADD keepalived.conf /etc/keepalived/
CMD keepalived -f /etc/keepalived/keepalived.conf -l -n -D --vrrp

10.2.3 构建并启动容器

$ bash build-command.sh
# 需要增加--privileged 使容器能够有权限操作宿主机
# 需要--network=host 使容器和宿主机共享使用同一个网卡等
$ docker run -d --privileged --network=host  harbor.maple.com/oss/keepalived:v1 


# VIP 50已经配置。
$ hostname -I
10.0.0.51 10.0.0.50 172.22.16.1 172.18.0.1 172.17.0.1 

10.2.4 创建ka2新的目录修改配置构建镜像

10.2.4.1 准备配置文件

$ cd /opt/dockerfile/web/keepalived-backup
$ ls
build-command.sh  Dockerfile  keepalived.conf

10.2.4.2 Dockerfile

$ cat Dockerfile 
FROM harbor.maple.com/oss/centos-base:v7.9.2009
RUN yum install -y keepalived
ADD keepalived.conf /etc/keepalived/
CMD keepalived -f /etc/keepalived/keepalived.conf -l -n -D --vrrp

10.2.4.3 修改keepalived.conf配置文件

$ cat keepalived.conf 
global_defs {
   notification_email {
     liangsircc@gmail.com
   }
   notification_email_from 395539184@qq.com
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka1.maple.com
   vrrp_skip_check_adv_addr
  # vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 66
    priority 70
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass admin
    }
    virtual_ipaddress {
        10.0.0.50 dev eth0 label eth0:1
    }
    unicast_src_ip 10.0.0.52
    unicast_peer {
       10.0.0.51
    }
}

10.2.4.4 导出到另一台主机

$ docker save harbor.maple.com/oss/keepalived-backup:v1 > keepalived-backup.tar
$ scp keepalived-backup.tar 10.0.0.52:/home/maple/

10.2.4.5 加载并启动

$ docker load -i keepalived-backup.tar 
$ docker run -d --privileged --network=host  harbor.maple.com/oss/keepalived-backup:v1 

测试 10.0.0.51 主节点停掉haproxy容器，vip自动切到 10.0.0.52节点，网站正常访问。