镜像制作规范
先制作最通用的一层,后面的每一个镜像都依赖于之前的父镜像。
分层构建镜像并部署tomcat高可用
tomcat1、nginx1、haproxy1和keepalived1部署在10.0.0.51节点
tomcat2、nginx2、haproxy2和keepalived2部署在10.0.0.52节点
镜像构建主要在51节点执行。
1. 创建基础centos-base镜像
1.1 创建镜像分类目录结构
$ mkdir /opt/dockerfile/{web/{nginx,tomcat,jdk,apache},system/{centos,ubuntu,redhat}} -pv
1.2 编写centos-base镜像dockerfile。
$ cd /opt/dockerfile/system/centos
$ mkdir 7.9 && cd 7.9
$ vim Dockerfile
FROM centos:7.9.2009
LABEL author=LiangDong
RUN rpm -ivh http://mirrors.aliyun.com/epel/epel-release-latest-7.noarch.rpm \
&& yum update -y \
&& yum install -y vim wget tree lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop \
&& groupadd -g 2021 maple \
&& useradd -u 2021 -g 2021 maple
1.3 将构建命令写到sh里,方便后续打镜像使用。
$ cat build-command.sh
#!/bin/bash
docker build -t harbor.maple.com/oss/centos-base:v7.9.2009 .
1.4 构建镜像并测试
$ bash build-command.sh
$ docker run -it --rm harbor.maple.com/oss/centos-base:v7.9.2009 bash
# 查看是否创建了用户,是否有vim等命令
2.从centos-base镜像基础上构建jdk8镜像
2.1 上传jdk8u291包
$ cd /opt/dockerfile/web/jdk
$ mkdir jdk8u291 && cd jdk8u291
# 上传jdk8u291包
$ ls
jdk-8u291-linux-x64.tar.gz
2.2 编写jdk Dockerfile
$ vim Dockerfile
FROM harbor.maple.com/oss/centos-base:v7.9.2009
MAINTAINER "LiangDong 395539184@qq.com"
ADD jdk-8u291-linux-x64.tar.gz /usr/local/src/
RUN ln -s /usr/local/src/jdk1.8.0_291 /usr/local/jdk \
&& echo "export JAVA_HOME=/usr/local/jdk" > /etc/profile.d/jdk.sh \
&& echo "export JRE_HOME=\$JAVA_HOME/jre" >> /etc/profile.d/jdk.sh \
&& echo "export CLASSPATH=\$JAVA_HOME/lib/:\$JRE_HOME/lib/" >> /etc/profile.d/jdk.sh \
&& echo "export PATH=\$PATH:\$JAVA_HOME/bin" >> /etc/profile.d/jdk.sh \
&& source /etc/profile.d/jdk.sh \
&& rm -rf /etc/localtime \
&& ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
2.3 构建并测试运行
$ ls
build-command.sh Dockerfile jdk-8u291-linux-x64.tar.gz
$ bash build-command.sh
$ docker run -it --rm harbor.maple.com/oss/centos-base-jdk:v1.8.291 bash
# 测试date是否是和本地时区时间一致
# 测试root用户是否有java环境变量,可以执行java命令
# 测试maple用户是否有java环境变量,可以执行java命令
3.从jdk8镜像基础上构建tomcat8镜像
3.1 上传tomcat包
$ cd /opt/dockerfile/web/tomcat/8.5.64
$ ls
apache-tomcat-8.5.64.tar.gz build-command.sh Dockerfile
3.2 创建tomcat的Dockerfile文件
$ cat Dockerfile
FROM harbor.maple.com/oss/centos-base-jdk:v1.8.291
LABEL author="LiangDong 395539184@qq.com"
ADD apache-tomcat-8.5.64.tar.gz /apps/
RUN ln -s /apps/apache-tomcat-8.5.64 /apps/tomcat
3.3 构建并测试
$ bash build-command.sh
$ docker run -it --rm -p 8080:8080 harbor.maple.com/oss/tomcat:v8.5.64 bash
# /apps/tomcat/bin/catalina.sh start
# 页面访问8080测试
4. 基于tomcat8镜像构建业务镜像1
4.1 准备配置文件和测试页面文件
$ cd /opt/dockerfile/web/tomcat/myapp1
$ mkdir myapp1
# 准备页面测试文件
$ echo "<h1>tomcat myapp1</h1>" > myapp1/index.jsp
$ tar cvf myapp1.tar.gz myapp1/
$ rm -rf myapp1
# 准备tomcat配置文件,修改tomcat项目目录
$ docker run -it --rm -p 8080:8080 harbor.maple.com/oss/tomcat:v8.5.64 bash
# 另一个终端执行复制
docker cp 9070a1b76dce:/apps/tomcat/conf/server.xml .
# 修改tomcat项目部署目录为/data/tomcat/webapps
# 修改不自动解压,不自动部署,容易造攻击。
vim server.xml
<Host name="localhost" appBase="/data/tomcat/webapps" unpackWARs="false" autoDeploy="false">
4.2 准备tomcat运行脚本,使tomcat服务不是容器PID为1的进程。
$ vim run_tomcat.sh
#!/bin/bash
echo "nameserver 223.5.5.5" > /etc/resolv.conf
# 避免以root身份运行tomcat
su - maple -c "/apps/tomcat/bin/catalina.sh start"
su - maple -c "tail -f /etc/hosts"
$ chmod +x run_tomcat.sh #需要有执行权限,不然容器内无法执行该脚本
4.3 编写myapp1的Dockerfile文件
$ vim Dockerfile
FROM harbor.maple.com/oss/tomcat:v8.5.64
ENV TOMCAT_HOME /apps/tomcat
COPY server.xml $TOMCAT_HOME/conf/server.xml
COPY run_tomcat.sh $TOMCAT_HOME/bin/
ADD myapp1.tar.gz /data/tomcat/webapps/
RUN chown -R maple.maple $TOMCAT_HOME/ \
&& chown -R maple.maple /data/tomcat
CMD ["/bin/bash","-c","$TOMCAT_HOME/bin/run_tomcat.sh"]
注意:
# 下面这个chown命令修改失败,但是修改/data/tomcat还是没问题,很奇怪
chown -R maple.maple $TOMCAT_HOME
#添加一个/可以修改权限成功
chown -R maple.maple $TOMCAT_HOME/
4.4 构建并测试
$ bash build-command.sh
$ docker run -it --rm -p 8080:8080 harbor.maple.com/oss/tomcat-myapp1:v1
5. 创建业务镜像2
$ cd /opt/dockerfile/web/tomcat
$ cp myapp1 -r myapp2
$ cd myapp2
# 修改构建命令
$ vim build-command.sh
docker build -t harbor.maple.com/oss/tomcat-myapp2:v1 .
# 修改测试页面内容
$ tar xf myapp1.tar.gz
$ vim myapp1/index.jsp
<h1>tomcat myapp2</h1>
$ tar cf myapp1.tar.gz myapp1
$ rm -rf myapp1
# 构建
$ bash build-command.sh
# 运行测试
$ docker run -it --rm -p 8080:8080 harbor.maple.com/oss/tomcat-myapp2:v1
导入到另一台docker主机
$ docker save harbor.maple.com/oss/tomcat-myapp2 > myapp2.tar
$ scp myapp2.tar 10.0.0.52:/home/maple/
另一台主机加载并运行
$ docker load -i myapp2.tar
$ docker run -it --rm -p 8080:8080 harbor.maple.com/oss/tomcat-myapp2:v1
6、创建nginx镜像
6.1 先构建一个测试nginx,获取nginx.conf配置文件
$ cd /opt/dockerfile/web/nginx
$ ls
Dockerfile index.html nginx-1.20.1.tar.gz
$ vim Dockerfile
FROM harbor.maple.com/oss/centos-base:v7.9.2009
MAINTAINER liangsircc@gmail.com
ADD nginx-1.20.1.tar.gz /usr/local/src
RUN yum install -y make vim wget tree lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop \
&& cd /usr/local/src/nginx-1.20.1 \
&& ./configure --prefix=/usr/local/nginx --with-http_sub_module \
&& make \
&& make install
COPY index.html /usr/local/nginx/html/index.html
RUN useradd -s /sbin/nologin nginx \
&& ln -s /usr/local/nginx/sbin/nginx /usr/sbin/nginx
CMD ["nginx","-g","daemon off;"]
$ docker cp 6b8e2d2a5f19:/usr/local/nginx/conf/nginx.conf .
$ ls
Dockerfile index.html nginx-1.20.1.tar.gz nginx.conf
6.2 修改配置文件
$ vim nginx.conf
user maple maple;
http {
up_stream myapp1 {
server 10.0.0.51:8080;
server 10.0.0.52:8080;
}
server {
location / {
root html;
index index.html index.htm;
}
location /myapp1 {
proxy_pass http://myapp1;
}
}
}
6.3 修改Dockerfile文件增加nginx配置文件
$ vim Dockerfile
FROM harbor.maple.com/oss/centos-base:v7.9.2009
MAINTAINER liangsircc@gmail.com
ADD nginx-1.20.1.tar.gz /usr/local/src
RUN yum install -y make vim wget tree lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop \
&& cd /usr/local/src/nginx-1.20.1 \
&& ./configure --prefix=/usr/local/nginx --with-http_sub_module \
&& make \
&& make install
COPY index.html /usr/local/nginx/html/index.html
ADD nginx.conf /usr/local/nginx/conf/
RUN ln -s /usr/local/nginx/sbin/nginx /usr/sbin/nginx
CMD ["nginx","-g","daemon off;"]
6.4 构建运行测试
$ bash build-command.sh
$ docker run -p 8888:80 -d harbor.maple.com/oss/nginx:v1
可以看到work进程是普通用户启动的
7. 拷贝镜像到另一台主机运行nginx2
$ docker save harbor.maple.com/oss/nginx:v1 > nginx.tar
$ scp nginx.tar 10.0.0.52:/home/maple/
#另一节点加载运行
$ docker load -i nginx.tar
$ docker run -p 8888:80 -d harbor.maple.com/oss/nginx:v1
测试
8. 配置haproxy镜像1
8.1 先准备配置文件
$ cd /opt/dockerfile/web/haproxy
$ ls
build-command.sh Dockerfile haproxy-2.4.1.tar.gz haproxy.cfg run_haproxy.sh
8.2 准备haproxy配置文件
$ cat haproxy.cfg
global
chroot /usr/local/haproxy
stats socket /var/lib/haproxy/haproxy.sock mode 600 level admin
uid 99
gid 99
daemon
nbproc 1
pidfile /usr/local/haproxy/run/haproxy.pid
log 127.0.0.1 local3 info
defaults
option http-keep-alive
option forwardfor
mode http
timeout connect 300000ms
timeout client 300000ms
timeout server 300000ms
listen stats
mode http
bind 0.0.0.0:9999
stats enable
log global
stats uri /haproxy-status
stats auth haadmin:123456
listen web_port
bind 0.0.0.0:80
mode http
log global
balance roundrobin
server web1 10.0.0.51:8888 check inter 3000 fall 2 rise 5
server web2 10.0.0.52:8888 check inter 3000 fall 2 rise 5
8.3 准备run_haproxy.sh脚本
$ cat run_haproxy.sh
#!/bin/bash
/usr/local/haproxy/sbin/haproxy -f /etc/haproxy/haproxy.cfg
tail -f /etc/hosts
$ chmod +x run_haproxy.sh
8.4 dockerfile文件
$ cat Dockerfile
FROM harbor.maple.com/oss/centos-base:v7.9.2009
ADD haproxy-2.4.1.tar.gz /usr/local/src/
ADD haproxy.cfg /etc/haproxy/
ADD run_haproxy.sh /usr/bin/
RUN yum install -y yum install gcc gcc-c++ glibc glibc-devel pcre pcre-devel openssl openssl-devel systemd-devel net-tools vim iotop bc zip unzip zlib-devel lrzsz tree screen lsof tcpdump wget ntpdate \
&& cd /usr/local/src/haproxy-2.4.1 \
&& make ARCH=x86_64 TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 \
USE_SYSTEMD=1 USE_CPU_AFFINITY=1 PREFIX=/usr/local/haproxy \
&& make install PREFIX=/usr/local/haproxy \
&& cp haproxy /usr/sbin/ \
&& mkdir /usr/local/haproxy/run \
&& mkdir /var/lib/haproxy
EXPOSE 80 9999
CMD ["/usr/bin/run_haproxy.sh"]
8.5 构建运行测试
$ docker run -d -p 80:80 -p 9999:9999 harbor.maple.com/oss/haproxy:v1
9. 配置haproxy镜像2
$ docker save harbor.maple.com/oss/haproxy:v1 > haproxy.tar
$ scp haproxy.tar 10.0.0.52:/home/maple/
#另一台主机加载并启动容器
$ docker load -i haproxy.tar
$ docker run -d -p 80:80 -p 9999:9999 harbor.maple.com/oss/haproxy:v1
10、配置keepalived
两种方式
- 宿主机运行keepalived
- 容器运行keepalived,需要配置–network=host
10.1 宿主机运行keepalived
10.1.1 安装
$ sudo apt install -y keepalived
#创建配置文件
$ cat /etc/keepalived/keepalived.conf
global_defs {
notification_email {
liangsircc@gmail.com
}
notification_email_from 395539184@qq.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.maple.com
vrrp_skip_check_adv_addr
# vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_script check_haproxy {
script "/etc/keepalived/check_haproxy.sh"
interval 1
weight -40
fall 3
rise 2
timeout 2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 66
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass admin
}
virtual_ipaddress {
10.0.0.50 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.51
unicast_peer {
10.0.0.52
}
track_script {
check_haproxy
}
}
#创建检测脚本
$ sudo vim /etc/keepalived/check_haproxy.sh
#!/bin/bash
if ss -tln | grep -E "\<80\>" &> /dev/null ;then
exit 0;
else
exit 1;
fi
#启动
sudo systemctl start keepalived.service
10.1.2 另一台主机
$ apt install -y keepalived
$ cat /etc/keepalived/keepalived.conf
global_defs {
notification_email {
liangsircc@gmail.com
}
notification_email_from 395539184@qq.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.maple.com
vrrp_skip_check_adv_addr
# vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 66
priority 70
advert_int 1
authentication {
auth_type PASS
auth_pass admin
}
virtual_ipaddress {
10.0.0.50 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.52
unicast_peer {
10.0.0.51
}
}
#启动
$ sudo systemctl start keepalived.service
不用修改haproxy,
因为haproxy配置监听的是0.0.0.0 包含了vip地址
测试停止haproxy容器
docker stop 07e3d40467c0
观测vip是否切到另一主机 10.0.0.52
网站是否正常访问
10.2 容器运行keepalived
10.2.1 准备包和配置文件 第一台主机
$ sudo systemctl stop keepalived.service #先停掉宿主机的keepalived
$ cd /opt/dockerfile/web/keepalived
$ sudo cp /etc/keepalived/* .
# 构建命令
$ cat build-command.sh
#!/bin/bash
docker build -t harbor.maple.com/oss/keepalived:v1 .
#检测脚本
$ cat check_haproxy.sh
#!/bin/bash
if ss -tln | grep -E "\<80\>" &> /dev/null ;then
exit 0;
else
exit 1;
fi
$ ls
build-command.sh check_haproxy.sh Dockerfile keepalived.conf
10.2.2 构建dockerfile
$ cat Dockerfile
FROM harbor.maple.com/oss/centos-base:v7.9.2009
RUN yum install -y keepalived
ADD check_haproxy.sh /etc/keepalived/
ADD keepalived.conf /etc/keepalived/
CMD keepalived -f /etc/keepalived/keepalived.conf -l -n -D --vrrp
10.2.3 构建并启动容器
$ bash build-command.sh
# 需要增加--privileged 使容器能够有权限操作宿主机
# 需要--network=host 使容器和宿主机共享使用同一个网卡等
$ docker run -d --privileged --network=host harbor.maple.com/oss/keepalived:v1
# VIP 50已经配置。
$ hostname -I
10.0.0.51 10.0.0.50 172.22.16.1 172.18.0.1 172.17.0.1
10.2.4 创建ka2新的目录修改配置构建镜像
10.2.4.1 准备配置文件
$ cd /opt/dockerfile/web/keepalived-backup
$ ls
build-command.sh Dockerfile keepalived.conf
10.2.4.2 Dockerfile
$ cat Dockerfile
FROM harbor.maple.com/oss/centos-base:v7.9.2009
RUN yum install -y keepalived
ADD keepalived.conf /etc/keepalived/
CMD keepalived -f /etc/keepalived/keepalived.conf -l -n -D --vrrp
10.2.4.3 修改keepalived.conf配置文件
$ cat keepalived.conf
global_defs {
notification_email {
liangsircc@gmail.com
}
notification_email_from 395539184@qq.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.maple.com
vrrp_skip_check_adv_addr
# vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 66
priority 70
advert_int 1
authentication {
auth_type PASS
auth_pass admin
}
virtual_ipaddress {
10.0.0.50 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.52
unicast_peer {
10.0.0.51
}
}
10.2.4.4 导出到另一台主机
$ docker save harbor.maple.com/oss/keepalived-backup:v1 > keepalived-backup.tar
$ scp keepalived-backup.tar 10.0.0.52:/home/maple/
10.2.4.5 加载并启动
$ docker load -i keepalived-backup.tar
$ docker run -d --privileged --network=host harbor.maple.com/oss/keepalived-backup:v1
测试 10.0.0.51 主节点停掉haproxy容器,vip自动切到 10.0.0.52节点,网站正常访问。