docker-compose方式部署高可用多节点keepalived+nginx集群

彼岸花@开

已于 2023-11-01 15:40:43 修改

阅读量1.1k

点赞数 1

分类专栏： linux 文章标签： docker nginx 云原生

于 2023-08-21 15:14:23 首次发布

本文链接：https://blog.csdn.net/weixin_42211693/article/details/132407267

版权

linux 专栏收录该内容

3 篇文章 0 订阅

订阅专栏

1.前言

一次构建，随处运行，这是docker的理念

无论你是Ubuntu，还是centos8，还是麒麟，还是centos6，这些都不重要，重要的是我们需要一个docker的环境。有了docker的运行环境，那么docker的理念就能体现下去

keepalived+nginx高负载均衡集群是常见的组合，是一个标准负载均衡方式。

在docker环境下可以很方便的构建统一镜像，运行服务，实现keepalived多节点的高可用，nginx的负载均衡和反向代理功能。

其中keepalived轮询检查nginx是否存活，动态切换虚拟IP地址

2.构建镜像

2.1.配置nginx下载源

新建nginx.repo

[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key

[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key

2.2.生成nginx检测脚本

新建check_nginx.sh

#!/bin/bash
A=$(ps -ef  | grep nginx: | grep -v  grep |  wc  -l)
if [ $A -eq 0 ];then
  nginx
  echo  "restart nginx, sleep 2 s"
  sleep  2
  num=$(ps -ef  | grep  nginx:  |  grep -v grep | wc  -l)
  if [ $num -eq  0  ];then
    ps -ef | grep keepalived  |  grep  -v grep  | awk '{print $2}'| xargs kill -9
    echo  "start nginx failed,kill keepalived"
  fi
else
  echo  "nginx not  dead"
fi

2.3.生成Dockfile文件

新建Dockerfile

适当的优化，RUN命令通过&&合并了，并在最后清除缓存层。

避免build的镜像过大，降低build的效率

FROM centos:7
ENV TZ "Asia/Shanghai"
ADD nginx.repo /etc/yum.repos.d/
ADD check_nginx.sh /etc/keepalived/
RUN chmod +x /etc/keepalived/check_nginx.sh && \
    yum install -y iproute && \
    yum install -y net-tools && \
    yum install -y yum-utils && \
    yum install -y keepalived && \
    yum install -y nginx && \
    yum clean all

2.4.构建命令

docker build -t centos-base:7 .

3.编排说明

3.1.功能须知

build的镜像由于主要的用途是跑keepalived服务

keepalived服务用来定时检测nginx进程是否存活

keepalived配置文件中需要多个节点共享虚拟IP（虚拟IP自定义即可，但是需要在当前节点同一网段下）

3.2.各个节点keepalived配置文件修改

虚拟IP地址是192.168.217.100/16 ，跟节点主网卡同在217网点上即可

3.2.1.keepalived master的配置文件（主节点）

查看当前节点网卡，可以看到，主网卡是ens33，IP地址是192.168.217.23

[root@node3 media]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:0c:29:70:12:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.217.23/24 brd 192.168.217.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe70:1212/64 scope link 
       valid_lft forever preferred_lft forever

keepalived的配置文件是keepalived.conf

其中ens33是当前节点的主网卡，修改内容如下（加粗部分是修改项）：

! Configuration File for keepalived

global_defs {
router_id LVS_Master

#vrrp_strict #需要注释掉vrrp_strict
}

vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight -5
fall 3
rise 2
}

vrrp_instance VI_1 {
state MASTER #指定instance初始状态，实际根据优先级决定.backup节点不一样
interface ens33 #虚拟IP所在网
virtual_router_id 51 #VRID，相同VRID为一个组，决定多播MAC地址
priority 100 #优先级，另一台改为90.backup节点不一样
advert_int 1 #检查间隔
authentication {
auth_type PASS #认证方式，可以是pass或ha
auth_pass 1111 #认证密码
}
virtual_ipaddress {
192.168.217.100/16 #VIP地址
}

track_script {
check_nginx
}

}

3.2.2.keepalived backup的配置文件（备节点）

查看当前节点网卡，可以看到，主网卡是enp2s0f0，IP地址是192.168.217.24

[root@node4 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 00:0c:29:5b:a4:eb brd ff:ff:ff:ff:ff:ff
inet 192.168.217.24/24 brd 192.168.217.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe5b:a4eb/64 scope link
valid_lft forever preferred_lft forever

keepalived的配置文件是keepalived.conf

其中enp2s0f0是当前节点的主网卡，修改内容如下（加粗部分是修改项）：

! Configuration File for keepalived

global_defs {
router_id LVS_Backup
#vrrp_strict
}

vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight -5
fall 3
rise 2
}

vrrp_instance VI_1 {
state BACKUP
interface enp2s0f0
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.90.100/16
}
track_script {
check_nginx
}
}

4.docker-compose编排文件

keepalived服务依赖宿主机网卡，需要docker中hosts网络方式

在每个节点上

新建docker-compose.yml文件

version: '3'

services:
  kn-master:
    image: centos-base:7
    container_name: kn-master
    privileged: true
    network_mode: host
    cap_add:
      - NET_ADMIN
    environment:
      - TZ=Asia/Shanghai
    volumes:
      - ./keepalived.conf:/etc/keepalived/keepalived.conf
    command: ["/usr/sbin/keepalived", "-n", "-f", "/etc/keepalived/keepalived.conf"]

如果需要日志信息的话，可以用

command: ["/usr/sbin/keepalived", "-n","--all", "-d", "-D", "-f", "/etc/keepalived/keepalived.conf", "--log-console"]

启动和停止的命令

docker-compose up -d
docker-compose down

5.测试

启动主节点和备节点docker-compose服务后

5.1.测试虚拟IP存在

查看主节点的VIP（虚拟IP），可以看到，有VIP了

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 00:0c:29:70:12:12 brd ff:ff:ff:ff:ff:ff
inet 192.168.217.23/24 brd 192.168.217.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.217.100/16 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe70:1212/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:55:fc:14:b8 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:55ff:fefc:14b8/64 scope link
valid_lft forever preferred_lft forever

查看备节点的VIP（虚拟IP），可以看到，没有VIP

[root@node4 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 00:0c:29:5b:a4:eb brd ff:ff:ff:ff:ff:ff
inet 192.168.217.24/24 brd 192.168.217.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe5b:a4eb/64 scope link
valid_lft forever preferred_lft forever

5.2.测试虚拟IP是否漂移和nginx服务是否高可用

停掉keepalived master节点

docker-compose down

看备节点的VIP（虚拟IP），可以看到虚拟IP发生了漂移

[root@node4 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:0c:29:5b:a4:eb brd ff:ff:ff:ff:ff:ff
    inet 192.168.217.24/24 brd 192.168.217.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.217.100/16 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe5b:a4eb/64 scope link 
       valid_lft forever preferred_lft forever

同时nginx服务正常，访问不受影响

curl http://192.168.217.100

彼岸花@开

关注

1
点赞
踩
2

收藏

觉得还不错? 一键收藏
打赏
0
评论
docker-compose方式部署高可用多节点keepalived+nginx集群

keepalived+nginx高负载均衡集群是常见的组合，是一个标准负载均衡方式。在docker环境下可以很方便的构建统一镜像，运行服务，实现keepalived多节点的高可用，nginx的负载均衡和反向代理功能。
复制链接

扫一扫