应用nginx+keepalive搭建的高可用高性能web集群

轩苍

已于 2024-04-19 13:09:30 修改

阅读量635

点赞数 13

文章标签： nginx mysql 运维网络服务器 linux ansible

于 2024-04-13 14:09:45 首次发布

本文链接：https://blog.csdn.net/huan_hao/article/details/137610635

版权

项目名称：应用nginx+keepalive搭建的高可用高性能web集群

项目描述：

为满足企业内部web项目需求，模拟构建一个高可用、高性能的web集群。使用nginx作为web服务器，并搭配Prometheus和Grafana构建监控系统，实现集群运行状态的实时监控。同时借助nginx负载均衡与keepalived技术，确保集群的高可用性。并引入ansible实现集群的自动化运维，提高运维效率。

项目架构图：

主机清单

主机名	IP
web1	192.168.2.161
web2	192.168.2.162
LB1	192.168.2.163
LB2	192.168.2.164
table	192.168.16.165
total(dns+ansible+nfs+堡垒机+监控)	192.168.2.166
mysql	192.168.2.167
vip1	192.168.2.199
vip2	192.168.2.188

项目步骤：

一. 部署基础环境，关闭防火墙和selinux，配置好静态ip地址和主机名

配置nfs服务

[root@total ~]# service nfs restart
Redirecting to /bin/systemctl restart nfs.service
[root@total ~]# systemctl enable nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service

在web1、web2上进行同样的配置

设置nfs的共享目录

[root@total ~]# vim /etc/exports
[root@total ~]# cat /etc/exports
/web/html 192.168.2.0/24(rw,sync,all_squash)
[root@total ~]# mkdir -p /web/html #提前建立相应文件夹
[root@total ~]# exportfs -rv  #刷新服务
exporting 192.168.2.0/24:/web/html

设置/etc/exports的权限，允许其他人访问

[root@total ~]# cat /etc/passwd|grep nfs #查看nfs服务自建用户
rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin
nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin
[root@total ~]# cd /web
[root@total web]# chown nfsnobody:nfsnobody html/ #给予该用户访问权限

再次刷新nfs服务

在/web/html上部署html文件

[root@total html]# cat index.html 
<!DOCTYPE html>
<html>
<head>
<title> nfs success </title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1> nfs success </h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using www.</em></p>
</body>
</html>

将nfs挂载到web服务器上（以下步骤可在后面搭建完nginx服务后进行）

[root@web1 ~]# mount  192.168.2.166:/web/html  /usr/local/wangguang/html/

[root@web2 html]# mount  192.168.2.166:/web/html  /usr/local/wangguang/html/www

#查看挂载情况
[root@web1 ~]# df -Th
文件系统                类型      容量  已用  可用 已用% 挂载点
devtmpfs                devtmpfs  1.4G     0  1.4G    0% /dev
tmpfs                   tmpfs     1.4G     0  1.4G    0% /dev/shm
tmpfs                   tmpfs     1.4G  9.6M  1.4G    1% /run
tmpfs                   tmpfs     1.4G     0  1.4G    0% /sys/fs/cgroup
/dev/mapper/centos-root xfs        50G  1.9G   49G    4% /
/dev/sda1               xfs      1014M  151M  864M   15% /boot
/dev/mapper/centos-home xfs        47G   33M   47G    1% /home
tmpfs                   tmpfs     276M     0  276M    0% /run/user/0
192.168.2.166:/web/html nfs4       17G  2.7G   15G   16% /usr/local/wangguang/html

[root@web2 html]# df -Th
文件系统                类型      容量  已用  可用 已用% 挂载点
devtmpfs                devtmpfs  1.4G     0  1.4G    0% /dev
tmpfs                   tmpfs     1.4G     0  1.4G    0% /dev/shm
tmpfs                   tmpfs     1.4G  9.6M  1.4G    1% /run
tmpfs                   tmpfs     1.4G     0  1.4G    0% /sys/fs/cgroup
/dev/mapper/centos-root xfs        50G  1.8G   49G    4% /
/dev/sda1               xfs      1014M  151M  864M   15% /boot
/dev/mapper/centos-home xfs        47G   33M   47G    1% /home
tmpfs                   tmpfs     276M     0  276M    0% /run/user/0

在浏览器进行访问，查看挂载效果

二. 配置ansible服务器，为自动化运维提供便利

部署ansible插件

[root@total web]# yum install epel-release -y
[root@total web]# yum install ansible -y

编辑主机清单

[root@total web]# vim /etc/ansible/hosts
[web]
192.168.2.161
192.168.2.162

[lib]
192.168.2.163
192.168.2.164

[fw]
192.168.16.165
[db]
192.168.2.167

和内部机器建立免密通道

[root@total ~]# ssh-keygen
ssh-copy-id root@192.168.2.161
ssh-copy-id root@192.168.2.162
ssh-copy-id root@192.168.2.163
ssh-copy-id root@192.168.2.164
ssh-copy-id root@192.168.16.165
ssh-copy-id root@192.168.2.167

三. 部署mysql服务器，用于存储web所需要的数据

准备好需要的安装包

编写脚本进行安装

[root@mysql ~]# cat onekey_install_mysql_binary.sh 
#!/bin/bash

#解决软件的依赖关系
yum install cmake ncurses-devel gcc gcc-c++ vim lsof bzip2 openssl-devel ncurses-compat-libs net-tools -y

#解压mysql二进制安装包
tar xf mysql-5.7.38-linux-glibc2.12-x86_64.tar.gz

#移动mysql解压后的文件到/usr/local下改名叫mysql
mv mysql-5.7.38-linux-glibc2.12-x86_64 /usr/local/mysql

#新建组和用户mysql
groupadd mysql
#mysql这个用户的shell是/bin/false属于mysql组
useradd -r -g mysql -s /bin/false mysql

#关闭firewalld防火墙服务，并且设置开机不要启动
service firewalld stop
systemctl disable firewalld

#临时关闭selinux
setenforce 0
#永久关闭selinux
sed -i '/^SELINUX=/ s/enforcing/disabled/' /etc/selinux/config

#新建存放数据的目录
mkdir /data/mysql -p
#修改/data/mysql目录的权限归mysql用户和mysql组所有，这样mysql用户就可以对这个文件夹进行读写了
chown mysql:mysql /data/mysql/
#只是允许mysql这个用户和mysql组可以访问，其他人都不能访问
chmod 750 /data/mysql/

#进入/usr/local/mysql/bin目录
cd /usr/local/mysql/bin/

#初始化mysql
./mysqld --initialize --user=mysql --basedir=/usr/local/mysql/ --datadir=/data/mysql &>passwd.txt

#让mysql支持ssl方式登录的设置
./mysql_ssl_rsa_setup --datadir=/data/mysql/

#获得临时密码
tem_passwd=$(cat passwd.txt |grep "temporary"|awk '{print $NF}')
	#$NF表示最后一个字段
	#abc=$(命令) 优先执行命令，然后将结果赋值给abc

#修改PATH变量，加入mysql bin目录的路径
#临时修改PATH变量的值
export PATH=/usr/local/mysql/bin:$PATH
#重新启动linux系统后也生效，永久修改
echo 'PATH=/usr/local/mysql/bin:$PATH' >>/root/.bashrc

#复制support-files里的mysql.server文件到/etc/init.d/目录下叫mysqld
cp ../support-files/mysql.server /etc/init.d/mysqld

#修改/etc/init.d/mysql脚本文件里的datadir目录的值
sed -i '70c datadir=/data/mysql' /etc/init.d/mysqld

#生成/etc/my.cnf配置文件
cat >/etc/my.cnf <<EOF
[mysqld_safe]

[client]
socket=/data/mysql/mysql.sock

[mysqld]
socket=/data/mysql/mysql.sock
port = 3306
open_files_limit = 8192
innodb_buffer_pool_size = 512M
character-set-server=utf8

[mysql]
auto-rehash
prompt=\\u@\\d \\R:\\m  mysql>
EOF

#修改内核的open file的数量
ulimit -n 1000000
#设置开机启动的时候也配置生效
echo "ulimit -n 1000000" >>/etc/rc.local
chmod +x /etc/rc.d/rc.local


#启动mysql进程
service mysqld start

#将mysqld添加到linux系统里服务器管理名单里
/sbin/chkconfig --add mysqld
#设置mysqld服务开机启动
/sbin/chkconfig mysqld on

#初次修改密码需要使用--connet-expired-pasword 选项
#-e 后面接的表示是在mysql里需要执行命令 execute 执行
#set password='Sanchuang123#'; 修改root用户的密码为Sanchuang123#
mysql -uroot -p$tem_passwd --connect-expired-password -e "set password='Sanchuang123#';"

#检验上一步修改密码是否成功，如果有输出能看到mysql里的数据库，说明成功
mysql -uroot -p'Sanchuang123#' -e "show databases;"

进入mysql检验安装状况

[root@mysql ~]# mysql -uroot -p'Sanchuang123#'
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.38 MySQL Community Server (GPL)

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

root@(none) 13:38  mysql>

四. 部署Prometheus服务器，实现对于集群整体状态的实时监控

准备好所需的包

[root@total ~]# mkdir /prom
[root@total ~]# cd /prom

解压包，并进行path变量的修改

[root@total prom]# tar xf prometheus-2.43.0.linux-amd64.tar.gz 
[root@total prom]# mv  prometheus-2.43.0.linux-amd64  prometheus
[root@total prom]# cd prometheus
[root@total prometheus]# ls
console_libraries  consoles  LICENSE  NOTICE  prometheus  prometheus.yml  promtool
[root@total prometheus]# PATH=/prom/prometheus:$PATH #临时修改
[root@total prometheus]# echo 'PATH=/prom/prometheus:$PATH' >>/etc/profile #永久修改

将Prometheus制作成server方便管理

[root@total prometheus]# vim /usr/lib/systemd/system/prometheus.service
[root@total prometheus]# systemctl daemon-reload
[root@total prometheus]# systemctl start prometheus
[root@total prometheus]# systemctl restart prometheus
[root@total prometheus]# ps aux|grep prom
root      19905  0.3  2.3 799212 44012 ?        Ssl  14:23   0:00 /prom/prometheus/prometheus --config.file=/prom/prometheus/prometheus.yml
root      19915  0.0  0.0 112824   976 pts/0    S+   14:23   0:00 grep --color=auto prom

设置Prometheus开机启动

[root@total prometheus]# systemctl enable prometheus

通过本机地址9090端口进入Prometheus

在需要监控的节点进行exporter安装

将所需的包通过ansible传输到各节点中

ansible all -m copy  -a 'src=node_exporter-1.4.0-rc.0.linux-amd64.tar.gz dest=/root/'

编写一个用于一键node_exporter的脚本，将其传输到各节点并且执行

[root@total prom]# vim  install_node_exporter.sh
#!/bin/bash
tar xf /root/node_exporter-1.4.0-rc.0.linux-amd64.tar.gz  -C /
cd  /
mv node_exporter-1.4.0-rc.0.linux-amd64/ node_exporter
cd /node_exporter/
echo 'PATH=/node_exporter/:$PATH' >>/etc/profile

#生成nodeexporter.service文件
cat >/usr/lib/systemd/system/node_exporter.service  <<EOF
[Unit]
Description=node_exporter
[Service]
ExecStart=/node_exporter/node_exporter --web.listen-address 0.0.0.0:9090 
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

#让systemd进程识别node_exporter服务
systemctl daemon-reload
#设置开机启动
systemctl  enable node_exporter
#启动node_exporter
systemctl  start node_exporter

[root@total prom]# ansible all -m script  -a "/prom/install_node_exporter.sh"

在Prometheus服务器上添加监控节点

[root@total prom]# cd prometheus
[root@total prometheus]# ls
console_libraries  consoles  LICENSE  NOTICE  prometheus  prometheus.yml  promtool
[root@total prometheus]# vim prometheus.yml
添加下列设置
 - job_name: "web1"
    static_configs:
      - targets: ["192.168.2.161:9090"]
  - job_name: "web2"
    static_configs:
      - targets: ["192.168.2.162:9090"]
  - job_name: "lb1"
    static_configs:
      - targets: ["192.168.2.163:9090"]
  - job_name: "lb2"
    static_configs:
      - targets: ["192.168.2.164:9090"]
  - job_name: "table"
    static_configs:
      - targets: ["192.168.16.165:9090"]
  - job_name: "mysql"
    static_configs:
      - targets: ["192.168.2.167:9090"]
[root@total prometheus]# service prometheus restart
Redirecting to /bin/systemctl restart prometheus.service #重启服务

安装grafana出图工具进行展示

[root@total ~]# yum install grafana-enterprise-9.1.2-1.x86_64.rpm -y
[root@total ~]# systemctl start grafana-server
[root@total ~]# systemctl enable grafana-server #设置开机启动

默认的用户名和密码是

用户名admin

密码admin

五. 部署DNS服务器，提供集群内部的域名解析服务

安装bind

[root@total ~]# yum install bind* -y
[root@total ~]# service named restart
[root@total ~]# systemctl enable named

对/etc/named.conf进行配置

options {
        listen-on port 53 { any;  };
        listen-on-v6 port 53 { any;  };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        recursing-file  "/var/named/data/named.recursing";
        secroots-file   "/var/named/data/named.secroots";
        allow-query     { any; };

#配置完记得重启服务

将dns服务改为搭建的dns服务器的ip地址

[root@total ~]# vim /etc/resolv.conf 

# Generated by NetworkManager
#nameserver 114.114.114.114
nameserver 192.168.2.166
[root@web1 ~]# vim /etc/resolv.conf 

# Generated by NetworkManager
#nameserver 114.114.114.114
nameserver 192.168.2.166

查看dns解析状况

[root@web1 ~]# nslookup  www.baidu.com
Server:		192.168.2.166
Address:	192.168.2.166#53

增加自己项目的zone文件

[root@total ~]#  vim /etc/named.rfc1912.zones
zone "sc.com" IN {
        type master;
        file "sc.com.zone";
        allow-update { none; };
};

在存放域名解析的数据文件 /var/named/ 创建sc.com的数据文件

[root@total ~]# cd /var/named/
[root@total named]# ls
chroot  chroot_sdb  data  dynamic  dyndb-ldap  named.ca  named.empty  named.localhost  named.loopback  slaves
[root@total named]# cp -a named.localhost  sc.com.zone
[root@total named]# vim sc.com.zone #修改sc.com.zone文件，添加我们自己的相关服务的A记录和别名记录
$TTL 1D
@       IN SOA  @ rname.invalid. (
                                        0       ; serial
                                        1D      ; refresh
                                        1H      ; retry
                                        1W      ; expire
                                        3H )    ; minimum
        NS      @
        A       127.0.0.1
        AAAA    ::1
www  A  192.168.203.161
db   A  192.168.203.167
web  CNAME www
*  A  192.168.203.161

在其他机器测试DNS服务

[root@web1 ~]# nslookup  www.sc.com
Server:		192.168.2.166
Address:	192.168.2.166#53

Name:	www.sc.com
Address: 192.168.203.161

[root@web1 ~]# nslookup  db.sc.com
Server:		192.168.2.166
Address:	192.168.2.166#53

Name:	db.sc.com
Address: 192.168.203.167

六. 部署防火墙和堡垒机，提高集群整体的安全性，限制外部访问对于集群内部造成的影响

配置防火墙服务器的LAN口ip

[root@table ~]# cd  /etc/sysconfig/network-scripts/
[root@table network-scripts]# vim ifcfg-ens36
BOOTPROTO="none"
NAME="ens36"
DEVICE="ens36"
ONBOOT="yes"
IPADDR="192.168.2.2"
PREFIX=24
刷新服务
WAN口ip也需要进行相应修改
[root@table ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
BOOTPROTO="none"
NAME="ens33"
UUID="f4cf9294-a239-4a13-bb28-595c015e1feb"
DEVICE="ens33"
ONBOOT="yes"
IPADDR="192.168.1.165"
PREFIX=24
GATEWAY="192.168.1.1"
DNS1=114.114.114.114

配置SNAT和DNAT功能

[root@table ~]# vim /etc/sysctl.conf #开启路由功能
net.ipv4.ip_forward = 1
[root@table ~]# sysctl -p
net.ipv4.ip_forward = 1
编写SNAT和DNAT功能脚本
[root@table ~]# vim set_table_snat_dnat.sh

o 1 >/proc/sys/net/ipv4/ip_forward
#修改/etc/sysctl.conf里添加下面的配置
#net.ipv4.ip_forward = 1

#清除防火墙规则
iptables=/usr/sbin/iptables

$iptables -F
$iptables -t nat -F

#set snat policy
$iptables  -t nat -A POSTROUTING  -s 192.168.2.0/24  -o ens33  -j MASQUERADE
#set dnat policy 发布web1出去，提供web服务
$iptables  -t nat -A PREROUTING -d 192.168.1.165 -i ens33 -p tcp --dport 80 -j DNAT --to-destination 192.168.2.161

#发布堡垒机,访问防火墙的2233端口转发到堡垒机的22端口
$iptables  -t nat -A PREROUTING -d 192.168.1.165 -i ens33 -p tcp --dport 2233 -j DNAT --to-destination 192.168.2.166:22
[root@table ~]# bash set_table_snat_dnat.sh
虚拟机配置完成可能需要等待一段时间

保存规则并设置dnat和snat开机自启动

[root@table ~]#  iptables-save  >/etc/sysconfig/iptables_rules
[root@table ~]# vim /etc/rc.local
iptables-restore  </etc/sysconfig/iptables_rules
[root@table ~]# chmod +x /etc/rc.d/rc.local

部署堡垒机

将内部服务器配置tcp wrappers设置为仅仅允许堡垒机进行访问
在堡垒机上编写脚本配置hosts.allow和hosts.deny文件
[root@total ~]# vim set_tcp_wrappers.sh
#!/bin/bash
#set /etc/hosts.allow文件的内容，只允许堡垒机访问sshd服务
 echo  'sshd:192.168.2.166'  >>/etc/hosts.allow
#单独允许我的windows系统也可以访问
 echo  'sshd:192.168.2.1'  >>/etc/hosts.allow
#拒绝其他的所有的机器访问sshd
echo  'sshd:ALL'  >>/etc/hosts.deny
在内网服务器上执行脚本
[root@total ~]# ansible all -m script -a "/root/set_tcp_wrappers.sh"
此时内网服务器仅可以通过堡垒机进行连接

七. 搭载负载均衡器，分散web服务器承担的压力，增强集群对于高并发的应对能力

安装负载均衡器

使用脚本安装nginx

[root@total ~]# cat onekey_install_nginx.sh 
#!/bin/bash

#解决软件的依赖关系，需要安装的软件包
yum install epel-release -y
yum -y install zlib zlib-devel openssl openssl-devel pcre pcre-devel gcc gcc-c++ autoconf automake make psmisc net-tools lsof vim  wget -y

#新建wangguang用户和组
id  wangguang|| useradd wangguang -s /sbin/nologin

#下载nginx软件
mkdir  /wangguang -p
cd /wangguang
wget  https://nginx.org/download/nginx-1.24.0.tar.gz

#解压软件
tar xf nginx-1.24.0.tar.gz 
#进入解压后的文件夹
cd nginx-1.24.0

#编译前的配置
./configure --prefix=/usr/local/wangguang  --user=wangguang --group=wangguang  --with-http_ssl_module   --with-threads  --with-http_v2_module  --with-http_stub_status_module  --with-stream   --with-http_gunzip_module

#如果上面的编译前的配置失败，直接退出脚本
if (( $? != 0));then
	exit
fi
#编译，启动2个进程去编译，这样速度快
make -j 2
#编译安装
make  install

#修改PATH变量
echo  "PATH=$PATH:/usr/local/wangguang/sbin" >>/etc/bashrc

#firewalld and selinux

#stop firewall和设置下次开机不启动firewalld
service firewalld stop
systemctl disable firewalld

#临时停止selinux和永久停止selinux
setenforce 0
sed  -i '/^SELINUX=/ s/enforcing/disabled/' /etc/selinux/config

#开机启动
chmod +x /etc/rc.d/rc.local
echo  "/usr/local/wangguang/sbin/nginx" >>/etc/rc.local

#启动nginx
/usr/local/wangguang/sbin/nginx

修改配置文件，添加负载均衡，执行轮询（rr）算法

[root@lb1 ~]# cd /usr/local/wangguang/conf/
[root@lb1 conf]# vim nginx.conf
keepalive_timeout  65;
#定义一个负载均衡器scapp
    upstream scapp {
        server 192.168.2.161;
        server 192.168.2.162;
    }	
	
    #gzip  on;

    server {
        listen       80;
        server_name  localhost;

        location / {
        #访问网页根目录直接转发到负载均衡器上进行处理
             proxy_pass http://scapp;
        }
#记得重启服务

测试负载均衡效果

[root@web2 html]# curl 192.168.2.163
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@web2 html]# curl 192.168.2.163
<!DOCTYPE html>
<html>
<head>
<title>Welcome to www!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to www!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using www.</em></p>
</body>
</html>

重新配置防火墙的DNAT转发

[root@table ~]# vim set_table_snat_dnat.sh 

echo 1 >/proc/sys/net/ipv4/ip_forward
#修改/etc/sysctl.conf里添加下面的配置
#net.ipv4.ip_forward = 1

#清除防火墙规则
iptables=/usr/sbin/iptables

$iptables -F
$iptables -t nat -F

#set snat policy
$iptables  -t nat -A POSTROUTING  -s 192.168.2.0/24  -o ens33  -j MASQUERADE

#set dnat policy 使用2个外网的ip对应内部的2个vip，将2个vip发布出去，提供web服务
$iptables  -t nat -A PREROUTING -d 192.168.16.165 -i ens33 -p tcp --dport 80 -j DNAT --to-destination 192.168.2.163


#发布堡垒机,访问防火墙的2233端口转发到堡垒机的22端口
$iptables  -t nat -A PREROUTING -d 192.168.16.165 -i ens33 -p tcp --dport 2233 -j DNAT --to-destination 192.168.2.166:22

在网站上尝试访问，验证负载均衡效果

八. 部署keepalive服务，增强负载均衡效果的稳定性，实现web集群的高可用

部署keepalive

[root@lb1 ~]# yum install keepalived -y
[root@lb2 ~]# yum install keepalived -y
修改keepalive配置文件，配置双vip
192.168.2.188
192.168.2.199
lb1
[root@lb1 ~]# cat  /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   #vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 180
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 11112222
    }
    virtual_ipaddress {
        192.168.2.188
    }
}
vrrp_instance VI_2 {
    state MASTER
    interface ens33
    virtual_router_id 181
    priority 120
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 11112222
    }
    virtual_ipaddress {
        192.168.2.199
    }
}
lb2
[root@lb2 ~]# cat  /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   #vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 180
    priority 120
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 11112222
    }
    virtual_ipaddress {
        192.168.2.188
    }
}

vrrp_instance VI_2 {
    state BACKUP
    interface ens33
    virtual_router_id 181
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 11112222
    }
    virtual_ipaddress {
        192.168.2.199
    }
}

重启服务，查看ip确定效果

[root@lb1 ~]# service keepalived restart
Redirecting to /bin/systemctl restart keepalived.service
[root@lb1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:2d:5c:bf brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.163/24 brd 192.168.2.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever

[root@lb2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:71:aa:79 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.164/24 brd 192.168.2.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.2.188/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe71:aa79/64 scope link 
       valid_lft forever preferred_lft forever
[root@lb2 ~]# service keepalived stop
Redirecting to /bin/systemctl stop keepalived.service
[root@lb2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:71:aa:79 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.164/24 brd 192.168.2.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe71:aa79/64 scope link 
       valid_lft forever preferred_lft forever
[root@lb1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:2d:5c:bf brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.163/24 brd 192.168.2.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.2.199/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.2.188/32 scope global ens33
       valid_lft forever preferred_lft forever

验证vip漂移效果

九. 使用ab压力测试工具对集群进行测试，检测集群压力上限

安装压力测试工具
```
[root@ab ~]# yum install httpd-tools -y
```

对网址进行检测

[root@ab ~]# ab -n 1000 -c 100 http://192.168.16.165/index.html
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.16.165 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx/1.24.0
Server Hostname:        192.168.16.165
Server Port:            80

Document Path:          /index.html
Document Length:        609 bytes

Concurrency Level:      100
Time taken for tests:   0.128 seconds
Complete requests:      1000
Failed requests:        499
   (Connect: 0, Receive: 0, Length: 499, Exceptions: 0)
Write errors:           0
Total transferred:      844994 bytes
HTML transferred:       611994 bytes
Requests per second:    7838.34 [#/sec] (mean)
Time per request:       12.758 [ms] (mean)
Time per request:       0.128 [ms] (mean, across all concurrent requests)
Transfer rate:          6468.12 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    4   1.0      4       7
Processing:     3    8   1.5      7      14
Waiting:        1    6   1.1      6      10
Total:          7   12   1.5     12      17

Percentage of the requests served within a certain time (ms)
  50%     12
  66%     12
  75%     12
  80%     13
  90%     14
  95%     14
  98%     16
  99%     16
 100%     17 (longest request)
[root@ab ~]# ab -n 1000 -c 100 http://192.168.16.168/index.html
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.16.168 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx/1.24.0
Server Hostname:        192.168.16.168
Server Port:            80

Document Path:          /index.html
Document Length:        615 bytes

Concurrency Level:      100
Time taken for tests:   0.138 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      848000 bytes
HTML transferred:       615000 bytes
Requests per second:    7262.59 [#/sec] (mean)
Time per request:       13.769 [ms] (mean)
Time per request:       0.138 [ms] (mean, across all concurrent requests)
Transfer rate:          6014.33 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   1.9      1      10
Processing:     3   11   2.8     11      19
Waiting:        1   10   3.0     10      19
Total:          6   13   2.9     12      22

Percentage of the requests served within a certain time (ms)
  50%     12
  66%     13
  75%     13
  80%     14
  90%     19
  95%     20
  98%     21
  99%     21
 100%     22 (longest request)

项目心得

进行项目的过程中，存档可以更方便在面对问题时进行规避和解决
列好项目表单，可以使得操作更加有条理性
加深了对于nginx负载均衡和web代理功能的了解
对于keepalive的使用更加深入，进一步了解了双vip的搭建过程和使用优点，也见识了脑裂现象的特点和危害
掌握了使用Prometheus+grafana监控工具的使用
增进了对于防火墙和跳板机部署的认识，对于跳板机存在的意义更加清楚
使用ab压力测试工具，对系统性能资源的重要性的认知更加完善，对集群在压力下的瓶颈更有概念

轩苍

关注

13
点赞
踩
10

收藏

觉得还不错? 一键收藏
4
评论
应用nginx+keepalive搭建的高可用高性能web集群

为满足企业内部web项目需求，模拟构建一个高可用、高性能的web集群。使用nginx作为web服务器，并搭配Prometheus和Grafana构建监控系统，实现集群运行状态的实时监控。同时借助nginx负载均衡与keepalived技术，确保集群的高可用性。并引入ansible实现集群的自动化运维，提高运维效率。
复制链接

扫一扫