综合web集群项目

1.关闭防火墙、selinux
#systemctl stop firewalld

#systemctl disabled firewalld

#sed -i '/^selinux=/ s /enforcing/disabled/' /etc/rc.local

2.网卡统一选择nat模式(后续得改成hostonly)，配置静态ip地址

#vim /etc/sysyconfig/network-script/ifcfg-ens33

#
BOOTPROTO="static"
DEFROUTE="yes"
NAME="ens33"
DEVICE="ens33"
ONBOOT="yes"
IPADDR="192.168.159.136"
NETMASK="255.255.255.0"
GATEWAY="192.168.159.2"
DNS1="114.114.114.114"
#

#systemctl restart network

3.下载好后续常用的软件

#yum install -y epel-release net-tools vim wget bind-utils glances lrzsz yum-utils

1.2.根据图中的规划，克隆好所有需要的虚拟机

依次配置好每个机器的静态ip(192.168.159.136~142)、给每台机器设置好主机名
#hostnamectl set-hostname web1 等等

二、Ansible服务器的配置和使用

1.建立免密通道

[root@nfs-ansible ansible]# ssh-keygen

[root@nfs-ansible ansible]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.159.137

Are you sure you want to continue connecting (yes/no)? yes

1.1 测试免密通道是否成功

[root@nfs-ansible ansible]# ssh 'root@192.168.159.137'

Last login: Sat Sep 23 09:22:11 2023 from 192.168.159.136

[root@web1 ~]# exit 退出

1.2 和其他服务器全部建立免密通道

[root@nfs-ansible ansible]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.203.138

[root@nfs-ansible ansible]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.203.139

[root@nfs-ansible ansible]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.203.140

[root@nfs-ansible ansible]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.203.141

[root@nfs-ansible ansible]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.203.142

2.主控机安装ansible

[root@nfs-ansible web]# yum install ansible -y

2.1 配置主机清单

[root@nfs-ansible ansible]# vim /etc/ansible/hosts

添加分组

[web]
192.168.159.140
192.168.159.141
[fw]
192.168.159.139
[lb]
192.168.159.137
192.168.159.138
[db]
192.168.159.142

2.2 测试ansible服务器能否控制所有的服务器

[root@nfs-ansible ansible]# ansible all -m shell -a 'ip add'

192.168.159.142 | CHANGED | rc=0 >>
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:b8:b9:fe brd ff:ff:ff:ff:ff:ff
inet 192.168.159.143/24 brd 192.168.159.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::d2cc:c11d:55fc:392c/64 scope link noprefixroute
valid_lft forever preferred_lft forever
出现类似这样的结果则为成功

三、配置Prometheus+node_exporter+grafana监控所有的服务器

1.下载源码安装包

从Download | Prometheus下载好需要的prometheus和node_exporter安装包
通过rz命令上传到ansible主控服务器上

[root@nfs-ansible-prom ansible]# mkdir /prom

[root@nfs-ansible-prom ansible]# cd /prom

[root@nfs-ansible-prom ansible]# rz

[root@nfs-ansible-prom prom]# ls

prometheus-2.43.0.linux-amd64.tar.gz node_exporter-1.4.0-rc.0.linux-amd64.tar.gz

2.安装Prometheus

[root@nfs-ansible-prom prom]# tar xf prometheus-2.43.0.linux-amd64.tar.gz

2.1 修改解压后的压缩包名字prometheus

[root@nfs-ansible-prom prom]# mv prometheus-2.43.0.linux-amd64 prometheus

2.2 永久修改PATH变量，添加prometheus的路径

[root@nfs-ansible-prom prom]# cd prometheus
[root@nfs-ansible-prom prometheus]# pwd
/prom/prometheus

[root@nfs-ansible-prom prom]# echo 'PATH=/prom/prometheus:$PATH' >>/etc/profile

2.3 把prometheus做成一个服务来进行管理，非常方便日后维护和使用

[root@nfs-ansible-prom prom]# vim /usr/lib/systemd/system/prometheus.service

[Unit]

Description=prometheus

[Service]

ExecStart=/prom/prometheus/prometheus --config.file=/prom/prometheus/prometheus.yml

ExecReload=/bin/kill -HUP $MAINPID

KillMode=process

Restart=on-failure

[Install]

WantedBy=multi-user.target

2.4 重新加载systemd相关的服务，识别Prometheus服务的配置文件

[root@nfs-ansible-prom prom]# systemctl daemon-reload

启动Prometheus服务

[root@nfs-ansible-prom prom]# systemctl start prometheus

[root@nfs-ansible-prom prom]# systemctl restart prometheus

[root@nfs-ansible-prom prom]# ps aux|grep prome

root 2740 2.7 0.9 798956 37312 ? Ssl 14:57 0:00 /prom/prometheus/prometheus --config.file=/prom/prometheus/prometheus.yml

root 2748 0.0 0.0 112824 976 pts/0 S+ 14:57 0:00 grep --color=auto prome

设置开机启动

[root@nfs-ansible-prom prom]# systemctl enable prometheus

2.5 访问Prometheus 的web server

3.给各个服务器安装exporter监控

3.1 将node-exporter传递到所有的服务器上的/root目录下

[root@nfs-ansible-prom prom]# ansible all -m copy -a 'src=/prom/node_exporter-1.4.0-rc.0.linux-amd64.tar.gz dest=/root/'

3.2 在主控机上编写安装node_exporter的脚本

[root@nfs-ansible-prom prom]#vim install_node_exporter.sh

#!/bin/bash

tar xf /root/node_exporter-1.4.0-rc.0.linux-amd64.tar.gz -C /

cd /

mv node_exporter-1.4.0-rc.0.linux-amd64/ node_exporter

cd /node_exporter/

echo 'PATH=/node_exporter/:$PATH' >>/etc/profile

#生成nodeexporter.service文件

cat >/usr/lib/systemd/system/node_exporter.service <<EOF

[Unit]

Description=node_exporter

[Service]

ExecStart=/node_exporter/node_exporter --web.listen-address 0.0.0.0:9090

ExecReload=/bin/kill -HUP $MAINPID

KillMode=process

Restart=on-failure

[Install]

WantedBy=multi-user.target

EOF

#让systemd进程识别node_exporter服务

systemctl daemon-reload

#设置开机启动

systemctl enable node_exporter

#启动node_exporter

systemctl start node_exporter

3.3 在主控机上执行安装脚本

[root@nfs-ansible-prom prom]# ansible all -m script -a "/prom/install_node_exporter.sh"

192.168.159.137 | CHANGED => {

    "changed": true,

    "rc": 0,

    "stderr": "Shared connection to 192.168.159.137 closed.\r\n",

    "stderr_lines": [

        "Shared connection to 192.168.159.137 closed."

    ],

    "stdout": "Created symlink from /etc/systemd/system/multi-user.target.wants/node_exporter.service to /usr/lib/systemd/system/node_exporter.service.\r\n",

    "stdout_lines": [

        "Created symlink from /etc/systemd/system/multi-user.target.wants/node_exporter.service to /usr/lib/systemd/system/node_exporter.service."

    ]

}

......

在其他的服务器上查看是否安装node_exporter成功

[root@lb2 ~]# ps aux|grep node

root       5408 0.0 0.2 716288 11068 ?        Ssl 15:25   0:00 /node_exporter/node_exporter --web.listen-address 0.0.0.0:9090

root       5524 0.0 0.0 112824   976 pts/1    S+   15:27   0:00 grep --color=auto node

3.4 添加被监控主机

[root@nfs-ansible-prom prometheus]# vim /prom/prometheus/prometheus.yml

添加以下内容

scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "web1"
static_configs:
- targets: ["192.168.159.140:9090"]
- job_name: "lb1"
static_configs:
- targets: ["192.168.159.137:9090"]
- job_name: "fw"
static_configs:
- targets: ["192.168.159.139:9090"]

重启Prometheus服务

[root@nfs-ansible-prom prometheus]# service prometheus restart

在windows的浏览器访问prometheus服务器的/targets路径

http://192.168.159.136:9090/targets

4. 在Prometheus上添加grafana

4.1 下载grafana

grafana出图软件可以在官网选择版本下载

Download Grafana | Grafana Labs

[root@nfs-ansible-prom prom]# yum install -y https://dl.grafana.com/enterprise/release/grafana-enterprise-10.4.1-1.x86_64.rpm

启动grafana

[root@nfs-ansible-prom prom]# systemctl start grafana-server

设置开机启动

[root@nfs-ansible-prom prom]# systemctl enable grafana-server

4.2 查看网页效果

grafana默认监听的是3000端口

在浏览器里访问 http://192.168.203.135:3000

默认的用户名和密码是

用户名admin

密码admin

登录后加入主界面,找到添加数据源的地方

填写监控名称和数据源地址

4.3 导入grafana的模板

选择8919模块

需要其他模板可以参考官网https://grafana.com/grafana/dashboards

得到监控效果

四、配置DNS服务

1.安装软件bind

bind是历史非常悠久，而且性能非常好的dns域名系统的软件

[root@nfs-ansible-prom prom]# yum install bind* -y

启动named服务

[root@nfs-ansible-prom prom]# service named restart

Redirecting to /bin/systemctl restart named.service

设置named服务开机启动

[root@nfs-ansible-prom prom]# systemctl enable named

查看开放udp 53号端口

[root@nfs-ansible-prom prom]# netstat -anplut|grep named

tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      14125/named

tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN      14125/named

tcp6       0      0 ::1:53                  :::*                    LISTEN      14125/named

2.修改配置文件

重启服务器允许其他电脑能过来查询dns域名

[root@nfs-ansible-prom prom]# vim /etc/named.conf

options {

        listen-on port 53 { any; }; #修改

        listen-on-v6 port 53 { any; }; #修改

        directory       "/var/named";

        dump-file       "/var/named/data/cache_dump.db";

        statistics-file "/var/named/data/named_stats.txt";

        memstatistics-file "/var/named/data/named_mem_stats.txt";

        recursing-file "/var/named/data/named.recursing";

        secroots-file   "/var/named/data/named.secroots";

        allow-query     { any; }; #修改

重新启动named服务

[root@nfs-ansible-prom prom]# service named restart

3.在其他机器上测试DNS服务器

修改本机和其他机器的dns服务器为自己搭建的dns服务器ip，添加2个DNS服务器地址，第1个DNS服务器地址优先级高，这里不建议在/etc/resolv.conf文件内修改，因为这个文件是根据网卡配置文件内的DNS设置的，不修改网卡配置文件重启服务器后DNS会被再次修改回去

在ansible主控机上编写替换ens33网卡文件中的DNS1.并添加DNS2的set_dns.sh脚本

#!/bin/bash

sed -i '/^DNS1=/ s/"114.114.114.114"/"192.168.159.136"/' /etc/sysconfig/network-scripts/ifcfg-ens33

echo "DNS2=114.114.114.114" >> /etc/sysconfig/network-scripts/ifcfg-ens33

cat /etc/sysconfig/network-scripts/ifcfg-ens33

在主控机上执行脚本

[root@nfs-ansible-prom ~]# ansible all -m script -a "/root/set_dns.sh"
重启网络服务

[root@nfs-ansible-prom ~]# ansible all -m shell -a "systemctl restart network"
使用nslookup和ping、dig命令在web机器上测试DNS服务是否生效
nslookup、dig命令是bind-utils包里面的，在初始化机器的时候已经下好了

[root@web1 ]# nslookup www.baidu.com

Server: 192.168.159.136

Address: 192.168.159.136#53

Non-authoritative answer:

www.baidu.com canonical name = www.a.shifen.com.

Name: www.a.shifen.com

Address: 183.2.172.42

Name: www.a.shifen.com

Address: 183.2.172.185

Name: www.a.shifen.com

Address: 240e:ff:e020:9ae:0:ff:b014:8e8b

Name: www.a.shifen.com

Address: 240e:ff:e020:966:0:ff:b042:f296

4 增加自己项目的zone文件

1.修改配置文件，告诉named为wy.com提供域名解析

[root@nfs-ansible-prom prom]# vim /etc/named.rfc1912.zones

//添加下面的配置，增加一个wy.com的域名

zone "wy.com" IN {

        type master;

        file "wy.com.zone";

        allow-update { none; };

};

2. /var/named/ 存放dns域名解析的数据文件wy.com.zone --》创建wy.com的数据文件

[root@nfs-ansible-prom prom]# cd /var/named/

[root@nfs-ansible-prom named]# cp -a named.localhost wy.com.zone

修改wy.com.zone文件，添加我们自己的相关服务的A记录和别名记录

[root@nfs-ansible-prom named]# vim wy.com.zone

$TTL 1D

@ IN SOA @ rname.invalid. (

0 ; serial

1D ; refresh

1H ; retry

1W ; expire

3H ) ; minimum

NS @

A 192.168.203.145

AAAA ::1

#添加以下配置

web1 A 192.168.159.140

web2 A 192.168.159.141

lb1 A 192.168.159.137

lb2 A 192.168.159.138

fw A 192.168.159.139

www CNAME web2 # CNAME 别名

刷新named服务，让新的配置生效

[root@nfs-ansible-prom named]# service named restart

检查数据文件是否有错误

[root@nameserver data]# named-checkzone wy.com /var/named/wy.com.zone

zone wy.com/IN: loaded serial 0

OK

检查配置文件

[root@nameserver data]# named-checkconf /etc/named.rfc1912.zones

5. 测试DNS服务器记录

在其他的服务器上进行测试，我们添加的A记录和别名记录，泛域名解析记录

[root@web1 ~]# nslookup web2.wy.com

Server: 192.168.159.136

Address: 192.168.159.136#53

Name: web2.wy.com

Address: 192.168.159.141

[root@web1 ~]# nslookup www.wy.com

Server: 192.168.159.136

Address: 192.168.159.136#53

Name: web2.wy.com

Address: 192.168.159.141

五、配置DMZ区+防火墙+路由

1.给防火墙服务器配置2块网卡，ens33作为WAN口，ens36作为LAN口

WAN口设置为桥接模式，LAN口设置成hostonly模式作为DMZ区的默认路由

[root@firewall ~]# cd /etc/sysconfig/network-scripts/

[root@firewall network-scripts]# ls

ifcfg-ens33

[root@firewall network-scripts]# vim ifcfg-ens33

BOOTPROTO="none"

NAME="ens33"

UUID="49630322-c7fc-4b3b-8519-c280d9d42d3b"

DEVICE="ens33"

ONBOOT="yes"

IPADDR=192.168.1.179

PREFIX=24

GATEWAY=192.168.1.1

DNS1=114.114.114.114

配置ens36网卡的静态ip，不配置网关和dns服务器地址，因为WAN口ens33里有配置

[root@firewall network-scripts]# cp ifcfg-ens33 ifcfg-ens36

[root@firewall network-scripts]# vim ifcfg-ens36

BOOTPROTO="none"

NAME="ens36"

DEVICE="ens36"

ONBOOT="yes"

IPADDR=192.168.159.2 #DMZ区内都是nat模式，网关都为192.168.159.2

PREFIX=24

刷新网络服务

[root@firewall network-scripts]# service network restart

Restarting network (via systemctl):                        [ 确定 ]

[root@firewall network-scripts]# ip add

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 00:0c:29:18:a3:b4 brd ff:ff:ff:ff:ff:ff

    inet 192.168.1.179/24 brd 192.168.1.255 scope global noprefixroute ens33

       valid_lft forever preferred_lft forever

   inet6 fe80::20c:29ff:fe18:a3be/64 scope link

       valid_lft forever preferred_lft forever

3: ens36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 00:0c:29:18:a3:be brd ff:ff:ff:ff:ff:ff

    inet 192.168.159.2/24 brd 192.168.203.255 scope global noprefixroute ens36

       valid_lft forever preferred_lft forever

    inet6 fe80::20c:29ff:fe18:a3be/64 scope link

       valid_lft forever preferred_lft forever

2.DMZ区里的所有服务器网卡类型调整为hostonly模式

注：由于DMZ区之前全是nat模式所有网段和hostonly模式网段不一样，这里我们只需要调换VMware里面的两个模式的网段即可省略修改网段的繁琐事。

在编辑打开虚拟网络编辑器，将hostonly的网段切换为nat的网段

3.防火墙开启路由功能，配置SNAT和DNAT功能

1. 开启路由功能

[root@firewall ~]# vim /etc/sysctl.conf

net.ipv4.ip_forward = 1

[root@firewall ~]# sysctl -p 让内核读取配置文件，开启路由功能

net.ipv4.ip_forward = 1

2. 编写SNAT和DNAT功能脚本

[root@firewall ~]# vim set_fw_snat_dnat.sh

#!/bin/bash

iptables=/usr/sbin/iptables

$iptables -F
$iptables -t nat -F

#set snat policy
$iptables -t nat -A POSTROUTING -s 192.168.159.0/24 -o ens33 -j MASQUERADE

#发布web服务
$iptables -t nat -A PREROUTING -d 192.168.1.179 -i ens33 -p tcp --dport 80 -j DNAT --to-destination 192.168.159.140

#发布Prometheus
$iptables -t nat -A PREROUTING -d 192.168.1.179 -i ens33 -p tcp --dport 9090 -j DNAT --to-destination 192.168.159.136:9090

#发布grafana
$iptables -t nat -A PREROUTING -d 192.168.1.179 -i ens33 -p tcp --dport 3000 -j DNAT --to-destination 192.168.159.136:3000

#发布堡垒机,访问防火墙的2233端口转发到堡垒机的22端口
$iptables -t nat -A PREROUTING -d 192.168.1.179 -i ens33 -p tcp --dport 2233 -j DNAT --to-destination 192.168.159.136:22

执行脚本

[root@firewall ~]# bash set_fw_snat_dnat.sh

查看脚本的执行效果

[root@firewalld ~]# iptables -L -t nat -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DNAT tcp -- 0.0.0.0/0 192.168.1.179 tcp dpt:80 to:192.168.159.143
DNAT tcp -- 0.0.0.0/0 192.168.1.179 tcp dpt:9090 to:192.168.159.136:9090
DNAT tcp -- 0.0.0.0/0 192.168.1.179 tcp dpt:3000 to:192.168.159.136:3000
DNAT tcp -- 0.0.0.0/0 192.168.1.179 tcp dpt:2233 to:192.168.159.136:22

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 192.168.159.0/24 0.0.0.0/0

3. 保存规则

[root@firewall ~]# iptables-save >/etc/sysconfig/iptables_rules

设置snat和dnat策略开机启动

[root@firewall ~]# vim /etc/rc.local

iptables-restore  </etc/sysconfig/iptables_rules

[root@firewall ~]# chmod +x /etc/rc.d/rc.local

[root@firewall ~]#

重启防火墙，验证规则是否开机加载

[root@firewall ~]# reboot

重新启动后，进入系统查看iptables里的nat表里的规则

[root@firewall ~]# iptables -L -t nat -n

Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DNAT tcp -- 0.0.0.0/0 192.168.1.179 tcp dpt:80 to:192.168.159.143
DNAT tcp -- 0.0.0.0/0 192.168.1.179 tcp dpt:9090 to:192.168.159.136:9090
DNAT tcp -- 0.0.0.0/0 192.168.1.179 tcp dpt:3000 to:192.168.159.136:3000
DNAT tcp -- 0.0.0.0/0 192.168.1.179 tcp dpt:2233 to:192.168.159.136:22

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 192.168.159.0/24 0.0.0.0/0

4.测试DMZ区里的服务器能否上网

在web1服务器上测试

[root@web1 network-scripts]# ping www.baidu.com

PING www.a.shifen.com (183.2.172.42) 56(84) bytes of data.

64 bytes from 183.2.172.42 (183.2.172.42): icmp_seq=1 ttl=51 time=26.7 ms

64 bytes from 183.2.172.42 (183.2.172.42): icmp_seq=2 ttl=51 time=23.9 ms

64 bytes from 183.2.172.42 (183.2.172.42): icmp_seq=3 ttl=51 time=23.6 ms

5. 在window机器上测试grafana、Prometheus是否发布

6.测试发布的web服务

这里是我自己找的一个web服务网站如需可以自行下载

wget http://download.comsenz.com/DiscuzX/2.5/Discuz_ X2.5_SC_ UTF8.zip

不想下载可以直接用nginx代替

下载网站需要的软件

yum install -y httpd mariadb-server mariadb php php-mysql gd php-gd

7.测试发布堡垒机

使用xshell连接防火墙的2233端口，连接到堡垒机

连接成功的效果如下：

4.Tcp wrappers 的配置

1.设置tcp wrappers

DMZ区所有的服务器应该只允许堡垒机ssh访问，将web集群里的web1和web2，LB1和LB2上进行tcp wrappers的配置，只允许堡垒机ssh进来，拒绝其他的机器ssh过去。

在堡垒机上编写脚本去配置hosts.allow和hosts.deny文件

[root@nfs-ansible-prom ~]# vim set_tcp_wrappers.sh

#!/bin/bash

#set /etc/hosts.allow文件的内容，只允许堡垒机访问sshd服务

echo 'sshd:192.168.159.136' >>/etc/hosts.allow

#单独允许我的windows系统也可以访问 --对应window vmware net1网卡ip地址

echo 'sshd:192.168.159.1' >>/etc/hosts.allow

#拒绝其他的所有的机器访问sshd

echo 'sshd:ALL' >>/etc/hosts.deny

对web和db组的机器配置tcp wrappers防火墙

[root@nfs-ansible-prom ~]# ansible all -m script -a "/root/set_tcp_wrappers.sh"

192.168.159.140 | CHANGED => {

    "changed": true,

    "rc": 0,

    "stderr": "Shared connection to 192.168.159.140 closed.\r\n",

    "stderr_lines": [

        "Shared connection to 192.168.159.140closed."

    ],

    "stdout": "",

    "stdout_lines": []

}

.......

2.测试tcp wrappers

lb1服务器去ssh连接web1服务器

[root@lb1 ~]# ssh root@192.168.159.140

ssh_exchange_identification: read: Connection reset by peer

堡垒机可以ssh过去，说明tcp wrappers 配置成功

[root@nfs-ansible-prom ~]# ssh root@192.168.159.140

Last login: Thu Mar 28 11:55:48 2024 from 192.168.159.136

通过以上测试验证，DMZ区里的服务器只有堡垒机（跳板机）可以ssh到其他的服务器，其他服务器之间不能互相ssh连接，提高了服务器的安全性

六、给两台LB服务器配置nginx的负载均衡功能

1.编写一键安装nginx脚本

[root@ansible ~]#vim install_nginx.sh
#!/bin/bash

#新建一个文件夹用来存放下载的nginx源码包
mkdir -p /nginx
cd /nginx

#新建用户
useradd wy -s /sbin/nologin

#下载nginx源码包
yum install wget -y
wget http://nginx.org/download/nginx-1.25.2.tar.gz

#解压nginx源码包
tar xf nginx-1.25.2.tar.gz

#解决依赖关系
yum -y install openssl openssl-devel pcre pcre-devel gcc autoconf automake make

#编译前的配置
./configure --prefix=/usr/local/wy --user=hanwei --with-threads --with-http_ssl_module --with-http_v2_module --with-http_stub_status_module --with-stream

#编译，开启2个进程同时编译，速度会快些
make -j 2

#安装
make install

#启动nginx
/usr/local/wy/sbin/nginx

#修改PATH变量
PATH=$PATH:/usr/local/wy/sbin/
echo "PATH=$PATH:/usr/local/wy/sbin/" >>/root/.bashrc

#设置nginx开机启动
echo "/usr/local/wy/sbin/nginx" >>/etc/rc.local
chmod +x /etc/rc.d/rc.local

2.编写playbook，给web、lb分组都安装上nginx

[root@ansible ~]# vim nginx.yaml
- hosts: web
remote_user: root
tasks:
- name: mkdir /web
file: path=/web state=directory
- name: cp install_nginx.sh to hosts
copy: src=/root/install_nginx.sh dest=/web/install_nginx.sh
- name: install nginx
shell: bash /web/install_nginx.sh

- hosts: lb
remote_user: root
tasks:
- name: mkdir /web
file: path=/web state=directory
- name: cp install_nginx.sh to hosts
copy: src=/root/install_nginx.sh dest=/web/install_nginx.sh
- name: install nginx
shell: bash /web/install_nginx.sh

[root@ansible ~]# ansible-playbook --syntax-check nginx.yaml

playbook: nginx.yaml

[root@ansible ~]# ansible-playbook nginx.yaml

3.在两台LB服务器上开启负载均衡，调度算法采用轮询。

3.1 开启七层负载均衡

[root@lb1 ~]# vim /usr/local/wy/conf/nginx.conf
http {
upstream nginx_web {
server 192.168.159.140;
server 192.168.159.141;

}
server {
listen 80;
location / {
proxy_pass http://nginx_web;
}
}
}

[root@lb1 conf]# nginx -t
nginx: the configuration file /usr/local/wy/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/wy/conf/nginx.conf test is successful

[root@lb1 conf]# nginx -s reload

[root@lb2 conf]# nginx -t
nginx: the configuration file /usr/local/wy/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/wy/conf/nginx.conf test is successful

[root@lb2 conf]# nginx -s reload

3.2 开启四层负载均衡

[root@lb1 conf]# cat nginx.conf

worker_processes 1;

events {
worker_connections 1024;
}

stream {

upstream web_servers {
server 192.168.159.140:80;
server 192.168.159.141:80;

}

server {
listen 80 ;
proxy_pass web_servers;
}
}

七、搭建nfs服务器，保证web集群的数据一致性

1.搭建nfs服务器

1.给nfs服务器和web集群安装好nfs-utils

[root@nfs-ansible-prom ~]# yum install -y nfs-utils

[root@nfs-ansible-prom ~]# ansible web -m yum -a "name=nfs-utils state=install"

2.新建共享目录和index.html
[root@nfs-ansible-prom ~]# mkdir /nginx
[root@nfs ~]# cd /nginx
[root@nfs nginx]# echo "hello world" >index.html
[root@nfs nginx]# ls
index.html

3.设置共享目录
[root@nfs ~]# vim /etc/exports
[root@nfs-ansible-prom ~]# cat /etc/exports
/nginx 192.168.159.0/24(ro,no_root_squash,sync)

4.刷新nfs或者重新输出共享目录
[root@nfs-ansible-prom ~]# exportfs -r #输出所有共享目录
[root@nfs-ansible-prom ~]# exportfs -v #显示输出的共享目录
/nginx 192.168.159.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,no_root_squash,no_all_squash)

5.重启nfs服务并且设置nfs开机自启
[root@nfs-ansible-prom ~]# systemctl restart nfs && systemctl enable nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.

6.挂载nfs服务器共享的目录
[root@web1 ~]# mount 192.168.159.136:/nginx /usr/local/wy/html

[root@web1 ~]# df -Th|grep nfs
192.168.159.136:/nginx nfs4 17G 1.5G 16G 9% /usr/local/wy/html

2.开机自动挂载nfs文件系统

# web-1、web-2上操作

vim /etc/fstab

192.168.159.136:/nginx /usr/local/wy/html nfs defaults 0 0

八、在负载均衡器上安装keepalived软件，给负载均衡器做高可用，防止单点故障

1.在ansible主控机上给lb安装keepalived

[root@nfs-ansible-prom ~]# ansible lb -m yum -a "name=keepalived state=install"

2.修改配置文件

[root@lb11 conf]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}

vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 58
priority 120
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.159.188
}
}

vrrp_instance VI_2 {
state BACKUP
interface ens33
virtual_router_id 59
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.159.189
}
}

[root@lb2 keepalived]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}

vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 58
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.159.188
}
}

vrrp_instance VI_2 {
state MASTER
interface ens33
virtual_router_id 59
priority 120
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.159.189
}
}

[root@lb1 keepalived]# service keepalived restart
Redirecting to /bin/systemctl restart keepalived.service

[root@lb2 keepalived]# service keepalived restart
Redirecting to /bin/systemctl restart keepalived.service

[root@lb2 keepalived]# ps aux|grep keepa
root 1708 0.0 0.0 123020 2032 ? Ss 16:14 0:00 /usr/sbin/keepalived -D
root 1709 0.0 0.1 133992 7892 ? S 16:14 0:00 /usr/sbin/keepalived -D
root 1712 0.0 0.1 133860 6160 ? S 16:14 0:00 /usr/sbin/keepalived -D
root 1719 0.0 0.0 112832 2392 pts/0 S+ 16:14 0:00 grep --color=auto keepa

查看vip
[root@lb1 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:37:fb:39 brd ff:ff:ff:ff:ff:ff
inet 192.168.159.137/24 brd 192.168.0.255 scope global noprefixroute dynamic ens33
valid_lft 1075sec preferred_lft 1075sec
inet 192.168.159.188/32 scope global ens33
valid_lft forever preferred_lft forever

[root@lb2 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:37:fb:39 brd ff:ff:ff:ff:ff:ff
inet 192.168.159.138/24 brd 192.168.0.255 scope global noprefixroute dynamic ens33
valid_lft 1075sec preferred_lft 1075sec
inet 192.168.159.189/32 scope global ens33
valid_lft forever preferred_lft forever

3. 监控负载均衡器上的nginx

如果负载均衡器上的nginx程序出现问题（例如：nginx没启动），就会导致访问web集群出现问题

解决思路：如果检查到nginx进程关闭，将优先级降低30，停止它的master身份，让位给其他的机器；或者关闭keepalived服务。

1.编写脚本

[root@lb1 web]# cat check_nginx.sh
#!/bin/bash

#检测nginx是否正常运行
if /usr/sbin/pidof nginx ;then
exit 0
else
exit 1
fi

[root@lb1 web]# chmod +x check_nginx.sh
[root@lb1 web]# cat halt_keepalived.sh
#!/bin/bash

service keepalived stop

[root@lb1 web]# chmod +x halt_keepalived.sh

2.在keepalived里定义监控脚本

#定义监控脚本chk_nginx
vrrp_script chk_nginx {
#当脚本/web/check_nginx.sh脚本执行返回值为0的时候，不执行下面的weight -30的操作，只有脚本执行失败，返回值非0的时候，就执行执行权重值减30的操作
script "/web/check_nginx.sh"
interval 1
weight -30
}

[root@lb-1 keepalived]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
#定义监控脚本chk_nginx
vrrp_script chk_nginx {
#当脚本/web/check_nginx.sh脚本执行返回值为0的时候，不执行下面的weight -30的操作，只有脚本执行失败，返回值非0的时候，就执行执行权重值减30的操作
script "/web/check_nginx.sh"
interval 1
weight -30
}

vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 58
priority 120
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.159.188
}
#调用监控脚本
track_script {
chk_nginx
}

#当本机成为backup的时候，立马执行下面的脚本
notify_backup "/web/halt_keepalived.sh"
}
vrrp_instance VI_2 {
state BACKUP
interface ens33
virtual_router_id 59
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.159.189
}

九、压力测试

进行压力测试可以帮助评估系统在高负载情况下的性能表现。可以使用工具如 Apache JMeter、ab、wrk 、阿里云的PTS等来模拟多用户同时访问服务器的情况，以测试系统的稳定性和吞吐量。

这里我使用的是apache自带的ab压测软件

yum install -y httpd-tools

[root@nfs-ansible-prom ~]# ab -c 1000 -n 10000 http://192.168.1.179/

This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.179 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software:
Server Hostname: 192.168.1.179
Server Port: 9090

Document Path: /
Document Length: 150 bytes

Concurrency Level: 1000
Time taken for tests: 2.066 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 2670000 bytes
HTML transferred: 1500000 bytes
Requests per second: 4839.83 [#/sec] (mean)
Time per request: 206.619 [ms] (mean)
Time per request: 0.207 [ms] (mean, across all concurrent requests)
Transfer rate: 1261.95 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 56 219.5 1 1011
Processing: 0 59 129.5 24 1893
Waiting: 0 58 129.6 23 1893
Total: 1 115 260.6 28 1934

Percentage of the requests served within a certain time (ms)
50% 28
66% 40
75% 51
80% 67
90% 314
95% 1018
98% 1062
99% 1079
100% 1934 (longest request)

在防火墙上和web1服务器上都安装glances软件

这样我们可以去获取大并发的情况下，具体那个资源消耗比较多，例如：cpu、内存、磁盘IO、网络带宽等

防火墙的资源情况

web机器

由此不难看出限制虚拟机服务器最大的是网络带宽，因为VMware和真实机联通的vmnet1网卡的最大速率只有100Mb。

十、尝试去优化整个web集群，提升性能（内核参数、nginx参数的调优）

1.内核参数调优

增加进程可以打开的文件数

[root@web-1 ~]# ulimit -n 100001

这是临时修改永久修改可以去修改配置文件 /etc/security/limits.conf

[root@web-1 ~]# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 14826
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 100001
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 14826
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

2.nginx参数调优

[root@web-1 conf]# cat nginx.conf

# 根据cpu核心的数量去修改（我的cpu核心数量是2）
worker_processes 2;

# 并发数量，同时可以允许多少人同时访问nginx
events {
worker_connections 2048;
}

http {
# 65秒后nginx会主动断开连接，可以根据自己的需求修改超时时间
keepalive_timeout 65;

}

3.多添加nginx的web模块

如文件压缩ngx_http_gzip_module、页面缓存ngx_http_headers_module、文件读取ngx_http_core_module、访问限制ngx_http_limit_req_module等等

青春猪头少年star

关注

23
点赞
踩
28

收藏

觉得还不错? 一键收藏
1
评论
综合web集群项目

提示：最近系统学习了运维方面的知识，想通过一个比较全面的项目来熟练掌握新知识本文仅是记录本人的学习记录，如有错误欢迎评论指正总参考图如下#定义监控脚本chk_nginx#当脚本/web/check_nginx.sh脚本执行返回值为0的时候，不执行下面的weight -30的操作，只有脚本执行失败，返回值非0的时候，就执行执行权重值减30的操作interval 1weight -30!#定义监控脚本chk_nginx。
复制链接

扫一扫