Nginx不仅是一款优秀的WEB服务器,同时可以根据nginx的反代理可以配置成强大的负载均衡器.这里就介绍如何把nginx配置成负载均衡器,并结合keepalived配置高可用的集群.
一般集群主要架构为:
前端为负载均衡器两个:主/备,两种工作方式,一种是备机待机状态,主机故障时备机接管主机工作实现故障庄毅,在主机故障恢复完成时备机继续仅需待机状态,第二种是主备同时工作,一台宕机另外一台自动接管另一台的工作实现故障转移.
第一种方式可以通过将域名解析到一个虚拟ip(vip)上,主负载均衡器绑定虚拟ip,当主负载均衡器出现故障时,通过keepalived自动将vip绑定到备用负载均衡器上同时arping网关刷新MAC地址.,避免单点故障.
第二种方式主备同时绑定一个vip,把域名通过DNS轮询的方式解析到这两个服务器上,主机出现故障,备机就将主机绑定vip绑定到备机上,同时arping网关刷新MAC地址.实现故障转移.
中间为WEB服务器作为real server,处理请求.
后端为数据库和分布式文件系统.数据库一般为主从两台.分布式文件系统有效解决WEB服务器之间的数据同步.有的还会将图片服务器单独分离出来放在后端.
本文使用环境:
CentOS 5.5 32位
nginx:nginx-1.0.11
keepalived：keepalived-1.1.19.tar.gz
主调度器:192.168.3.1
备调度器:192.168.3.2
real server:192.168.3.4/5/6
本文采用第一种方式来进行vip为:192.168.3.253
一、在主备服务器上部署nginx
1.下载
2.安装
1 | yum -y install zlib-devel pcre-devel openssl-devel # 安装依赖 |
2 | tar -zxvf nginx-1.0.11.tar.gz |
3 | cd nginx-1.0.11 |
4 | ./configure --prefix=/usr/local/nginx --with-http_ssl_module --with-http_flv_module --with-http_gzip_static_module |
5 | make && make install |
3.配置
配置主调度器的nginx,编辑nginx.conf
01 | vi /usr/local/nginx/conf/nginx.conf |
02 |
03 | http { |
04 | include mime.types; |
05 | default_type application/octet-stream; |
06 |
07 | #log_format main '$remote_addr - $remote_user [$time_local] "$request" ' |
08 | # '$status $body_bytes_sent "$http_referer" ' |
09 | # '"$http_user_agent" "$http_x_forwarded_for"'; |
10 |
11 | #access_log logs/access.log main; |
12 |
13 | sendfile on; |
14 | #tcp_nopush on; |
15 |
16 | #keepalive_timeout 0; |
17 | keepalive_timeout 65; |
18 |
19 | #gzip on; |
20 |
21 | # 添加一组真实的服务器地址池 |
22 | # 供proxy_pass和fastcgi_pass指令中使用的代理服务器 |
23 | upstream real_server_pool { |
24 | # 后台如果有动态应用的时候,ip_hash指令可以通过hash算法 |
25 | # 将客户端请求定位到同一台后端服务器上,解决session共享, |
26 | # 但建议用动态应用做session共享 |
27 | # ip_hash; |
28 |
29 | # server用于指定一个后端服务器的名称和参数 |
30 | # weight代表权,重默认为1,权重越高被分配的客户端越多 |
31 | # max_fails 指定时间内对后端请求失败的次数 |
32 | # fail_timeout 达到max_fails指定的失败次数后暂停的时间 |
33 | server 192.168.3.4:80 weight=1 max_fails=2 fail_timeout=30s; |
34 | # down参数用来标记为离线,不参与负载均衡.在ip_hash下使用 |
35 | # 在此做演示,后面测试会去掉 |
36 | server 192.168.3.5:80 weight=1 max_fails=2 fail_timeout=30s down; |
37 | # backup仅仅在非backup服务器宕机或繁忙的时候使用 |
38 | # (在此做演示,后面测试会去掉) |
39 | server 192.168.3.6:80 weight=1 max_fails=2 fail_timeout=30s backup; |
40 | } |
41 | server { |
42 | listen 192.168.3.1:80; |
43 | server_name localhost; |
44 |
45 | #charset koi8-r; |
46 |
47 | #access_log logs/host.access.log main; |
48 |
49 | location / { |
50 | #root html; |
51 | #index index.html index.htm; |
52 | # 使用upstream设置的一组代理服务器 |
53 | # 如果后端服务器出现502或504等执行错误时, |
54 | # 将自动将请求转发给负载均衡池中的另一台服务器. |
55 | proxy_next_upstream http_502 http_504 error timeout invalid_header; |
56 | proxy_pass http://real_server_pool; |
57 | proxy_set_header Host $host; |
58 | proxy_set_header X-Forwarded-For $remote_addr; |
59 | } |
60 | } |
61 | } |
(注意:配置文件中注释ip_hash,以为ip_hash这个功能将保证这个客户端请求总是被转发到一台服务器上,所以如果启用了ip_hash指令,将不能再使用weight(权重参数),配置文件中加入为解释ip_hash指令)
配置备用nginx,将监听ip改为备用调度器的ip
01 | http { |
02 | include mime.types; |
03 | default_type application/octet-stream; |
04 |
05 | #log_format main '$remote_addr - $remote_user [$time_local] "$request" ' |
06 | # '$status $body_bytes_sent "$http_referer" ' |
07 | # '"$http_user_agent" "$http_x_forwarded_for"'; |
08 |
09 | #access_log logs/access.log main; |
10 |
11 | sendfile on; |
12 | #tcp_nopush on; |
13 |
14 | #keepalive_timeout 0; |
15 | keepalive_timeout 65; |
16 |
17 | #gzip on; |
18 |
19 | upstream real_server_pool { |
20 | #ip_hash; |
21 | server 192.168.3.4:80 weight=1 max_fails=2 fail_timeout=30s; |
22 | server 192.168.3.5:80 weight=1 max_fails=2 fail_timeout=30s; |
23 | server 192.168.3.6:80 weight=1 max_fails=2 fail_timeout=30s; |
24 | } |
25 | server { |
26 | listen 192.168.3.2:80; # 监听ip改为本地ip |
27 | server_name localhost; |
28 |
29 | #charset koi8-r; |
30 |
31 | #access_log logs/host.access.log main; |
32 |
33 | location / { |
34 | #root html; |
35 | #index index.html index.htm; |
36 | proxy_next_upstream http_502 http_504 error timeout invalid_header; |
37 | proxy_pass http://real_server_pool; |
38 | proxy_set_header Host $host; |
39 | proxy_set_header X-Forwarded-For $remote_addr; |
40 | } |
然后启动主备nginx:
1 | /usr/local/nginx/sbin/nginx |
二、在主备服务器上部署keepalived
安装
安装依赖:
1 | yum -y install kernel-devel # 安装依赖 |
开启路由转发:
1 | vi /etc/sysctl.conf |
2 | net.ipv4.ip_forward = 1 # 此参数改为1 |
3 | sysctl -p # 使修改生效 |
首先安装ipvs:
1 | ln -s /usr/src/kernels/2.6.18-194.el5-i686/ /usr/src/linux # ipvs需要内核文件,做一个软连接 |
2 | # 下载 |
4 | tar -zxvf ipvsadm-1.24.tar.gz |
5 | cd ipvsadm-1.24 |
6 | make |
7 | make install |
然后安装keepalived
1 | # 下载 |
3 | tar -zxvf keepalived-1.1.19.tar.gz |
4 | cd keepalived-1.1.19 |
5 | ./configure --prefix=/ \ # 安装在默认位置(配置文件,二进制文件,启动脚本放到默认位置) |
6 | --mandir=/usr/local/share/man/ \ |
7 | --with-kernel-dir=/usr/src/kernels/2.6.18-194.el5-i686/ # 需要内核的头文件 |
8 | make && make install |
配置keepalived
编辑主调度器配置文件/etc/keepalived/keepalived.conf
01 | global_defs { |
02 | notification_email { |
03 | cold_night@linuxzen.com # 定义通知邮箱,有多个可以换行添加 |
04 | } |
05 | notification_email_from root@linuxzen.com# 定义发送邮件的邮箱 |
06 | smtp_server www.linuxzen.com # 定义发件服务器 |
07 | smtp_connect_timeout 30 # 定义连接smtp服务器超时时间 |
08 | router_id LVS_DEVEL |
09 | } |
10 |
11 | vrrp_instance VI_1 { |
12 | state MASTER # 标示主备,备机上改为BACKUP |
13 | interface eth0 # HA监测的端口 |
14 | virtual_router_id 51 # 主备的virtual_router_id的值必须相同 |
15 | priority 100 # 优先级,通常主要比备稍大 |
16 | advert_int 1 # VRRP Multicast 广播周期秒数 |
17 | authentication { # 定义认证 |
18 | auth_type PASS # 认证方式 |
19 | auth_pass 1111 # 认证口令字 |
20 | } |
21 | virtual_ipaddress { # 定义vip |
22 | 192.168.3.253 # 多个可换行添加,一行一个 |
23 | } |
24 | } |
25 |
26 | virtual_server 192.168.3.253 80 { |
27 | delay_loop 6 # 每隔 6 秒查询 realserver 状态 |
28 | lb_algo rr |
29 | lb_kind NAT |
30 | nat_mask 255.255.255.0 |
31 | persistence_timeout 50 # 同一IP 的连接50秒内被分配到同一台realserver |
32 | protocol TCP # 用TCP监测realserver的状态 |
33 |
34 | real_server 192.168.3.1 80 { |
35 | weight 3 # 权重 |
36 | TCP_CHECK { |
37 | connect_timeout 10 # 10秒无响应超时 |
38 | nb_get_retry 3 |
39 | delay_before_retry 3 |
40 | connect_port 80 |
41 | } |
42 | } |
43 |
44 | real_server 192.168.3.2 80 { |
45 | weight 3 |
46 | TCP_CHECK { |
47 | connect_timeout 3 |
48 | delay_before_retry 3 |
49 | connect_port 80 |
50 | } |
51 | } |
52 | } |
配置备用调度器的keepalived,只需要将state MASTER 改为state BACKUP,降低priority 100 的值:
01 | global_defs { |
02 | notification_email { |
03 | cold_night@linuxzen.com |
04 | } |
05 | notification_email_from root@linuxzen.com |
06 | smtp_server www.linuxzen.com |
07 | smtp_connect_timeout 30 |
08 | router_id LVS_DEVEL |
09 | } |
10 |
11 | vrrp_instance VI_1 { |
12 | state BACKUP # 备机上改为BACKUP |
13 | interface eth0 |
14 | virtual_router_id 51 # 主备的virtual_router_id的值必须相同 |
15 | priority 99 # 备用优先级小于主调度器 |
16 | advert_int 1 |
17 | authentication { |
18 | auth_type PASS |
19 | auth_pass 1111 |
20 | } |
21 | virtual_ipaddress { |
22 | 192.168.3.253 |
23 | } |
24 | } |
25 |
26 | virtual_server 192.168.3.253 80 { |
27 | delay_loop 6 |
28 | lb_algo rr |
29 | lb_kind NAT |
30 | nat_mask 255.255.255.0 |
31 | persistence_timeout 50 |
32 | protocol TCP |
33 |
34 | real_server 192.168.3.1 80 { |
35 | weight 3 |
36 | TCP_CHECK { |
37 | connect_timeout 10 |
38 | nb_get_retry 3 |
39 | delay_before_retry 3 |
40 | connect_port 80 |
41 | } |
42 | } |
43 |
44 | real_server 192.168.3.2 80 { |
45 | weight 3 |
46 | TCP_CHECK { |
47 | connect_timeout 3 |
48 | delay_before_retry 3 |
49 | connect_port 80 |
50 | } |
51 | } |
52 | } |
主备上启动keepalived:
1 | service keepalived start |
三、测试—–部署后端服务器
在后端服务器安装nginx,这里仅部署一台然后创建3个基于ip的虚拟主机供测试:
绑定ip:
1 | ifconfig eth0:1 192.168.3.4/24 |
2 | ifconfig eth0:2 192.168.3.5/24 |
3 | ifconfig eth0:3 192.168.3.6/24 |
安装nginx后编辑配置文件,在http块里添加:
01 | http { |
02 | server { |
03 | listen 192.168.3.4:80; |
04 | server_name 192.168.3.4; |
05 |
06 | location / { |
07 | root html/s1; |
08 | index index.html index.htm; |
09 | } |
10 | } |
11 |
12 | server { |
13 | listen 192.168.3.5:80; |
14 | server_name 192.168.3.5; |
15 |
16 | location / { |
17 | root html/s2; |
18 | index index.html index.htm; |
19 | } |
20 | } |
21 |
22 | server { |
23 | listen 192.168.3.6:80; |
24 | server_name 192.168.3.5; |
25 |
26 | location / { |
27 | root html/s3; |
28 | index index.html index.htm; |
29 | } |
30 | } |
31 | } |
创建虚拟主机根目录,并创建不通的首页文档:
1 | cd /usr/local/nginx/html/ |
2 | mkdir s1 s2 s3 |
3 | echo server1 > s1/index.html |
4 | echo server2 > s2/index.html |
5 | echo server3 > s3/index.html |
启动nginx:
1 | /usr/local/nginx/sbin/nginx |
打开浏览器访问http://192.168.3.253
刷新会看到显示不同的内容:server1,server2,server3(生产中的服务器应该是一样的)
现在停掉主调度器的keepalived
1 | pkill keepalived |
查看备调度器的日志:
1 | cat /var/log/messages |
2 | Feb 10 16:36:27 cfhost Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE |
3 | Feb 10 16:36:28 cfhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE |
4 | Feb 10 16:36:28 cfhost Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs. |
5 | Feb 10 16:36:28 cfhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.3.253 |
6 | Feb 10 16:36:28 cfhost Keepalived_vrrp: Netlink reflector reports IP 192.168.3.253 added |
7 | Feb 10 16:36:28 cfhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.3.253 added |
8 | Feb 10 16:36:33 cfhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.3.253 |
现在访问http://192.168.3.253依然可以访问.
大家也看到了备机keepalived只有检测主机的keepalived停止的时候才会切换vip,而不是检测一台real server的某一服务(比如检测80端口的HTTP)切换vip,所以在nginx进程停止的时候,如果服务器没有宕机这时候就无法实现故障转移,所以我们编写一个检测nginx状态的脚本结合keepalived实现故障转移:
01 | #!/bin/bash |
02 | #filename:nsc.sh |
03 | ps aux ¦ grep nginx ¦ grep -v grep 2> /dev/null 1>&2 # 过滤nginx进程 |
04 | if [[ $? -eq 0 ]] # 如果过滤有nginx进程会返回0则认为nginx存活 |
05 | then |
06 | sleep 5 # 使脚本进入休眠 |
07 | else |
08 | # 如果nginx没有存活尝试启动nginx,如果失败则杀死keepalived的进程 |
09 | /usr/local/nginx/sbin/nginx |
10 | ps aux ¦ grep nginx ¦ grep -v grep 2> /dev/null 1>&2 |
11 | if [[ $? -eq 0 ]] |
12 | then |
13 | pkill keepalived |
14 | fi |
15 | fi |
然后后台运行此脚本:
1 | nohup sh nsc.sh & |
这样就实现了群集的高可靠和高可用.
13万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
