nginx+keepalived+tomcat配置高可用web集群
基本架构:
角色 | ip | 安装软件 | 作用 | 主机名 |
nginx主 | 192.168.247.129 | nginx+keepalived | 反向代理 | nginxmaster.com |
nginx备 | 192.168.247.130 | nginx+keepalived | 反向代理 | nginxsalve.com |
tomcat1 | 192.168.247.128 | tomcat | web服务器 | tomcat1.com |
tomcat2 | 192.168.247.131 | tomcat | web服务器 | tomcat2.com |
nfs主 | 192.168.247.132 | nfs | 低端共享存储 | nfsmaster.com |
nfs备 | 192.168.247.133 | nfs | 低端共享存储 | nfsslave.com |
虚拟ip | 192.168.247.150 |
集群部署:
tomcat安装部署:
编译安装tomcat:(tomcat1,tomcat2)
安装JDK,运行java环境:(tomcat1,tomcat2)
卸载centos自带的JDK:
rpm -qa | grep java yum -y remove java-1.6.0-openjdk-1.6.0.0-1.7.b09.el5
下载安装JDK:
##下载地址:http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html ##下载最新的(64位):jdk-7u79-linux-x64.rpm
##在/usr下建立一个java目录,以备将java程序安装在此目录下 mkdir /usr/java cd /usr/local/src rpm -ivh jdk-7u79-linux-x64.rpm ##安装后,在/usr/java目录下就会生成一个新的目录jdk1.7.0_79,该目录下存放的是安装后的文件
##设置环境变量: vi /etc/profile ##在最后添加: export JAVA_HOME=/usr/java/jdk1.7.0_79 export PATH=$PATH:$JAVA_HOME/bin export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar source /etc/profile
##验证是否安装成功: java -version ##出现如下则正常 java version "1.7.0_79" Java(TM) SE Runtime Environment (build 1.7.0_79-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
安装tomcat7:(tomcat1,tomcat2)
软件:apache-tomcat-7.0.29.tar.gz cd /usr/local/src/ tar -zxf apache-tomcat-7.0.42.tar.gz cp -a apache-tomcat-7.0.42 /usr/local/tomcat7/
/usr/local/tomcat7/bin/startup.sh ##启动tomcat,出现如下则正常 Using CATALINA_BASE: /usr/local/tomcat7 Using CATALINA_HOME: /usr/local/tomcat7 Using CATALINA_TMPDIR: /usr/local/tomcat7/temp Using JRE_HOME: /usr/java/jdk1.7.0_79 Using CLASSPATH: /usr/local/tomcat7/bin/bootstrap.jar:/usr/local/tomcat7/bin/tomcat-juli.jar ##设置开机自动启动
##访问下: ip:8080 看是否正常
配置tomcat:(tomcat1,tomcat2)
配置server.xml文件:
cd /usr/local/tomcat7/conf/ vi server.xml ##找到: <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" /> ##将其注释掉 ##添加如下(已优化): <Connector port="8080" protocol="HTTP/1.1" maxHttpHeaderSize="8192" maxThreads="1000" minSpareThreads="100" maxSpareThreads="1000" minProcessors="100" maxProcessors="1000" enableLookups="false" compression="on" compressionMinSize="2048" compressableMimeType="text/html,text/xml,text/javascript,text/css,text/plain" connectionTimeout="20000" URIEncoding="utf-8" acceptCount="1000" redirectPort="8443" disableUploadTimeout="true"/> ##配置优化: 优化JVM堆内存: vi /usr/local/tomcat7/bin/catalina.sh 添加:JAVA_OPTS='-server -Xms1024m -Xmx2048m -XX:PermSize=256M -XX:MaxNewSize=256m -XX:MaxPermSize=256m' 修改日志目录:/data/tomcat7/logs vi /usr/local/tomcat7/conf/server.xml <Valve className="org.apache.catalina.valves.AccessLogValve" directory="/data/tomcat7/logs" vi /usr/local/tomcat7/conf/logging.properties 1catalina.org.apache.juli.FileHandler.level = FINE 1catalina.org.apache.juli.FileHandler.directory = /data/tomcat7/logs 1catalina.org.apache.juli.FileHandler.prefix = catalina. 2localhost.org.apache.juli.FileHandler.level = FINE 2localhost.org.apache.juli.FileHandler.directory = /data/tomcat7/logs 2localhost.org.apache.juli.FileHandler.prefix = localhost. 3manager.org.apache.juli.FileHandler.level = FINE 3manager.org.apache.juli.FileHandler.directory = /data/tomcat7/logs 3manager.org.apache.juli.FileHandler.prefix = manager. 4host-manager.org.apache.juli.FileHandler.level = FINE 4host-manager.org.apache.juli.FileHandler.directory = /data/tomcat7/logs 4host-manager.org.apache.juli.FileHandler.prefix = host-manager. vi /usr/local/tomcat7/bin/catalina.sh if [ -z "$CATALINA_OUT" ] ; then CATALINA_OUT=/data/tomcat7/logs/catalina.out fi 修改日志输出格式: vi /usr/local/tomcat7/conf/server.xml pattern="%{X-Real-IP}i %l %u %t "%r" %s %b" /> #nginx做反向代理,%{X-Real-IP}i 获取用户的真实ip ##找到:Valve标签 ##将pattern参数的值修改为:"%{X-Real-IP}i %l %u %t "%r" %s %b" ##{X-Real-IP}i,获取的是真实客户端ip,不是代理机ip ##每个参数的含义:http://twb.iteye.com/blog/182100 ##nginx前端代理tomcat取真实客户端IP:http://www.cnphp6.com/archives/60908
##重新启动tomcat: [root@tomcat1 ~]# /usr/local/tomcat7/bin/shutdown.sh Using CATALINA_BASE: /usr/local/tomcat7 Using CATALINA_HOME: /usr/local/tomcat7 Using CATALINA_TMPDIR: /usr/local/tomcat7/temp Using JRE_HOME: /usr/java/jdk1.7.0_79 Using CLASSPATH: /usr/local/tomcat7/bin/bootstrap.jar:/usr/local/tomcat7/bin/tomcat-juli.jar [root@tomcat1 ~]# /usr/local/tomcat7/bin/startup.sh Using CATALINA_BASE: /usr/local/tomcat7 Using CATALINA_HOME: /usr/local/tomcat7 Using CATALINA_TMPDIR: /usr/local/tomcat7/temp Using JRE_HOME: /usr/java/jdk1.7.0_79 Using CLASSPATH: /usr/local/tomcat7/bin/bootstrap.jar:/usr/local/tomcat7/bin/tomcat-juli.jar
##tomcat性能优化: 参考:http://www.cnblogs.com/ggjucheng/archive/2013/04/16/3024731.html
反向代理的部署:
编译安装nginx:(主,备)
安装nginx的依赖包:
yum -y install gcc gcc-c++ autoconf automake yum -y install zlib zlib-devel openssl openssl-devel pcre-devel
编译安装pcre,使nginx支持rewrite重写:
cd /usr/local/src/ wget http://sourceforge.net/projects/pcre/files/pcre/8.35/pcre-8.35.tar.gz tar -zxf pcre-8.35.tar.gz cd pcre-8.35 ./configure make make install
安装nginx:
cd /usr/local/src/ wget http://nginx.org/download/nginx-1.6.2.tar.gz tar -zxf nginx-1.6.2.tar.gz cd nginx-1.6.2 ./configure --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-http_gzip_static_module --with-http_flv_module --with-http_gunzip_module --with-http_realip_module make make install ###注意--with-http_ssl_module,后面要用ssl
配置nginx:(主,备)
创建nginx用户:
useradd -s /sbin/nologin nginx
修改文件打开数:
ulimit -n 65535 ##设置开机自动启动
生成ssl证书(模拟)(忽略)
##创建证书存放目录: cd /usr/local/nginx/ mkdir ca cd ca ##创建服务器私钥: openssl genrsa -des3 -out server.key 1024 ##创建签名请求的证书(CSR): openssl req -new -key server.key -out server.csr ##在加载SSL支持的Nginx并使用上述私钥时除去必须的口令 cp server.key server.key.org openssl rsa -in server.key.org -out server.key ##最后标记证书使用上述私钥和CSR: openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
生成ssl证书(模拟)
##创建证书存放目录: mkdir /usr/local/nginx/ca cd /usr/local/nginx/ca ##生成密钥文件key和csr文件: openssl req -new -newkey rsa:2048 -nodes -keyout server.key -out server.csr ##生成服务器认证文件crt: openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
修改配置文件:
# mv /usr/local/nginx/conf/nginx.conf /usr/local/nginx/conf/nginx.conf.old # vi /usr/local/nginx/conf/nginx.conf user nginx nginx; worker_processes 2; error_log logs/error.log; pid logs/nginx.pid; worker_rlimit_nofile 65535; events { use epoll; worker_connections 65535; } http { include mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" $http_x_forwarded_for'; server_names_hash_bucket_size 128; client_header_buffer_size 32k; large_client_header_buffers 4 32k; sendfile on; #tcp_nopush on; tcp_nodelay on; client_header_timeout 30; client_body_timeout 30; send_timeout 30; client_max_body_size 100M; keepalive_timeout 60; proxy_connect_timeout 100; proxy_send_timeout 100; proxy_read_timeout 100; proxy_buffer_size 16k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; gzip on; gzip_min_length 1k; gzip_buffers 4 16k; gzip_http_version 1.1; gzip_comp_level 2; gzip_types text/plain application/x-javascript text/css application/xml text/javascript; gzip_vary on; upstream web1{ ip_hash; server 192.168.247.128:8080 max_fails=3 fail_timeout=30s; server 192.168.247.131:8080 max_fails=3 fail_timeout=30s; } server { listen 443; server_name www.scj.com; root /opt/nginx/www/www.scj.com; index index.html index.htm index.jsp; ssl on; ssl_certificate /usr/local/nginx/ca/server.crt; ssl_certificate_key /usr/local/nginx/ca/server.key; access_log logs/www.scj.com/access.log main; error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } location ~ .*\.jsp$ { proxy_pass http://web1; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_next_upstream http_502 http_504 error timeout invalid_header; } location ~ .*\.(gif|jpg|png|swf|bmp|jpeg)$ { expires 10d; } location ~ .*\.(js|css)?$ { expires 1h; } } #server { #listen 80 default; #server_name _; #return 500; #} }
##创建日志目录和根目录 mkdir /usr/local/nginx/logs/www.scj.com ##日志目录 mkdir -p /opt/nginx/www/www.scj.com ##创建根目录
##启动nginx: # /usr/local/nginx/sbin/nginx # ps -ef | grep nginx root 7890 1 0 08:00 ? 00:00:00 nginx: master process /usr/local/nginx/sbin/nginx nginx 7891 7890 0 08:00 ? 00:00:00 nginx: worker process nginx 7892 7890 0 08:00 ? 00:00:00 nginx: worker process root 7896 7798 0 08:00 pts/0 00:00:00 grep nginx # netstat -tlnpa | grep nginx tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 7890/nginx ##设置开机自动启动
部署keepalived,实现自动切换:
安装keepalived:(主,备)
cd /usr/local/src/ wget http://www.keepalived.org/software/keepalived-1.2.15.tar.gz tar -zxf keepalived-1.2.15.tar.gz cd keepalived-1.2.15 ./configure --prefix=/usr/local/keepalived make make install
拷贝文件:(主,备)
cp -a /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/ cp -a /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ mkdir /etc/keepalived/ cp -a /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/ cp -a /usr/local/keepalived/sbin/keepalived /usr/sbin/ ##注意: /etc/sysconfig/keepalived 和 /etc/keepalived/keepalived.conf 的路径一定要正确, ##因为在执行/etc/init.d/keepalived这个启动脚本时,会读取/etc/sysconfig/keepalived 和 /etc/keepalived/keepalived.conf 这两个文件
修改配置文件:
nginx主:
[root@nginxmaster ~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.old [root@nginxmaster ~]# vi /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { 732233048@qq.com } notification_email_from root@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id nginx_ha } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 51 priority 150 advert_int 1 #nopreempt authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.247.150 } } virtual_server 192.168.247.150 443 { delay_loop 6 #lb_algo wrr #lb_kind DR #persistence_timeout 50 protocol TCP real_server 192.168.247.129 443 { #weight 3 notify_down /etc/keepalived/killkeepalived.sh TCP_CHECK { connect_timeout 10 nb_get_retry 3 delay_before_retry 3 connect_port 443 } } }
nginx备:
[root@nginxslave ~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.old [root@nginxslave ~]# vi /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { 732233048@qq.com } notification_email_from root@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id nginx_ha } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 51 priority 100 advert_int 1 #nopreempt authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.247.150 } } virtual_server 192.168.247.150 443 { delay_loop 6 #lb_algo wrr #lb_kind DR #persistence_timeout 50 protocol TCP real_server 192.168.247.130 443 { #weight 3 notify_down /etc/keepalived/killkeepalived.sh TCP_CHECK { connect_timeout 10 nb_get_retry 3 delay_before_retry 3 connect_port 443 } } }
创建/etc/keepalived/killkeepalived.sh脚本:(主,备)
vi /etc/keepalived/killkeepalived.sh #!/bin/bash # check nginx server status nginx_status=`netstat -tlnpa | grep 443 | wc -l` if [ $nginx_status -eq 0 ];then /usr/local/nginx/sbin/nginx sleep 1 nginx_status=`netstat -tlnpa | grep 443 | wc -l` if [ $nginx_status -eq 0 ];then /etc/init.d/keepalived stop fi fi chmod 755 /etc/keepalived/killkeepalived.sh
修改keepalived的日志文件:(主,备)
参考:http://chenwenming.blog.51cto.com/327092/745316
说明:
##centos6.3之后的syslog改名叫rsyslog了,默认在 /etc/rsyslog.conf
修改/etc/sysconfig/keepalived:
vi /etc/sysconfig/keepalived # Options for keepalived. See `keepalived --help' output and keepalived(8) and # keepalived.conf(5) man pages for a list of all options. Here are the most # common ones : # # --vrrp -P Only run with VRRP subsystem. # --check -C Only run with Health-checker subsystem. # --dont-release-vrrp -V Dont remove VRRP VIPs & VROUTEs on daemon stop. # --dont-release-ipvs -I Dont remove IPVS topology on daemon stop. # --dump-conf -d Dump the configuration data. # --log-detail -D Detailed log messages. # --log-facility -S 0-7 Set local syslog facility (default=LOG_DAEMON) # #KEEPALIVED_OPTIONS="-D" KEEPALIVED_OPTIONS="-D -d -S 0" ##在最后添加此行
修改/etc/rsyslog.conf:
vi /etc/rsyslog.conf ##在最后添加此行: local0.* /var/log/keepalived.log ## /etc/init.d/rsyslog restart Shutting down system logger: [ OK ] Starting system logger: [ OK ]
启动keepalived:
nginx主:
[root@nginxmaster ~]# /etc/init.d/keepalived start Starting keepalived: [ OK ]
##查看日志: Sep 23 20:16:27 nginxmaster Keepalived[2909]: Starting Keepalived v1.2.15 (09/23,2015) Sep 23 20:16:27 nginxmaster Keepalived[2910]: Starting Healthcheck child process, pid=2912 Sep 23 20:16:27 nginxmaster Keepalived[2910]: Starting VRRP child process, pid=2913 Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Netlink reflector reports IP 192.168.247.129 added Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Netlink reflector reports IP fe80::20c:29ff:fee0:d236 added Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Registering Kernel netlink reflector Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Registering Kernel netlink command channel Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Registering gratuitous ARP shared channel Sep 23 20:16:27 nginxmaster kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP) Sep 23 20:16:27 nginxmaster kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes) Sep 23 20:16:27 nginxmaster kernel: IPVS: ipvs loaded. Sep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Netlink reflector reports IP 192.168.247.129 added Sep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Netlink reflector reports IP fe80::20c:29ff:fee0:d236 added Sep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Registering Kernel netlink reflector Sep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Registering Kernel netlink command channel Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: Opening file '/etc/keepalived/keepalived.conf'. Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: Configuration is using : 63272 Bytes Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: Using LinkWatch kernel netlink reflector... Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Entering BACKUP STATE Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)] Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Opening file '/etc/keepalived/keepalived.conf'. Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Configuration is using : 11705 Bytes Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: IPVS: Scheduler or persistence engine not found Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: IPVS: No such process Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Using LinkWatch kernel netlink reflector... Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Activating healthchecker for service [192.168.247.129]:443 Sep 23 20:16:28 nginxmaster kernel: IPVS: Scheduler module ip_vs_ not found Sep 23 20:16:31 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Transition to MASTER STATE Sep 23 20:16:32 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Entering MASTER STATE Sep 23 20:16:32 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) setting protocol VIPs. Sep 23 20:16:32 nginxmaster Keepalived_healthcheckers[2912]: Netlink reflector reports IP 192.168.247.150 added Sep 23 20:16:32 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150 Sep 23 20:16:37 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150 ##设置开机自动启动 chkconfig keepalived on
nginx备:
[root@nginxslave ~]# /etc/init.d/keepalived start Starting keepalived: [ OK ]
##查看日志: Sep 24 09:43:29 nginxslave Keepalived[9481]: Starting Keepalived v1.2.15 (09/24,2015) Sep 24 09:43:29 nginxslave Keepalived[9482]: Starting Healthcheck child process, pid=9484 Sep 24 09:43:29 nginxslave Keepalived[9482]: Starting VRRP child process, pid=9485 Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Netlink reflector reports IP 192.168.247.130 added Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Netlink reflector reports IP fe80::20c:29ff:fe1e:8eab added Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Registering Kernel netlink reflector Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Registering Kernel netlink command channel Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Registering gratuitous ARP shared channel Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Opening file '/etc/keepalived/keepalived.conf'. Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Configuration is using : 63262 Bytes Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Using LinkWatch kernel netlink reflector... Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: VRRP_Instance(VI_1) Entering BACKUP STATE Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)] Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Netlink reflector reports IP 192.168.247.130 added Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Netlink reflector reports IP fe80::20c:29ff:fe1e:8eab added Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Registering Kernel netlink reflector Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Registering Kernel netlink command channel Sep 24 09:43:29 nginxslave kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP) Sep 24 09:43:29 nginxslave kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes) Sep 24 09:43:29 nginxslave kernel: IPVS: ipvs loaded. Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Opening file '/etc/keepalived/keepalived.conf'. Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Configuration is using : 11695 Bytes Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: IPVS: Scheduler or persistence engine not found Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: IPVS: No such process Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Using LinkWatch kernel netlink reflector... Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Activating healthchecker for service [192.168.247.130]:443 Sep 24 09:43:29 nginxslave kernel: IPVS: Scheduler module ip_vs_ not found ##设置开机自动启动 chkconfig keepalived on
查看vip绑定在哪台机器上:
[root@nginxmaster ~]#ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:e0:d2:36 brd ff:ff:ff:ff:ff:ff inet 192.168.247.129/24 brd 192.168.247.255 scope global eth0 inet 192.168.247.150/32 scope global eth0 inet6 fe80::20c:29ff:fee0:d236/64 scope link valid_lft forever preferred_lft forever
[root@nginxslave ~]#ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:1e:8e:ab brd ff:ff:ff:ff:ff:ff inet 192.168.247.130/24 brd 192.168.247.255 scope global eth0 inet6 fe80::20c:29ff:fe1e:8eab/64 scope link valid_lft forever preferred_lft forever
##由上发现vip150绑定在nginxmaster,正常
测试:
测试一:把nginx主的nginx服务stop掉
[root@nginxmaster ~]# /usr/local/nginx/sbin/nginx -s stop
##查看日志: Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: TCP connection to [192.168.247.129]:443 failed !!! Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Removing service [192.168.247.129]:443 from VS [192.168.247.150]:443 Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: IPVS: Service not defined Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Executing [/etc/keepalived/killkeepalived.sh] for service [192.168.247.129]:443 in VS [192.168.247.150]:443 Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Lost quorum 1-0=1 > 0 for VS [192.168.247.150]:443 Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Remote SMTP server [127.0.0.1]:25 connected. Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: SMTP alert successfully sent. Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: TCP connection to [192.168.247.129]:443 success. Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: Adding service [192.168.247.129]:443 to VS [192.168.247.150]:443 Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: IPVS: Service not defined Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: Gained quorum 1+0=1 <= 1 for VS [192.168.247.150]:443 Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: Remote SMTP server [127.0.0.1]:25 connected. Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: SMTP alert successfully sent ##keepalived会先把129移除,然后执行/etc/keepalived/killkeepalived.sh脚本,nginx服务正常启动,又把129加入到集群中
测试二:把nginx主的keepalived服务stop掉
[root@nginxmaster ~]# /etc/init.d/keepalived stop Stopping keepalived: [ OK ]
##查看nginx主的日志: Sep 23 23:41:05 nginxmaster Keepalived[3988]: Stopping Keepalived v1.2.15 (09/23,2015) Sep 23 23:41:05 nginxmaster Keepalived_vrrp[3991]: VRRP_Instance(VI_1) sending 0 priority Sep 23 23:41:05 nginxmaster Keepalived_vrrp[3991]: VRRP_Instance(VI_1) removing protocol VIPs. Sep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: Netlink reflector reports IP 192.168.247.150 removed Sep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: Removing service [192.168.247.129]:443 from VS [192.168.247.150]:443 Sep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: IPVS: Service not defined Sep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: IPVS: No such service
##查看nginx备日志: Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: TCP connection to [192.168.247.130]:443 success. Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: Adding service [192.168.247.130]:443 to VS [192.168.247.150]:443 Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: IPVS: Service not defined Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: Gained quorum 1+0=1 <= 1 for VS [192.168.247.150]:443 Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: Remote SMTP server [127.0.0.1]:25 connected. Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: SMTP alert successfully sent. Sep 24 12:58:05 nginxslave dhclient[906]: DHCPREQUEST on eth0 to 192.168.247.254 port 67 (xid=0x10a73226) Sep 24 12:58:05 nginxslave dhclient[906]: DHCPACK from 192.168.247.254 (xid=0x10a73226) Sep 24 12:58:07 nginxslave dhclient[906]: bound to 192.168.247.130 -- renewal in 705 seconds. Sep 24 13:05:23 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Transition to MASTER STATE Sep 24 13:05:24 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Entering MASTER STATE Sep 24 13:05:24 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) setting protocol VIPs. Sep 24 13:05:24 nginxslave Keepalived_healthcheckers[10012]: Netlink reflector reports IP 192.168.247.150 added Sep 24 13:05:24 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150 Sep 24 13:05:29 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150 ##nginx备变成master,绑定vip 150
测试三:把nginx主的nginx服务stop,并不再让它成功启动
##删掉日志目录,nginx便不能启动了 [root@nginxmaster logs]# rm -rf /usr/local/nginx/logs/www.scj.com/
##关闭nginx服务 [root@nginxmaster ~]# /usr/local/nginx/sbin/nginx -s stop
##查看nginx主日志: Sep 23 23:44:33 nginxmaster Keepalived_vrrp[4079]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150 Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: TCP connection to [192.168.247.129]:443 failed !!! Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Removing service [192.168.247.129]:443 from VS [192.168.247.150]:443 Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: IPVS: Service not defined Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Executing [/etc/keepalived/killkeepalived.sh] for service [192.168.247.129]:443 in VS [192.168.247.150]:443 Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Lost quorum 1-0=1 > 0 for VS [192.168.247.150]:443 Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Remote SMTP server [127.0.0.1]:25 connected. Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: SMTP alert successfully sent. Sep 23 23:44:39 nginxmaster Keepalived[4076]: Stopping Keepalived v1.2.15 (09/23,2015) Sep 23 23:44:39 nginxmaster Keepalived_vrrp[4079]: VRRP_Instance(VI_1) sending 0 priority Sep 23 23:44:39 nginxmaster Keepalived_vrrp[4079]: VRRP_Instance(VI_1) removing protocol VIPs. Sep 23 23:44:39 nginxmaster Keepalived_healthcheckers[4078]: Netlink reflector reports IP 192.168.247.150 removed Sep 23 23:44:39 nginxmaster Keepalived_healthcheckers[4078]: IPVS: No such service ##nginx服务不能启动了,然后把keepalived服务关掉
##查看nginx备日志 Sep 24 13:08:57 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Transition to MASTER STATE Sep 24 13:08:58 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Entering MASTER STATE Sep 24 13:08:58 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) setting protocol VIPs. Sep 24 13:08:58 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150 Sep 24 13:08:58 nginxslave Keepalived_healthcheckers[10012]: Netlink reflector reports IP 192.168.247.150 added Sep 24 13:09:03 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150 Sep 24 13:09:52 nginxslave dhclient[906]: DHCPREQUEST on eth0 to 192.168.247.254 port 67 (xid=0x10a73226) Sep 24 13:09:52 nginxslave dhclient[906]: DHCPACK from 192.168.247.254 (xid=0x10a73226) Sep 24 13:09:54 nginxslave dhclient[906]: bound to 192.168.247.130 -- renewal in 800 seconds. ##nginx备变为master,并绑定vip 150
测试四:把nginx备的nginx服务stop
##查看nginx备的nginx服务会不会又正常启动
测试五:把nginx备的keepalived服务stop
##nginx主不会有任何变化
测试六:把nginx备的nginx服务stop掉,并不再让它正常启动
##nginx备的keepalived服务会被关掉 ##nginx主不会有任何变化
注意:
修复后,一定要记得把keepalived服务启动起来
nfs部署:
http://732233048.blog.51cto.com/9323668/1669417
转载于:https://blog.51cto.com/732233048/1697705