关于Nginx+tomcat+memcached集群配置
-
前言
本文档主要采用Nignx+tomcat+memcached,实现负载均衡和tomcat集群配置,并解决tomcat之间session共享问题。
前一篇博文是通过tomcat中自带的Cluster方式进行配置。而本文档主要采取Nginx+tomcat+memcached的方式实现。
本方案中,session共享问题,采取第三方解决方案memcached。即节点中的所有session全部存储到memcached中。
MSM(memcached-session-manager) 支持tomcat6 和tomcat7 ,利用 Value(Tomcat 阀)对Request进行跟踪。Request请求到来时,从memcached加载session,Request请求结束时,将tomcat session更新至memcached,以达到session共享之目的, 支持 sticky 和 non-sticky 模式。
Sticky 模式:tomcat session 为 主session, memcached 为备 session。Request请求到来时, 从memcached加载备 session 到 tomcat (仅当tomcat jvmroute发生变化时,否则直接取tomcat session);Request请求结束时,将tomcat session更新至memcached,以达到主备同步之目的。
Non-Sticky模式:tomcat session 为 中转session, memcached1 为主 sessionmemcached 2 为备session。Request请求到来时,从memcached 2加载备 session 到 tomcat,(当 容器 中还是没有session 则从memcached1加载主 session 到 tomcat, 这种情况是只有一个memcached节点,或者有memcached1 出错时),Request请求结束时,将tomcat session更新至 主memcached1和备memcached2,并且清除tomcat session 。以达到主备同步之目的。
-
配置准备
本案例中,主要采取三台虚拟机实现集群配置,具体配置如下。
序号
|
机器名
|
系统
|
内存
|
应用
|
版本
|
端口
|
IP
|
1
|
nginx
|
ubuntu 16.04 LTS
|
1G
|
nginx
|
1.9.9
|
80
|
192.168.80.130
|
2
|
tomcat1
|
ubuntu 16.04 LTS
|
1G
|
tomcat
|
7.0.72
|
8080
|
192.168.80.131
|
3
|
tomcat1
|
ubuntu 16.04 LTS
|
1G
|
tomcat
|
7.0.72
|
8080
|
192.168.80.132
|
4
|
nginx
|
ubuntu 16.04 LTS
|
1G
|
memcached
|
11211
|
192.168.80.130
|
其中JDK版本:1.8.0_101。在ubuntu中安装memcached十分简单,直接通过apt-get安装。其余各个软件的安装与环境配置此处不再赘述。
-
配置安装
-
Nginx负载均衡配置
此处将192.168.80.130作为Nginx节点。192.168.80.131、192.168.80.132作为tomcat节点。实现负载均衡。即修改nginx的配置文件:nginx.conf。修改后的配置文件如下所示。
#user www www;
worker_processes 1;
#pid /usr/local/nginx/nginx.pid;
worker_rlimit_nofile 1024;
events
{
use epoll;
worker_connections 1024;
}
http
{
include mime.types;
default_type application/octet-stream;
fastcgi_intercept_errors on;
charset utf-8;
server_names_hash_bucket_size 128;
client_header_buffer_size 4k;
large_client_header_buffers 4 32k;
client_max_body_size 300m;
sendfile on;
tcp_nopush on;
keepalive_timeout 65;
tcp_nodelay on;
client_body_buffer_size 512k;
proxy_connect_timeout 5;
proxy_read_timeout 60;
proxy_send_timeout 5;
proxy_buffer_size 16k;
proxy_buffers 4 64k;
proxy_busy_buffers_size 128k;
proxy_temp_file_write_size 128k;
gzip on;
gzip_min_length 1k;
gzip_buffers 4 16k;
gzip_http_version 1.1;
gzip_comp_level 2;
gzip_types text/plain application/x-javascript text/css application/xml;
gzip_vary on;
###2012-12-19 change nginx logs
log_format main '$http_x_forwarded_for - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $request_time $remote_addr';
upstream www.test.com {
server 192.168.80.131:8080 weight=1 max_fails=2 fail_timeout=30s;
server 192.168.80.132:8080 weight=1 max_fails=2 fail_timeout=30s;
}
####chinaapp.sinaapp.com
server {
listen 80;
server_name www.test.com;
index index.jsp index.html index.htm;
#发布目录/data/www
root /data/www;
location /
{
proxy_next_upstream http_502 http_504 error timeout invalid_header;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://www.test.com; #此处需要和upstream保持一致
expires 3d;
}
}
}
此处配置参数具体信息,暂未弄清楚。日后详细查阅。其中upstream为负载均衡节点配置。
-
Tomcat集群配置
此处集群可以修改server.xml和context.xml中配置信息。此处我修改了context.xml中的配置信息。具体配置信息如下所示。
<?xml version='1.0' encoding='utf-8'?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- The contents of this file will be loaded for each web application -->
<Context>
<!-- Default set of monitored resources -->
<WatchedResource>WEB-INF/web.xml</WatchedResource>
<!-- Uncomment this to disable session persistence across Tomcat restarts -->
<!--
<Manager pathname="" />
-->
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:192.168.80.134:11211"
sticky="false"
sessionBackupAsync="false"
lockingMode="auto"
sessionBackupTimeout="100"
copyCollectionForSerialization="true"
memcachedProtocol="binary"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"
/>
<!-- Uncomment this to enable Comet connection tacking (provides events
on session expiration as well as webapp lifecycle) -->
<!--
<Valve className="org.apache.catalina.valves.CometConnectionManagerValve" />
-->
</Context>
其中黄色部分为Non-Sticky模式的配置。其中Sticky配置直接将如上配置修改成如下配置
<Manager className=" de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:192.168.80.134:11211 n2:localhost:11212"
failoverNodes="n1"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"
/>
如果配置两个memcached节点时,直接采用空格隔开即可。如下所示。
memcachedNodes="n1:127.0.0.1:11211 n2:127.0.0.1:11211"
为了方便访问。直接将tomcat的默认项目修改。修改配置如下所示。
<Context path="/" docBase="oems" reloadable="false"></Context>
-
导入JAR包
Tomcat要支持memcached管理Session,导入jar包的版本信息如下所示:
特别要注意,这些jar包的版本,要是版本出现问题。会出现各种错误。影响开发进度等。通过各种资料查询。现阶段网上的大部分的资料的版本数据均由不同的问题。若有需要请下载lib.zip
4.测试
Server Info:
SessionID:<%=session.getId()%><br>
SessionIP:<%=request.getServerName()%><br>
<br>SessionPort:<%=request.getServerPort()%>
<%out.println("111"); //标记后端节点%>
IP为192.168.80.132中的tomcat中webapp的ROOT项目中放入test.jsp,其中jsp的body标签中放入如下代码。
Server Info:
SessionID:<%=session.getId()%><br>
SessionIP:<%=request.getServerName()%><br>
<br>SessionPort:<%=request.getServerPort()%>
<%out.println("222"); //标记后端节点%>
测试结果:两种tomcat不断交换。但是ssessionId不变。