memcache 调优

旧版测试结果

 

1，搬迁以后的测试环境(使用的测试工具世http_load)
WEB SERVER： Intel(R) Xeon(R) CPU   E5645  @ 2.40GHz 16G Mem (三台： 10.3.63.203、10.3.63.205、10.3.63.206 )
Nginx : 一台: 10.3.63.203 分别将请求转发到 10.3.63.203、10.3.63.205、10.3.63.206 三台webserver
resin启动参数：
 /data/server/jdk1.6.0_16/bin/java -Djava.util.logging.manager=com.caucho.log.LogManagerImpl -Djava.system.class.loader=com.caucho.loader.SystemClassLoader -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl -Djava.awt.headless=true -Dresin.home=/data/server/resin/ -Xms4096m -Xmx4096m -Xdebug -Xss512k -XX:PermSize=256M -XX:MaxPermSize=512m -Djava.util.logging.manager=com.caucho.log.LogManagerImpl -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl -Djava.awt.headless=true -Dresin.home=/data/server/resin/ -Dresin.root=/data/server/resin/ -Dcom.sun.management.jmxremote -Djava.util.logging.manager=com.caucho.log.LogManagerImpl -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl -Djava.awt.headless=true -Dresin.home=/data/server/resin/ -Dresin.root=/data/server/resin/ -Dcom.sun.management.jmxremote com.caucho.server.resin.Resin --root-directory /data/server/resin/ -conf /data/server/resin/conf/resin.conf -socketwait 53053 start

Memcache: version 1.4.4  /data/server/memcached/bin/memcached -d -m 2048m -u root -c 1024 -p 11211 -P /data/server/memcached/11211.pid (一台机器四个实例)

xmemcache client，参数如下：
private static final int connectionPoolSize = 25;
builder.setConnectionPoolSize(connectionPoolSize);
builder.setSessionLocator(new KetamaMemcachedSessionLocator());      
builder.setBufferAllocator(new SimpleBufferAllocator());
builder.setCommandFactory(new BinaryCommandFactory());
builder.setTranscoder(new SerializingTranscoder());

测试说明： 绑定 10.3.63.203 www.chewen.com 测试， 使用 http_load 测试

./http_load -parallel 100 -fetches 1000 -rate 50 wwwhome.txt
1000 fetches, 37 max parallel, 1.94166e+08 bytes, in 20.1187 seconds
194166 mean bytes/connection
49.705 fetches/sec, 9.65102e+06 bytes/sec
msecs/connect: 0.303695 mean, 0.864 max, 0.072 min
msecs/first-response: 109.308 mean, 921.198 max, 81.796 min
HTTP response codes:
  code 200 -- 1000

./http_load -parallel 200 -fetches 20000 -rate 100 wwwhome.txt
20000 fetches, 20 max parallel, 3.88332e+09 bytes, in 200.154 seconds
194166 mean bytes/connection
99.9232 fetches/sec, 1.94017e+07 bytes/sec
msecs/connect: 0.351568 mean, 1.453 max, 0.069 min
msecs/first-response: 115.097 mean, 206.325 max, 95.411 min
2 bad byte counts
HTTP response codes:
  code 200 -- 20000

./http_load -parallel 210 -fetches 20000 -rate 150 wwwhome.txt
20000 fetches, 47 max parallel, 3.88332e+09 bytes, in 120.225 seconds
194166 mean bytes/connection
166.354 fetches/sec, 3.23004e+07 bytes/sec
msecs/connect: 0.448111 mean, 1.612 max, 0.068 min
msecs/first-response: 184.471 mean, 258.142 max, 106.027 min
HTTP response codes:
  code 200 -- 20000

wwwhome.txt内容为：http://www.chewen.com

当rate 150 超过 170 以后， 则 portal 开始抛异常， 首页关于portal部分无法显示
相应速度变慢。异常如下：
[23 17:39:03,008 ERROR] [http--8080-1316$1851416958] impl.PortalWaitInterceptor - x[1/7] waiting[timeout]: jingpin
java.util.concurrent.TimeoutException
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228)
    at java.util.concurrent.FutureTask.get(FutureTask.java:91)
    at net.paoding.rose.web.portal.impl.WindowFuture.get(WindowFuture.java:89)
    at net.paoding.rose.web.portal.impl.PortalWaitInterceptor.after(PortalWaitInterceptor.java:107)
    at net.paoding.rose.web.ControllerInterceptorAdapter.roundInvocation(ControllerInterceptorAdapter.java:107)
    at net.paoding.rose.web.InterceptorDelegate.roundInvocation(InterceptorDelegate.java:106)
    at net.paoding.rose.web.impl.thread.ActionEngine$InvocationChainImpl.doNext(ActionEngine.java:468)
    at net.paoding.rose.web.ControllerInterceptorAdapter.round(ControllerInterceptorAdapter.java:136)

 

 

 

新版测试结果

 

1，搬迁以后的测试环境
WEB SERVER：同旧版测试环境
Nginx : 一台: 同旧版测试环境
resin启动参数： 同旧版测试环境
Memcache: 同旧版测试环境
测试说明： 绑定 10.3.63.203 www.chewen.com 测试， 使用 http_load 测试

xmemcache client，参数如下：
private static final int connectionPoolSize = 10;
// 在高负载环境下，nio的单连接也会遇到瓶颈，此时你可以通过设置连接池来让更多的连接分担memcached的请求负载，
// 从而提高系统的吞吐量。连接池通常不建议设置太大，我推荐在0-30之间为好，太大则浪费系统资源，太小无法达到分担负载的目的
builder.setConnectionPoolSize(connectionPoolSize);
builder.setSessionLocator(new KetamaMemcachedSessionLocator());      
builder.setBufferAllocator(new SimpleBufferAllocator());
builder.setCommandFactory(new BinaryCommandFactory());
builder.setTranscoder(new SerializingTranscoder());
// 如果你的数据较小，如在1K以下，默认的配置选项已经足够。
// 如果你的数据较大，我会推荐你调整网络层的TCP选项，如设置socket的接收和发送缓冲区更大，启用Nagle算法
builder.setSocketOption(StandardSocketOption.SO_RCVBUF, 32* 1024);// 设置接收缓存区为32K，默认16K
builder.setSocketOption(StandardSocketOption.SO_SNDBUF, 16 *1024); // 设置发送缓冲区为16K，默认为8K
builder.setSocketOption(StandardSocketOption.TCP_NODELAY, false); // 启用nagle算法，提高吞吐量，默认关闭

//默认采用阻塞读，在局域网内能带来更好的效率，可以设置网络参数，linux系统在多节点memcached下，
//强烈建议将读线程数设置大于0，接近或者等于memcached节点数（具体参数看你的测试结果）
builder.getConfiguration().setReadThreadCount(4); // 设置读线程数    

// 默认如果连接超过5秒没有任何IO操作发生即认为空闲并发起心跳检测，你可以调长这个时间
builder.getConfiguration().setSessionIdleTimeout(10000); // 设置为10秒;

this.client = builder.build();
// Xmemcached默认会做两个优化：将连续的单个get合并成一个multi get批量操作获取，
// 将连续的请求合并成socket发送缓冲区大小的buffer发送。
//client.setMergeFactor(50);  //默认是150，缩小到50
client.setOptimizeMergeBuffer(false); //关闭合并buffer的优化
//如果你对吞吐量更在意，那么也可将合并因子调大，默认是150。但是也不可太大，太大可能导致平均响应时间延长。
//如果你对响应时间比较在意，那么可以将合并的因子减小，或者关闭合并buffer的优化

./http_load -parallel 100 -fetches 1000 -rate 50 wwwhome.txt

1000 fetches, 22 max parallel, 1.99025e+08 bytes, in 20.0433 seconds
199025 mean bytes/connection
49.8919 fetches/sec, 9.92974e+06 bytes/sec
msecs/connect: 0.233508 mean, 0.767 max, 0.07 min
msecs/first-response: 35.6236 mean, 423.272 max, 21.386 min
HTTP response codes:
  code 200 -- 1000

./http_load -parallel 200 -fetches 20000 -rate 100 wwwhome.txt
20000 fetches, 16 max parallel, 3.9805e+09 bytes, in 200.046 seconds
199025 mean bytes/connection
99.977 fetches/sec, 1.98979e+07 bytes/sec
msecs/connect: 0.273639 mean, 1.27 max, 0.065 min
msecs/first-response: 26.5203 mean, 159.425 max, 20.754 min
HTTP response codes:
  code 200 -- 20000

./http_load -parallel 210 -fetches 20000 -rate 150 wwwhome.txt
20000 fetches, 94 max parallel, 3.9805e+09 bytes, in 120.05 seconds
199025 mean bytes/connection
166.597 fetches/sec, 3.31569e+07 bytes/sec
msecs/connect: 0.340793 mean, 3.996 max, 0.065 min
msecs/first-response: 30.1232 mean, 509.998 max, 21.755 min
HTTP response codes:

./http_load -parallel 250 -fetches 20000 -rate 170 wwwhome.txt
20000 fetches, 38 max parallel, 3.9805e+09 bytes, in 100.044 seconds
199025 mean bytes/connection
199.911 fetches/sec, 3.97874e+07 bytes/sec
msecs/connect: 0.381352 mean, 1.919 max, 0.066 min
msecs/first-response: 30.7799 mean, 3183.78 max, 21.889 min
HTTP response codes:
  code 200 -- 20000

./http_load -parallel 300 -fetches 20000 -rate 200 wwwhome.txt
20000 fetches, 25 max parallel, 3.9805e+09 bytes, in 100.046 seconds
199025 mean bytes/connection
199.909 fetches/sec, 3.97868e+07 bytes/sec
msecs/connect: 0.375727 mean, 1.611 max, 0.064 min
msecs/first-response: 28.1484 mean, 238.445 max, 21.461 min
HTTP response codes:
  code 200 -- 20000

./http_load -parallel 350 -fetches 50000 -rate 250 wwwhome.txt
50000 fetches, 20 max parallel, 9.95125e+09 bytes, in 200.054 seconds
199025 mean bytes/connection
249.932 fetches/sec, 4.97427e+07 bytes/sec
msecs/connect: 0.391892 mean, 1.808 max, 0.058 min
msecs/first-response: 29.5759 mean, 97.085 max, 22.135 min
HTTP response codes:
  code 200 -- 50000

 

 

新版两台同时测试结果

1, 测试环境同上面的新版测试环境一样， 只不过 http_load 在两台机器上同时测试。分别在 （10.3.63.13  10.3.63.14）
两台机器分别绑定host 10.3.63.203 www.chewen.com

2, 测试结果如下：

./http_load -parallel 300 -fetches 50000 -rate 200 wwwhome.txt  

13测试结果:
50000 fetches, 21 max parallel, 9.95125e+09 bytes, in 250.049 seconds
199025 mean bytes/connection
199.961 fetches/sec, 3.97972e+07 bytes/sec
msecs/connect: 0.583215 mean, 2.975 max, 0.062 min
msecs/first-response: 37.2083 mean, 89.168 max, 23.198 min
HTTP response codes:
  code 200 -- 50000

14测试结果:
50000 fetches, 19 max parallel, 9.95125e+09 bytes, in 250.066 seconds
199025 mean bytes/connection
199.947 fetches/sec, 3.97945e+07 bytes/sec
msecs/connect: 0.608945 mean, 2.666 max, 0.069 min
msecs/first-response: 37.3391 mean, 91.373 max, 23.66 min
HTTP response codes:
  code 200 -- 50000  

./http_load -parallel 350 -fetches 50000 -rate 250 wwwhome.txt

13:
199025 mean bytes/connection
249.882 fetches/sec, 4.97328e+07 bytes/sec
msecs/connect: 1.04885 mean, 4.006 max, 0.064 min
msecs/first-response: 52.9335 mean, 137.015 max, 23.975 min
1 bad byte counts
HTTP response codes:
  code 200 -- 50000

14:
50000 fetches, 33 max parallel, 9.95125e+09 bytes, in 200.051 seconds
199025 mean bytes/connection
249.936 fetches/sec, 4.97435e+07 bytes/sec
msecs/connect: 1.02444 mean, 3.817 max, 0.074 min
msecs/first-response: 52.9131 mean, 126.601 max, 23.276 min
HTTP response codes:
  code 200 -- 50000

这种测试情况下， 其中一台web服务抛出一下异常
[24 14:39:59,930 ERROR] [portalExecutor-60] impl.WindowTask -
org.springframework.web.util.NestedServletException: GET /4sAction; nested exception is java.lang.reflect.InvocationTargetException: [Rose-1.0.1-20100929@Spring-2.5.6.SEC03]Error happended: GET /4sAction->String get4sAction(Invocation) params=[/4sAction]
    at net.paoding.rose.RoseFilter.throwServletException(RoseFilter.java:553)
    at net.paoding.rose.RoseFilter.doFilter(RoseFilter.java:351)
    at com.caucho.server.dispatch.FilterFilterChain.doFilter(FilterFilterChain.java:87)
    at com.caucho.server.webapp.DispatchFilterChain.doFilter(DispatchFilterChain.java:97)
    at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:265)
    at com.caucho.server.webapp.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:293)
    at com.caucho.server.webapp.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111)
    at net.paoding.rose.web.portal.impl.WindowTask.run(WindowTask.java:81)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.reflect.InvocationTargetException: [Rose-1.0.1-20100929@Spring-2.5.6.SEC03]Error happended: GET /4sAction->String get4sAction(Invocation) params=[/4sAction]
    at net.paoding.rose.web.impl.thread.ActionEngine.createException(ActionEngine.java:516)
    at net.paoding.rose.web.impl.thread.ActionEngine.execute(ActionEngine.java:393)
    at net.paoding.rose.web.impl.thread.LinkedEngine.execute(LinkedEngine.java:69)
    at net.paoding.rose.web.impl.thread.Rose.doNext(Rose.java:121)
    at net.paoding.rose.web.impl.thread.ControllerEngine.execute(ControllerEngine.java:83)
    at net.paoding.rose.web.impl.thread.LinkedEngine.execute(LinkedEngine.java:69)
    at net.paoding.rose.web.impl.thread.Rose.doNext(Rose.java:121)
    at net.paoding.rose.web.impl.thread.ModuleEngine.execute(ModuleEngine.java:117)
    at net.paoding.rose.web.impl.thread.LinkedEngine.execute(LinkedEngine.java:69)
    at net.paoding.rose.web.impl.thread.Rose.doNext(Rose.java:121)
    at net.paoding.rose.web.impl.thread.RootEngine.execute(RootEngine.java:126)
    at net.paoding.rose.web.impl.thread.LinkedEngine.execute(LinkedEngine.java:69)
    at net.paoding.rose.web.impl.thread.Rose.doNext(Rose.java:121)
    at net.paoding.rose.web.impl.thread.Rose.innerStart(Rose.java:256)
    at net.paoding.rose.web.impl.thread.Rose.start(Rose.java:109)
    at net.paoding.rose.RoseFilter.doFilter(RoseFilter.java:348)
    ... 12 more
Caused by: java.lang.NullPointerException
    at net.paoding.rose.web.portal.impl.WindowImpl.isCancelled(WindowImpl.java:117)
    at net.paoding.rose.web.portal.impl.WindowCancelableSupportInterceptor.round(WindowCancelableSupportInterceptor.java:50)
    at net.paoding.rose.web.ControllerInterceptorAdapter.roundInvocation(ControllerInterceptorAdapter.java:104)
    at net.paoding.rose.web.InterceptorDelegate.roundInvocation(InterceptorDelegate.java:106)
    at net.paoding.rose.web.impl.thread.ActionEngine$InvocationChainImpl.doNext(ActionEngine.java:468)
    at net.paoding.rose.web.impl.thread.ActionEngine.innerExecute(ActionEngine.java:445)
    at net.paoding.rose.web.impl.thread.ActionEngine.execute(ActionEngine.java:391)
    ... 26 more

./http_load -parallel 400 -fetches 60000 -rate 300 wwwhome.txt

抛出一下异常， 发现所有请求只落在了一台机器上， stderr.log 抛出以下异常
[24 14:46:56,555 ERROR] [http--8080-149$2055662099] impl.PortalWaitInterceptor - x[1/2] waiting[timeout]: dealerAction
java.util.concurrent.TimeoutException
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228)
    at java.util.concurrent.FutureTask.get(FutureTask.java:91)
    at net.paoding.rose.web.portal.impl.WindowFuture.get(WindowFuture.java:89)
    at net.paoding.rose.web.portal.impl.PortalWaitInterceptor.after(PortalWaitInterceptor.java:107)
    at net.paoding.rose.web.ControllerInterceptorAdapter.roundInvocation(ControllerInterceptorAdapter.java:107)
    at net.paoding.rose.web.InterceptorDelegate.roundInvocation(InterceptorDelegate.java:106)
    at net.paoding.rose.web.impl.thread.ActionEngine$InvocationChainImpl.doNext(ActionEngine.java:468)
    at net.paoding.rose.web.ControllerInterceptorAdapter.round(ControllerInterceptorAdapter.java:136)
    at net.paoding.rose.web.ControllerInterceptorAdapter.roundInvocation(ControllerInterceptorAdapter.java:104)
    at net.paoding.rose.web.InterceptorDelegate.roundInvocation(InterceptorDelegate.java:106)
    at net.paoding.rose.web.impl.thread.ActionEngine$InvocationChainImpl.doNext(ActionEngine.java:468)
    at net.paoding.rose.web.ControllerInterceptorAdapter.round(ControllerInterceptorAdapter.java:136)
    at net.paoding.rose.web.ControllerInterceptorAdapter.roundInvocation(ControllerInterceptorAdapter.java:104)
    at net.paoding.rose.web.InterceptorDelegate.roundInvocation(InterceptorDelegate.java:106)
    at net.paoding.rose.web.impl.thread.ActionEngine$InvocationChainImpl.doNext(ActionEngine.java:468)
    at net.paoding.rose.web.ControllerInterceptorAdapter.round(ControllerInterceptorAdapter.java:136)
    at net.paoding.rose.web.portal.impl.WindowCancelableSupportInterceptor.round(WindowCancelableSupportInterceptor.java:39)
    at net.paoding.rose.web.ControllerInterceptorAdapter.roundInvocation(ControllerInterceptorAdapter.java:104)
    at net.paoding.rose.web.InterceptorDelegate.roundInvocation(InterceptorDelegate.java:106)
    at net.paoding.rose.web.impl.thread.ActionEngine$InvocationChainImpl.doNext(ActionEngine.java:468)
    at net.paoding.rose.web.impl.thread.ActionEngine.innerExecute(ActionEngine.java:445)

 

 

关于客户端和服务器在同一个主机的情况说明

 

测试发现，如果客户端和memcache服务器在同一个主机上时，性能要比不在同一台机器上的好。原因如下：

memcached 有可配置的两种模式: UNIX 域套接字和 TCP/UDP，允许客户端以两种方式向 memcached 发起请求。
客户端和服务器在同一个主机上的情况下可以用 UNIX 域套接字，否则可以采用 TCP/UDP 的模式。两种模式是不兼容的。
在初始化过程中介绍了这两种模式，memcached 这么做为的是让其能更加可配置。TCP/UDP 自不用说，UNIX 域套接字
有独特的优势:

在同一台主机上进行通信时，是不同主机间通信的两倍
UNIX 域套接口可以在同一台主机上，不同进程之间传递套接字描述符
UNIX 域套接字可以向服务器提供客户的凭证（用户id或者用户组id）

 

memcache 如何优化

 

1, 服务器段的优化
加入 -c 的参数， 增加并发数
由原来两个 memcache 实例， 增加为 四个 memcache 实例

2, 客户段的优化， 主要是针对 xmemcache 客户段的优化
a) 具体参考上面 *新版测试结果* 的 xmemcache client 的优化
b) 将value的大小控制在 1KByte 以内， 根据目前的情况发现大部分会超过1KByte，但具体
的value应该尽可能的小

 

xmemcache client optimize refer :

http://www.blogjava.net/killme2008/archive/2010/07/08/325564.html

before refer :

http://blog.jobbole.com/53861/

xmemcache client performence test refer :

http://xmemcached.googlecode.com/svn/trunk/benchmark/benchmark.html