redis 测试之–读取性能
目的
- 熟悉jedis操作redis
- 对redis性能指标有个大概认知
测试环境
- ubuntu
- 机器双核4G内存普通机
- 外网流量4M
- redis版本: 3.2.6
- redis 和测试服务程序在一台服务器上
redis 配置
添加密码和注释了bind:127.0.0.1,其他均为默认配置
测试思路
- redis 存储一个测试key( 测试key是672字节)
- 开启jedis线程池，为个工作线程提供一个线程
- 工作线程不断读取key,获取到key后直接计数器记录
- 开启一个定时打印定时器，将结果输出
测试结果
日志记录
=======================================================
---------开始时间--------------结束时间-------------获取条数-----每秒吞吐量-----分钟吞吐量-----小时吞吐量-----测试运行线程数量----每个消息的大小
-2016-12-10 15:28:42---2016-12-10 17:01:44------240670794------43115---------2587857---------240670792-----------2-----------672byte-----
---------开始时间--------------结束时间-------------获取条数-----每秒吞吐量-----分钟吞吐量-----小时吞吐量-----测试运行线程数量----每个消息的大小
-2016-12-10 10:40:25---2016-12-10 12:55:12------457891036------56620---------3417097---------228945517-----------4-----------672byte-----
---------开始时间--------------结束时间-------------获取条数-----每秒吞吐量-----分钟吞吐量-----小时吞吐量-----测试运行线程数量----每个消息的大小
-2016-12-10 14:10:26---2016-12-10 15:11:28------231494306------63215---------3794988---------231494306-----------6-----------672byte-----
[framework] 2016-12-10 15:11:28,466 - com.jiazq.jizq.redis.mq.RedisReadTaskSchuder -3662056 [pool-1-thread-1] ERROR com.jiazq.jizq.redis.mq.RedisReadTaskSch
--------开始时间--------------结束时间-------------获取条数-----每秒吞吐量-----分钟吞吐量-----小时吞吐量-----测试运行线程数量----每个消息的大小
-2016-12-10 13:00:49---2016-12-10 14:09:06------275406638------67221---------4050097---------275406638-----------8-----------672byte-----
cpu记录
// 2 个线程
top - 17:02:56 up 16 days, 18:35, 2 users, load average: 1.15, 1.00, 1.02
Tasks: 116 total, 4 running, 112 sleeping, 0 stopped, 0 zombie
%Cpu(s): 18.2 us, 24.7 sy, 0.0 ni, 39.3 id, 0.0 wa, 0.0 hi, 14.6 si, 3.3 st
KiB Mem: 4046856 total, 3113916 used, 932940 free, 233184 buffers
KiB Swap: 0 total, 0 used, 0 free. 1501364 cached Mem
// 4 线程
top - 12:57:07 up 16 days, 14:29, 2 users, load average: 1.64, 1.85, 1.92
Tasks: 117 total, 3 running, 114 sleeping, 0 stopped, 0 zombie
%Cpu(s): 19.9 us, 30.6 sy, 0.0 ni, 21.7 id, 0.0 wa, 0.0 hi, 26.4 si, 1.4 st
KiB Mem: 4046856 total, 3125252 used, 921604 free, 233180 buffers
KiB Swap: 0 total, 0 used, 0 free. 1499612 cached Mem
// 6 线程
top - 14:30:01 up 16 days, 16:02, 2 users, load average: 2.21, 2.73, 2.95
Tasks: 117 total, 3 running, 114 sleeping, 0 stopped, 0 zombie
%Cpu(s): 19.8 us, 39.0 sy, 0.0 ni, 14.4 id, 0.0 wa, 0.0 hi, 26.0 si, 0.8 st
KiB Mem: 4046856 total, 3115260 used, 931596 free, 233184 buffers
KiB Swap: 0 total, 0 used, 0 free. 1500320 cached Mem
// 8 线程
top - 15:21:23 up 16 days, 16:53, 2 users, load average: 4.17, 3.42, 3.14
Tasks: 117 total, 3 running, 114 sleeping, 0 stopped, 0 zombie
%Cpu(s): 21.0 us, 39.1 sy, 0.0 ni, 13.2 id, 0.0 wa, 0.0 hi, 26.1 si, 0.6 st
KiB Mem: 4046856 total, 3099824 used, 947032 free, 233184 buffers
KiB Swap: 0 total, 0 used, 0 free. 1500688 cached Mem
结果分析
- 读取线程数量2 的吞吐量为 4.3/S 平均每个线程读取能力2.1W/S 负载1.0
- 读取线程数量4（cpu内核2倍）的吞吐量为 5.6W/S 平均每个线程读取能力 1.4W/S 负载 1.85
- 读取线程数量6 (cpu内核3倍) 的吞吐量为 6.3W/S 平均每个线程读取能力 1W/S 负载 2.73
- 读取线程数量8 (cpu内核4倍) 的吞吐量为 6.7W/S 平均每个线程读取能力 8300/S 负载3.42
随着处理线程增加整体吞吐量也在增加,但增帐率下降，单线程处理能力下降，CPU使用率浮动较小，负载情况成倍增加。
结论
- 单机redis最高读取性能，还可以更高，吞吐量还应在 7W/S以上
- 客户端线程处理能力，应该在IO线程数量上，合理的客户端连接数量可提高整体处理能力.
- 客户端读取线程数量 为内核个数为佳.
核心处理代码
工作线程组(RedisReadTaskSchuder)
package com.jiazq.jizq.redis.mq;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import org.apache.log4j.Logger;
import redis.clients.jedis.Jedis;
public class RedisReadTaskSchuder {
private static Logger logger = Logger.getLogger(RedisReadTaskSchuder.class);
// 消息计数器
AtomicLong counter = new AtomicLong(0);
// 定时器
ScheduledExecutorService executorService = Executors.newScheduledThreadPool(1);
volatile boolean runState = true;
// 开启时间
long startTime = 0;
// 工作线程
Thread[] workers = null;
// 线程数量
int threadNumber = 0;
public RedisReadTaskSchuder(int threadNumber) {
// 默认线程数量为硬件内核数的2倍
this.threadNumber = threadNumber;
workers = new Thread[threadNumber];
for (int i = 0; i < threadNumber; i++) {
workers[i] = new Thread(new RedisReadTask(JedisManager.instance().getJedis()));
workers[i].setDaemon(true);
workers[i].setName(ConfigManager.read_thread_name + "i");
}
executorService.scheduleAtFixedRate(new PrintTimer(), 2, 15, TimeUnit.SECONDS);
}
/**
* 启动工作线程
*/
public void start() {
for (int i = 0; i < threadNumber; i++) {
workers[i].start();
}
startTime = System.currentTimeMillis();
}
/**
* 计数器重置
*
*/
public void resetCount() {
this.counter.set(0);
startTime = System.currentTimeMillis();
}
public long getCount() {
return this.counter.get();
}
/**
* 关闭线程
*/
public void shutdown() {
runState = false;
executorService.shutdown();
}
public void printReuslt() {
logger.info("---获取到数据数量:--" + counter.get());
}
class RedisReadTask implements Runnable {
private Jedis jedis = null;
RedisReadTask(Jedis jedis) {
this.jedis = jedis;
}
@Override
public void run() {
while (runState) {
try {
byte[] str = jedis.get(Main.testKey.getBytes());
if (null != str) {
counter.incrementAndGet();
}
} catch (Throwable t) {
logger.error("",t);
// 连接失败
if (!jedis.isConnected()) {
//返回连接池里面
jedis.close();
// 重新获取连接
jedis = JedisManager.instance().getJedis();
}
}
}
// 返回去jedis pool 里面
jedis.close();
}
}
class PrintTimer implements Runnable {
@Override
public void run() {
try {
StringBuilder sb = new StringBuilder();
SimpleDateFormat format = new SimpleDateFormat("YYYY-MM-dd HH:mm:ss");
long _count = counter.get();
long _endTime = System.currentTimeMillis();
long throughput_s = ( _count * 1000) / (_endTime - startTime);
long minTime = (_endTime - startTime) /(1000 * 60);
long hourTime = (_endTime - startTime) /(1000 * 60 * 60);
long throughput_m = (minTime != 0) ? _count /minTime : 0;
long throughput_h = (hourTime != 0) ? _count /hourTime : 0;
sb.append("\n======================================================\n");
sb.append("---------开始时间--------------结束时间-------------获取条数-----每秒吞吐量-----分钟吞吐量-----小时吞吐量-----测试运行线程数量----每个消息的大小\n");
sb.append("-");
sb.append(format.format(new Date(startTime)));
sb.append("---");
sb.append(format.format(new Date()));
sb.append("------");
sb.append(counter.get());
sb.append("------");
sb.append(throughput_s);
sb.append("---------");
sb.append(throughput_m);
sb.append("---------");
sb.append(throughput_h);
sb.append("-----------");
sb.append(threadNumber);
sb.append("-----------");
sb.append("672byte");
sb.append("-----");
logger.error(sb.toString());
logger.error("\n");
} catch (Throwable t) {
logger.error("",t);
}
}
}
}
jedis线程池
package com.jiazq.jizq.redis.mq;
import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.apache.log4j.Logger;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
public class JedisManager {
private static Logger logger = Logger.getLogger(JedisManager.class);
private static JedisManager instance = null;
private JedisPool jedisPool = null;
private JedisManager () {
}
public static synchronized JedisManager instance() {
if (null == instance) {
instance = new JedisManager();
instance.init();
}
return instance;
}
public void init() {
logger.info("------init redis start------");
GenericObjectPoolConfig jedisPoolConfig = new GenericObjectPoolConfig();
jedisPoolConfig.setMaxTotal(ConfigManager.redis_pool_maxTotal);
jedisPoolConfig.setMaxIdle(ConfigManager.redis_pool_maxIdle);
jedisPoolConfig.setTestOnBorrow(true);
jedisPoolConfig.setMaxWaitMillis(ConfigManager.redis_pool_maxWait);
jedisPool = new JedisPool(jedisPoolConfig,
ConfigManager.redis_host,
ConfigManager.redis_port,
ConfigManager.redis_pool_timeOut,
ConfigManager.redis_auth,
ConfigManager.redis_pool_database);
logger.info("------init redis end------");
}
public Jedis getJedis() {
return jedisPool.getResource();
}
public void rebackPool(Jedis jedis) {
jedis.close();
}
public String getString(String key) {
Jedis jedis = getJedis();
String result = jedis.get(key);
jedis.close();
return result;
}
public void set(String key, String value) {
Jedis jedis = getJedis();
jedis.set(key, value);
jedis.close();
}
public void shutdown() {
logger.info("---redis close--" + jedisPool.getNumActive());
jedisPool.close();
}
}
主函数Main
package com.jiazq.jizq.redis.mq;
import org.apache.log4j.Logger;
public class Main {
private static Logger logger = Logger.getLogger(Main.class);
static String testStr = "{.....}";
static String testKey = "testkey676K";
// 执行时间
static int exetime = 1;
public static void main(String[] args) throws InterruptedException {
logger.info("----start-testServer----start--");
// 初始化redis连接池
JedisManager.instance();
// 创建线程调度
RedisReadTaskSchuder schuder = new RedisReadTaskSchuder(ConfigManager.read_thread_number);
// 线程启动
schuder.start();
logger.info("----start-testServer----end--");
}
public void testNative() throws InterruptedException {
// 初始化redis连接池
JedisManager.instance();
// 创建线程调度
RedisReadTaskSchuder schuder = new RedisReadTaskSchuder(ConfigManager.read_thread_number);
// 线程启动
schuder.start();
//预热一秒
Thread.sleep(1000);
// 计数器重置
schuder.resetCount();
// 执行五秒
Thread.sleep(60000 * exetime);
schuder.shutdown();
JedisManager.instance().shutdown();
schuder.printReuslt();
logger.info(schuder.getCount() / (60 * exetime));
}
}
1339
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
