做了一个测试,redis队列,在2000个线程同时并发处理四万个简单消息时,推入队列会报错;
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
从队列里面拿消息时,完全没问题,不报错;(可能是消息内容比较简单)
如果消息检入或者检出出错,做setnx锁,会严重限制速度;应该有容错错失,比如把异常的消息重新放入缓冲中,再进行一次推入或出列操作
机器的问题,吞吐量
入队时吞吐量是40000/66338 s 大概也有每秒六七千;
出队时吞吐量是40000/65047 s 多次尝试,大概比入队慢那么几秒
如果线程数>2000个,可能吞吐量更大;我想应该超过10000/s;
代码
JedisPoolConfig config = new JedisPoolConfig();
config.setMaxActive(5000);
config.setMaxWait(1000*10);
config.setMaxIdle(300);
config.setTestOnBorrow(true);
public static void main(String[] args) throws PiRedisExecption, InterruptedException {
int count = 2000;
final CountDownLatch latch = new CountDownLatch(count);
final CountDownLatch latch2 = new CountDownLatch(count);
Random rand = new Random(5);
final List <Integer>list = new ArrayList<Integer>();
list.add(0);
Long start = 0L;
System.out.println(RedisManager.getInstance().len("a"));
// System.out.println(RedisManager.getInstance().del("a"));
for(int i=0;i<count;i++){
final int j = i;
if(i==count-1){
System.out.println("==============================");
start = System.currentTimeMillis();
}
new Thread(){
public void run(){
try {
latch.await();
} catch (InterruptedException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
for(int a=0;a<200;a++){
try {
// RedisManager.getInstance().lPush("a", j+"_"+a);
System.out.println(RedisManager.getInstance().rPop("a"));
///list.set(0, list.get(0)+1);
} catch (PiRedisExecption e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
System.out.println("-------------------");
latch2.countDown();
}
}.start();
latch.countDown();
}
latch2.await();
System.out.println("========"+(System.currentTimeMillis()-start));
System.out.println(RedisManager.getInstance().len("a"));
// for(int a = 0;a<count;a++){
// if(list.get(0)==count){}
// }
}
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
从队列里面拿消息时,完全没问题,不报错;(可能是消息内容比较简单)
如果消息检入或者检出出错,做setnx锁,会严重限制速度;应该有容错错失,比如把异常的消息重新放入缓冲中,再进行一次推入或出列操作
机器的问题,吞吐量
入队时吞吐量是40000/66338 s 大概也有每秒六七千;
出队时吞吐量是40000/65047 s 多次尝试,大概比入队慢那么几秒
如果线程数>2000个,可能吞吐量更大;我想应该超过10000/s;
代码
JedisPoolConfig config = new JedisPoolConfig();
config.setMaxActive(5000);
config.setMaxWait(1000*10);
config.setMaxIdle(300);
config.setTestOnBorrow(true);
public static void main(String[] args) throws PiRedisExecption, InterruptedException {
int count = 2000;
final CountDownLatch latch = new CountDownLatch(count);
final CountDownLatch latch2 = new CountDownLatch(count);
Random rand = new Random(5);
final List <Integer>list = new ArrayList<Integer>();
list.add(0);
Long start = 0L;
System.out.println(RedisManager.getInstance().len("a"));
// System.out.println(RedisManager.getInstance().del("a"));
for(int i=0;i<count;i++){
final int j = i;
if(i==count-1){
System.out.println("==============================");
start = System.currentTimeMillis();
}
new Thread(){
public void run(){
try {
latch.await();
} catch (InterruptedException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
for(int a=0;a<200;a++){
try {
// RedisManager.getInstance().lPush("a", j+"_"+a);
System.out.println(RedisManager.getInstance().rPop("a"));
///list.set(0, list.get(0)+1);
} catch (PiRedisExecption e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
System.out.println("-------------------");
latch2.countDown();
}
}.start();
latch.countDown();
}
latch2.await();
System.out.println("========"+(System.currentTimeMillis()-start));
System.out.println(RedisManager.getInstance().len("a"));
// for(int a = 0;a<count;a++){
// if(list.get(0)==count){}
// }
}