线上出现以下异常:
redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:51)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at com.zhe800.recommender.base.utils.JedisTemplate.getResource(JedisTemplate.java:84)
at com.zhe800.recommender.base.utils.RedisUtil.getStringValueByField(RedisUtil.java:524)
at com.zhe800.bigdata.recommender.util.RedisResultAble.call(GetRecomDataUtil.java:326)
at com.zhe800.bigdata.recommender.util.RedisResultAble.call(GetRecomDataUtil.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.NoSuchElementException: Unable to validate object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:497)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
at redis.clients.util.Pool.getResource(Pool.java:49)
... 9 more
跟踪JedisPool源码发现,招聘异常是因为JedisFactory 中的验证方法返回false,方法如下:
public boolean validateObject(PooledObject<Jedis> pooledJedis) {
final BinaryJedis jedis = pooledJedis.getObject();
try {
HostAndPort hostAndPort = this.hostAndPort.get();
String connectionHost = jedis.getClient().getHost();
int connectionPort = jedis.getClient().getPort();
return hostAndPort.getHost().equals(connectionHost)
&& hostAndPort.getPort() == connectionPort && jedis.isConnected()
&& jedis.ping().equals("PONG");
} catch (final Exception e) {
return false;
}
}
根据代码初步判断认为是Redis集群中的某些节点Ping不通导致,通知运维查看Redis集群监控,发现是两个Redis节点IP冲突,导致Ping不通。