redis 获取不到_从Redis连接池获取连接失败的原因说起

问题描述

其他业务线的同学在测试环境发现应用程序一直不能获取redis连接，我帮忙看了下。

首先看应用错误日志

Caused by: org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:97)

at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:143)

at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:41)

at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:85)

at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:55)

at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:169)

at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:149)

... 76 more

Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

at redis.clients.util.Pool.getResource(Pool.java:22)

at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:90)

... 83 more

Caused by: java.util.NoSuchElementException: Could not create a validated object, cause: ValidateObject failed

at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:871)

at redis.clients.util.Pool.getResource(Pool.java:20)

... 84 more

问题调查

确定环境

发现是使用spring-data-redis通过jedis连接的redis服务端。

这个系统的代码很久没动，已经忘记了。先看看使用的jar版本吧。

查看应用程序使用的相关jar：

lsof -p 19377 | grep -E "jedis|pool|redis"

发现输出的jar包含：commons-pool-1.3.jar、spring-data-redis-1.1.1.RELEASE.jar、jedis-2.1.0.jar

翻了下commons pool相关代码

try {

_factory.activateObject(latch.getPair().value);

if(_testOnBorrow &&

!_factory.validateObject(latch.getPair().value)) {

throw new Exception("ValidateObject failed");

}

synchronized(this) {

_numInternalProcessing--;

_numActive++;

}

return latch.getPair().value;

}

catch (Throwable e) {

PoolUtils.checkRethrow(e);

// object cannot be activated or is invalid

try {

_factory.destroyObject(latch.getPair().value);

} catch (Throwable e2) {

PoolUtils.checkRethrow(e2);

// cannot destroy broken object

}

synchronized (this) {

_numInternalProcessing--;

if (!newlyCreated) {

latch.reset();

_allocationQueue.add(0, latch);

}

allocate();

}

if(newlyCreated) {

throw new NoSuchElementException("Could not create a validated object, cause: " + e.getMessage());

}

else {

continue; // keep looping

}

}

可见客户端应该是配置了testOnBorrow，在校验连接时失败了。

java操作redis有多种客户端，项目使用spring-data-redis操作redis，在spring-data-redis中也有不同的客户端实现如jedis，lettuce等。根据错误日志推断使用的redis客户端实现为jedis。

查看JedisConnectionFactory源码

在JedisPool中定义了校验对象的代码。

public boolean validateObject(final Object obj) {

if (obj instanceof Jedis) {

final Jedis jedis = (Jedis) obj;

try {

return jedis.isConnected() && jedis.ping().equals("PONG");

} catch (final Exception e) {

return false;

}

} else {

return false;

}

}

通过wireshark查看TCP包并确定问题原因

熟悉redis的同学都知道，redis客户端发送“PING”后服务端会返回一个“PONG“作为回应，一般会作为连接的检验方法。

既然校验报错，那抓包看看请求和响应吧！

<