redis 连接报错 GET_LIKE_ERROR 处理过程

1，redis报警

运营同事说app登录不了，运维工程师说卡在redis这里，然后去后台看了app工程日志，有如下报警信息：

 

2016-06-06 16:26:03,838 [http-6600-2:ERROR] com.plocc.ugc.redis.service.impl.RedisServiceImpl - GET_LIKE_ERROR org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool                 at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:97)                 at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:143)                 at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:41)                 at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:128)                 at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:91)                 at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:78)                 at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:177)                 at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152)                 at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:84)                 at org.springframework.data.redis.core.DefaultHashOperations.get(DefaultHashOperations.java:44)                 at com.plocc.ugc.dao.impl.UgcRedisDaoImpl.getIsLike(UgcRedisDaoImpl.java:77)                 at com.plocc.ugc.redis.service.impl.RedisServiceImpl.getIsFave(RedisServiceImpl.java:126)                 at com.plocc.ugc.service.impl.RecommendationH5ServiceImpl.queryRecommendationList(RecommendationH5ServiceImpl.java:119)                 at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                 at java.lang.reflect.Method.invoke(Method.java:606)                 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)                 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)                 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)                 at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77)                 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)                 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)                 at com.sun.proxy.$Proxy188.queryRecommendationList(Unknown Source)                 at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)

               

               

 

2，在应用服务器上连接下当前连接的redis

在应用服务器上telnet端口，发现close了：

[tomcat@azure_ipower_app_3_29 logs]$ telnet192.168.3.40 6379

Trying 192.168.3.40...

Connected to 192.168.3.40.

Escape character is '^]'.

-ERR max number of clients reached

Connection closed by foreign host.

[tomcat@azure_ipower_app_3_29 logs]$

 

 

3，去redis服务器查看redis服务

但是在redis服务器上，redis是起着的：

[root@azure_ipower_cache1_3_40 ~]# ps-eaf|grep redis

root    43660     1  7 16:54 ?        00:02:06/usr/local/redis/bin/redis-server 127.0.0.1:6379                

root    43874 43409  0 17:24 pts/2    00:00:00 grep redis

[root@azure_ipower_cache1_3_40 ~]#

 

 

4，问题分析解决

重启了应用tomcat服务器以及重启了redis服务器还是一样的报错信息，后来想到了，最近做了redis服务器迁移，可能是旧的redis服务器的配置引起的，然后尝试停止了旧的redis的服务器，去掉修改了/etc/hosts里面的kafka、zookpeer指定：

 

注释掉kafka、zookpeer的映射：

vim /etc/hosts #10.250.1.11     k1 #10.250.1.21     k2 #10.250.1.22     k3 # （这里使用redis是经过了zookpeer、kafka的转发的，所以对/etc/hosts里面配置的ip和别名的映射）。

 

 

然后在应用服务器上，telnetredis 端口，ok，可以用了：

[tomcat@azure_ipower_app_3_29ugc_service]$  telnet 192.168.3.40 6379

Trying 192.168.3.40...

Connected to 192.168.3.40.

Escape character is '^]'.

 

然后再启动应用服务器就ok了，应用能正常对redis进行操作了，公司业务系统也可以正常运转了。

 

 

网上搜索看到别人经历过的出错原因有：

（1）      linux防火墙iptables没有关闭

（2）      redis假死重启redis

（3）      连接的redis是从库不是主库导致