Could not get a resource from the pool

一 问题：

redis.clients.jedis.exceptions.JedisConnectionException: 

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
	at redis.clients.util.Pool.getResource(Pool.java:50)
	at redis.clients.jedis.JedisSentinelPool.getResource(JedisSentinelPool.java:209)
	at redis.clients.jedis.JedisSentinelPool.getResource(JedisSentinelPool.java:17)
	at djedis.djedispool.DjedisConnectionPool.getResource(DjedisConnectionPool.java:115)
	at djedis.dal.ConnectionHelper.get(ConnectionHelper.java:68)
	at djedis.dal.Djedis.get(Djedis.java:86)
	...
	at 
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
	at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:620)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:609)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
	at com.daojia.trade.aspect.LogAspect.around(LogAspect.java:34)
	at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:620)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:609)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
	at 
...
com.daojia.spat.mq.client.runners.NotifyMsgRunner$Runner.run(NotifyMsgRunner.java:66)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.NoSuchElementException: Pool exhausted
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:464)
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
	at redis.clients.util.Pool.getResource(Pool.java:48)
	... 44 more

问题定位：以为分布式锁redislock出问题了。获取不到资源。

看了下封装的jedis，的确是获取不到连接了。

二 排查

     报错场景是偶发的，看日志结合业务场景就是收到mq之后，有个分布式锁去处理消息。现在是资源不够了。

排除一个case：就是没有释放。不贴代码了。

观察一下端口占用情况。

也是正常的。再看整个机器的监控，cpu负载、线程数都不高。

对比下报错的日志。

结合一下业务：mq来了，先加redis锁。

看下日志：那一秒的日志有800条，加锁成功日志400多条就出问题，应该是资源不足了。

所以咨询了mq方面，缺失把积压的消息一下全部重推，（已经是脏数据，没有ack掉，正常的已经消费了，）。

所以调大配置资源：从10 -》100.删除无效积压消息。积压消息会定时重推，也符合报警时间。

dba不推荐太大，毕竟超过1000还不行就有问题。需要排查原因。

*************************

关注业务，关注报警。