redis连接超时

问题描述：查看cmc,cmdb，cos等日志发现有redis timeout的报错信息，同时登陆面板会提示soc，cmc服务异常等，在cmp相关组 件服务重启后会显示正常30分钟到60分钟，随后会接着报错。重装redis和cmp环境均不行；

2020‐08‐14 12:33:00.001 ERROR [SockJS‐7] o.s.s.s.TaskUtils$LoggingErrorHandler:96 ‐ Unexpected  error occurred in scheduled task. org.springframework.data.redis.RedisSystemException: Redis exception; nested exception is  io.lettuce.core.RedisException: io.netty.channel.unix.Errors$NativeIoException: syscall:read(..)  failed: Connection reset by peer   at  org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExcepti onConverter.java:74)   at  org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExcepti onConverter.java:41)   at  org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExcept ionTranslationStrategy.java:44)   at  org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTra nslationStrategy.java:42)   at  org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException( LettuceConnection.java:268)   at  org.springframework.data.redis.connection.lettuce.LettuceSetCommands.convertLettuceAccessException (LettuceSetCommands.java:520)   at  org.springframework.data.redis.connection.lettuce.LettuceSetCommands.sMembers(LettuceSetCommands.j ava:245)   at  org.springframework.data.redis.connection.DefaultedRedisConnection.sMembers(DefaultedRedisConnecti on.java:607)   at  org.springframework.data.redis.core.DefaultSetOperations.lambda$members$6(DefaultSetOperations.jav a:158)   at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224)   at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:184)   at  org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:95)   at  org.springframework.data.redis.core.DefaultSetOperations.members(DefaultSetOperations.java:158)   at  org.springframework.data.redis.core.DefaultBoundSetOperations.members(DefaultBoundSetOperations.ja va:152)   at  org.springframework.session.data.redis.RedisSessionExpirationPolicy.cleanExpiredSessions(RedisSess ionExpirationPolicy.java:132)   at  org.springframework.session.data.redis.RedisOperationsSessionRepository.cleanupExpiredSessions(Red isOperationsSessionRepository.java:430)   at  org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration.la mbda$configureTasks$0(RedisHttpSessionConfiguration.java:248)   at 
org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandling Runnable.java:54)   at  org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)   at java.util.concurrent.FutureTask.run(FutureTask.java:266)   at  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoo lExecutor.java:180)   at  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecut or.java:293)   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)   at java.lang.Thread.run(Thread.java:745) Caused by: io.lettuce.core.RedisException: io.netty.channel.unix.Errors$NativeIoException:  syscall:read(..) failed: Connection reset by peer   at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:129)   at  io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvoc ationHandler.java:123)   at  io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)   at com.sun.proxy.$Proxy129.smembers(Unknown Source)   at  org.springframework.data.redis.connection.lettuce.LettuceSetCommands.sMembers(LettuceSetCommands.j ava:243)   ... 19 common frames omitted Caused by: io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection  reset by peer   at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source) 2020‐08‐14 12:33:00.053 INFO  [lettuce‐eventExecutorLoop‐1‐9] i.l.c.p.ConnectionWatchdog:173 ‐  Reconnecting, last destination was 172.27.130.12/172.27.130.12:5379 2020‐08‐14 12:33:00.054 INFO  [lettuce‐epollEventLoop‐4‐12] i.l.c.p.ReconnectionHandler:177 ‐  Reconnected to 172.27.130.12:5379 2020‐08‐14 13:29:59.860 ERROR [http‐nio‐18091‐exec‐1] o.a.c.c.C.[.[.[.[dispatcherServlet]:175 ‐  Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request  processing failed; nested exception is org.springframework.data.redis.RedisSystemException: Redis  exception; nested exception is io.lettuce.core.RedisException:  io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer]  with root cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer   at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)

解：更改/etc/redis_5379.conf的tcp‐keepalive和protected‐mode的参数，然后重启redis服务，更改完成后需要观察24 小时，没有新的报错即为正常，刚更改完的一个小时会出现报错，属于正常；

ansible ‐i cmp cmp  ‐m shell ‐a "sed ‐i 's/tcp‐keepalive 300/tcp‐keepalive 60/g'   /etc/redis_5379.conf"
ansible ‐i cmp cmp  ‐m shell ‐a "sed ‐i 's/protected‐mode yes/protected‐mode no/g'  /etc/redis_5379.conf"

redis‐server /etc/redis_5379.conf 重启命令