背景本文2019年中原创首发于博客园,当时使用CSRedisCore的排障思路引起很大反响,当时被张队公众号翻牌,本次转回公号。
上次Redis MQ分布式改造之后,编排的容器稳定运行一个多月,昨天突然收到ETL端同事通知,没有采集到解析日志。
赶紧进服务器 docker ps查看容器:
用于数据接收的ReceiverApp容器挂掉了;
尝试docker container start [containerid],几分钟后该容器再次崩溃。
Redis连接超限docker log [containerid] 查看容器日志: 显示连接Redis服务的客户端数量超限。
CSRedis.RedisException: ERR max number of clients reached.
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2] Executed action EqidManager.Controllers.EqidController.BatchPutEqidAndProfileIds (EqidReceiver) in 7.1767msfail: Microsoft.AspNetCore.Server.Kestrel[13] Connection id "0HLPR3AP8ODKH", Request id "0HLPR3AP8ODKH:00000001": An unhandled exception was thrown by the application.CSRedis.RedisException: ERR max number of clients reached at CSRedis.CSRedisClient.GetAndExecute[T](RedisClientPool pool, Func`2 handler, Int32 jump, Int32 errtimes) at CSRedis.CSRedisClient.ExecuteScalar[T](String key, Func`3 hander) at CSRedis.CSRedisClient.LPush[T](String key, T[] value) at RedisHelper.LPush[T](String key, T[] value) at EqidManager.Controllers.EqidController.BatchPutEqidAndProfileIds(List`1 eqidPairs) in /home/gitlab-runner/builds/haD2h5xC/0/webdissector/datasource/eqid-manager/src/EqidReceiver/Controllers/EqidController.cs:line 31 at lambda_method(Closure , Object ) at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult() at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.AwaitableResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at System.Threading.Tasks.ValueTask`1.get_Result() at Microsoft.AspNetCore.Mvc.Internal.ControllerActionIn