线上故障之Redis缓存数据异常

最新推荐文章于 2024-07-11 00:09:47 发布

小徐很努力

最新推荐文章于 2024-07-11 00:09:47 发布

阅读量984

点赞数 21

分类专栏：故障分析文章标签： redis 数据库缓存

本文链接：https://blog.csdn.net/weixin_44543482/article/details/135328240

版权

故障分析专栏收录该内容

7 篇文章 2 订阅

订阅专栏

文章描述了一次Redis在线用户缓存数据出现异常的情况，由于多线程共享非线程安全的RedisAPI导致数据错乱。文章提出了优化解决方案，包括使用不同Redis实例、批量操作代替多线程、业务层面的错误处理以及增加监控告警。

摘要由CSDN通过智能技术生成

故障现象

Redis online_user_{id}缓存数据异常问题

redis存在异常数据

{"queueList":["10000993","10000994","10000995"],"agent":{"queueList":["10000993","10000994","10000995"],"webrtcSipKey":"wsIFFl0d7C","departmentConfig":{"departmentIdList":[],"leaderId":""},"roles":["6114cd553f09bc0011563d17","6125b15cc019c600111524ba","6125b27878aefa001126f3c1"],"mobile":"","agentName":"孙李锁","updateUser":"62a9482305695f65e4b3a8d1","toAnswerOffline":"0","updateTime":"2022-06-15 10:47:15","agentNumber":"2020","accountId":"2000687","crmId":"","lastLoginTime":"","sipNumber":"","lockTime":"2022-04-04 12:01:59","createTime":"2021-07-20 14:47:10","agentVersion":"L00016","createUser":"60f66f31f912040011401940","_id":"60f6716ebc28d30011b90372","webrtcSipNumber":"2000687w2020","userIcon":"","email":"","status":"enable"},"pushServerPort":3000,"webrtcSipKey":"wsIFFl0d7C","departmentConfig":{"departmentIdList":[],"leaderId":""},"roles":["6114cd553f09bc0011563d17","6125b15cc019c600111524ba","6125b27878aefa001126f3c1"],"toAnswerOffline":"0","connTime":"2022-06-15 13:06:39","socketId":"fbSTWB_akZVJX7XxABh3","pushServerIp":"10.42.14.181","sipNumber":"","lockTime":"2022-04-04 12:01:59","email":"","mobile":"","agentName":"孙李锁","updateUser":"60f67199bc28d30011b903ec","updateTime":"2022-06-15 13:57:03","agentNumber":"2020","accountId":"2000687","crmId":"","lastLoginTime":"","createTime":"2021-07-20 14:47:10","agentVersion":"L00016","createUser":"60f66f31f912040011401940","_id":"60f6716ebc28d30011b90372","webrtc*3 $3 SET $41 cache_onlineuser_6257c85d6f21d7001150784"}

可以看到在正常的json数据后面多了一些没用的数据，看着是redis的另一个key的信息

问题结论分析

原因：cache_onlineuser_6257c85d6f21d7001150784 这个缓存数据存在异常，导致业务解析失败

经排查代码里使用了自己写的RedisAPI工具类，且多个线程公用了同一个redisAPI实例导致的

        CompletableFuture.allOf(
                keySet.stream().map(
                        v -> {
                            JSONObject userCacheInfo = JSONObject.parseObject(onlineUsers.get(v));
                            JSONObject userJson = WebUtils.toJsonObject(agents.get(Integer.parseInt(v)));
                            userJson.remove("__v");
                            userJson.remove("hashPassword");
                            userCacheInfo.putAll(userJson);
                            String key = RedisConstant.REDIS_PREFIX_CACHE_ONLINEUSER + agents.get(Integer.parseInt(v)).get_id();
                            return CompletableFuture.runAsync(() -> redis.set(key, userCacheInfo.toJSONString()))

                        }).toArray(CompletableFuture[]::new)

        ).join();

经确认，redis的这个工具类是非线程安全的（jedis本身也是线程不安全的），表现为在set, get方法在多线程调用时会出现数据错乱。jedis发送命令和获取返回值时使用全局变量RedisOutputStream和RedisInputStream，所以会导致数据偶发的异常

问题优化

1、redis操作优化

优化此处逻辑，存online_user_{id} redis key的时候，避免多线程使用同一个Redis实例。避免掉异常数据。
类似代码逻辑调整。所有使用Redis包的地方，排查此类问题并修复

方案1：改为使用不同的redis实例

方案2：使用批量操作mset, mget等优化替代多线程操作redis

优化后的代码：

      Set<String> keySet = onlineUsers.keySet();
        keySet.forEach(v -> {
            JSONObject userCacheInfo = JSONObject.parseObject(onlineUsers.get(v));
            JSONObject userJson = WebUtils.toJsonObject(agents.get(Integer.parseInt(v)));
            userJson.remove("__v");
            userJson.remove("hashPassword");
            userCacheInfo.putAll(userJson);
            String key = RedisConstant.REDIS_PREFIX_CACHE_ONLINEUSER + agents.get(Integer.parseInt(v)).get_id();
            redis.set(key, userCacheInfo.toJSONString());
        });