redisson Unexpected exception while processing command Only 1 of 2 slaves were synced

目录

背景:

现象:

问题定位:

问题原因：

解决:

---

背景:

生产环境一个活动给某个用户发送积分失败，核心业务接口使用Redisson分布式锁
同事答复：redis主从切换导致的问题。

个人表示怀疑，所以想定位下真实原因。

redisson 3.17.3
sentinel模式:master slave1 slave2

     <dependency>
                <groupId>org.redisson</groupId>
                <artifactId>redisson-spring-boot-starter</artifactId>
                <version>3.17.3</version>
     </dependency>

    RLock  rLock = redisson.getLock("xxxxxx");
    rLock.lock(15, TimeUnit.SECONDS);

现象:

rLock.lock(15, TimeUnit.SECONDS);

没走到下边的业务代码就报错

16:52:58.382 2023-11-14 16:52:57 [http-nio-8080-exec-142] ERROR c.c.p.configure.aspect.ApiAspect 60 - Failed to invoke interface [Map cn.xx.xx.api.customer.xx.yyy(String)]
16:52:58.382 org.redisson.client.RedisException: Unexpected exception while processing command
16:52:58.382  at org.redisson.command.CommandAsyncService.convertException(CommandAsyncService.java:276) ~[redisson-3.17.3.jar!/:3.17.3]
16:52:58.382  at org.redisson.command.CommandAsyncService.get(CommandAsyncService.java:115) ~[redisson-3.17.3.jar!/:3.17.3]
16:52:58.382  at org.redisson.RedissonObject.get(RedissonObject.java:82) ~[redisson-3.17.3.jar!/:3.17.3]
16:52:58.382  at org.redisson.RedissonLock.tryAcquire(RedissonLock.java:144) ~[redisson-3.17.3.jar!/:3.17.3]


16:52:58.486 Caused by: java.lang.IllegalStateException: Only 1 of 2 slaves were synced
16:52:58.486  at org.redisson.RedissonBaseLock.lambda$evalWriteAsync$0(RedissonBaseLock.java:226) ~[redisson-3.17.3.jar!/:3.17.3]
16:52:58.486  at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) ~[na:1.8.0_201]
16:52:58.486  at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) ~[na:1.8.0_201]
16:52:58.487  at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_201]
16:52:58.487  at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[na:1.8.0_201]
16:52:58.487  at org.redisson.command.CommandBatchService.lambda$executeAsync$7(CommandBatchService.java:322) ~[redisson-3.17.3.jar!/:3.17.3]
16:52:58.487  at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[na:1.8.0_201]

关键错误日志

c.c.p.configure.aspect.ApiAspect 60 - Failed to invoke interface [Map cn.xx.xx.api.customer.xx.yyy(String)]

org.redisson.client.RedisException: Unexpected exception while processing command

16:52:58.486 Caused by: java.lang.IllegalStateException: Only 1 of 2 slaves were synced

问题定位:

找运维同学，看监控，没看到异常，反馈其他应用没问题。从自己代码定位问题。

带着问题看源码
org.redisson.command.CommandBatchService#executeAsync

org.redisson.RedissonBaseLock#createCommandBatchService

问题原因：

分布式锁，默认值是等1秒全部slave 同步成功master再继续，如果slave没全部成功就报错(本例中一台slave成功一台没在1s内成功)。

经过和运维确认，那个时间点redis在进行数据持久化，持久化策略RDB，运维侧看，不能避免1秒内全部完成所有节点。

解决:

升级客户端版本，3.21以上，据说3.21.* 以上，只要一个slave成功即可(懒得贴代码了)。
 

。

if (getServiceManager().getCfg().isCheckLockSyncedSlaves()&& res.getSyncedSlaves() == 0 && availableSlaves > 0) {
    throw new CompletionException(
            new IllegalStateException("None of slaves were synced"));
}