记一次线上 redis 使用分布式锁坑

最新推荐文章于 2021-02-16 00:06:55 发布

weixin_33845881

最新推荐文章于 2021-02-16 00:06:55 发布

阅读量154

点赞数

文章标签：数据库 java python

原文链接：https://my.oschina.net/superwen/blog/1571214

版权

2019独角兽企业重金招聘Python工程师标准>>>

起因

早上grafana 发来邮件警告系统异常过多..

输入图片说明

排查

发现错误一:

Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.UnknownHostException: redis.marathon.l4lb.thisdcos.directory
	at redis.clients.jedis.Connection.connect(Connection.java:207)
	at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93)
	at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767)
	at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106)
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
	at redis.clients.util.Pool.getResource(Pool.java:49)
	... 119 common frames omitted
Caused by: java.net.UnknownHostException: redis.marathon.l4lb.thisdcos.directory
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at redis.clients.jedis.Connection.connect(Connection.java:184)
	... 126 common frames omitted

dcos vip 映射 ip 出错,导致服务中找不到 redis

再查看 grafana redis 监控,发现 redis io,setnx 命令都异常.如下图

输入图片说明

排查相关服务. 发现下面代码

while (!flag && start <= System.currentTimeMillis() + time) {

            flag = redisTemplate.execute(new RedisCallback<Boolean>() {
                @Override
                public Boolean doInRedis(RedisConnection connection) throws DataAccessException {
                    Jedis jedis = (Jedis) connection.getNativeConnection();
                    if (jedis.setnx(key, UID) == 1L) {
                        jedis.expire(key, LOCK_DEATH_TIME);//300秒过期,防止死锁.如果在这步前jvm挂了,会导致一直死锁.
                        LOCK_MAP.put(RedisLock.this, 1);
                        setExclusiveOwnerThread(Thread.currentThread());
                        return true;
                    }
                    
                    return false;
                }
            });

在 setnx 方法的时候没有休眠,导致一直循环..

解决方案,while 加入 sleep

 while (!flag && start <= System.currentTimeMillis() + time) {

            flag = redisTemplate.execute(new RedisCallback<Boolean>() {
                @Override
                public Boolean doInRedis(RedisConnection connection) throws DataAccessException {
                    Jedis jedis = (Jedis) connection.getNativeConnection();
                    if (jedis.setnx(key, UID) == 1L) {
                        jedis.expire(key, LOCK_DEATH_TIME);//300秒过期,防止死锁.如果在这步前jvm挂了,会导致一直死锁.
                        LOCK_MAP.put(RedisLock.this, 1);
                        setExclusiveOwnerThread(Thread.currentThread());
                        return true;
                    }
                    //循环竞争锁的时候添加休眠
                    try {
                        TimeUnit.MILLISECONDS.sleep(20L);
                    } catch (InterruptedException e) {
                        log.error("lock error", e);
                    }
                    log.info("竞争锁: " + key);
                    return false;
                }
            });

转载于:https://my.oschina.net/superwen/blog/1571214