基于Redis Sentinel主从切换以及Sharding分片的Redis集群高可用方案

最新推荐文章于 2024-06-10 10:00:57 发布

java20150326

最新推荐文章于 2024-06-10 10:00:57 发布

阅读量2.6k

点赞数 2

分类专栏： redis与tomcat集群

redis与tomcat集群专栏收录该内容

16 篇文章 0 订阅

订阅专栏

本文主要介绍一种通过Jedis&Sentinel实现Redis集群高可用方案，该方案需要使用Jedis2.2.2及以上版本（强制），Redis2.8及以上版本(可选，Sentinel最早出现在Redis2.4中，Redis2.8中Sentinel更加稳定)，Redis集群是以分片(Sharding)加主从的方式搭建，满足可扩展性的要求；

Redis Sentinel介绍

Redis Sentinel是Redis官方提供的集群管理工具，主要有三大功能：
监控，能持续监控Redis的主从实例是否正常工作；
通知，当被监控的Redis实例出问题时，能通过API通知系统管理员或其他程序；
自动故障恢复，如果主实例无法正常工作，Sentinel将启动故障恢复机制把一个从实例提升为主实例，其他的从实例将会被重新配置到新的主实例，且应用程序会得到一个更换新地址的通知。
Redis Sentinel是一个分布式系统，可以部署多个Sentinel实例来监控同一组Redis实例，它们通过Gossip协议来确定一个主实例宕机，通过Agreement协议来执行故障恢复和配置变更，一般在生产环境中部署多个实例来提高系统可用性，只要有一个Sentinel实例运行正常，就能保证被监控的Redis实例运行正常（类似Zookeeper，通过多个Zookeeper来提高系统可用性）；
本文不涉及Sentinel的实现细节和工作原理，读者可以阅读其他文章了解；

Redis HA方案

HA的关键在于避免单点故障及故障恢复，在Redis Cluster未发布之前，Redis一般以主/从方式部署（这里讨论的应用从实例主要用于备份，主实例提供读写，有不少应用是读写分离的，读写操作需要取不同的Redis实例，该方案也可用于此种应用，原理都是相通的，区别在于数据操作层如何封装），该方式要实现HA主要有如下几种方案：
1，keepalived：通过keepalived的虚拟IP，提供主从的统一访问，在主出现问题时，通过keepalived运行脚本将从提升为主，待主恢复后先同步后自动变为主，该方案的好处是主从切换后，应用程序不需要知道(因为访问的虚拟IP不变)，坏处是引入keepalived增加部署复杂性；
2，zookeeper：通过zookeeper来监控主从实例，维护最新有效的IP，应用通过zookeeper取得IP，对Redis进行访问；
3，sentinel：通过Sentinel监控主从实例，自动进行故障恢复，该方案有个缺陷：因为主从实例地址(IP&PORT)是不同的，当故障发生进行主从切换后，应用程序无法知道新地址，故在Jedis2.2.2中新增了对Sentinel的支持，应用通过redis.clients.jedis.JedisSentinelPool.getResource()取得的Jedis实例会及时更新到新的主实例地址。
笔者所在的公司先使用了方案1一段时间后，发现keepalived在有些情况下会导致数据丢失，keepalived通过shell脚本进行主从切换，配置复杂，而且keepalived成为新的单点，后来选用了方案3，使用Redis官方解决方案；（方案2需要编写大量的监控代码，没有方案3简便，网上有人使用方案2读者可自行查看）

选用Sentinel出现的问题

Sentinel&Jedis看上去是个完美的解决方案，这句话只说对了一半，在无分片的情况是这样，但我们的应用使用了数据分片-sharing，数据被平均分布到4个不同的实例上，每个实例以主从结构部署，Jedis没有提供基于Sentinel的ShardedJedisPool，也就是说在4个分片中，如果其中一个分片发生主从切换，应用所使用的ShardedJedisPool无法获得通知，所有对那个分片的操作将会失败。
本文提供一个基于Sentinel的ShardedJedisPool，能及时感知所有分片主从切换行为，进行连接池重建，源码见ShardedJedisSentinelPool.java

ShardedJedisSentinelPool实现分析

构造函数

类似之前的Jedis Pool的构造方法，需要参数poolConfig提供诸如maxIdle，maxTotal之类的配置，masters是一个List，用来保存所有分片Master在Sentinel中配置的名字（注意master的顺序不能改变，因为Shard算法是依据分片位置进行计算，如果顺序错误将导致数据存储混乱），sentinels是一个Set，其中存放所有Sentinel的地址(格式：IP:PORT，如127.0.0.1:26379)，顺序无关；

初始化连接池

在构造函数中，通过方法

取得当前所有分片的master地址(IP&PORT)，对每个分片，通过顺次连接Sentinel实例，获取该分片的master地址，如果无法获得，即所有Sentinel都无法连接，将休眠1秒后继续重试，直到取得所有分片的master地址，代码块如下：

通过

初始化连接池，到此连接池中的所有连接都指向分片的master；

监控每个Sentinel

在方法

最后，会为每个Sentinel启动一个Thread来监控Sentinel做出的更改：

该线程的run方法通过Jedis Pub/Sub API（实现JedisPubSub接口，并通过jedis.subscribe进行订阅）向Sentinel实例订阅“+switch-master”频道，当Sentinel进行主从切换时，该线程会得到新Master地址的通知，通过master name判断哪个分片进行了切换，将新master地址替换原来位置的地址，并调用initPool(List masters)进行Jedis连接池重建；后续所有通过该连接池取得的连接都指向新Master地址，对应用程序透明；

应用示例

maven:

		<dependency>
			<groupId>redis.clients</groupId>
			<artifactId>jedis</artifactId>
			<version>2.9.0</version>
		</dependency>
		<dependency>
			<groupId>commons-pool</groupId>
			<artifactId>commons-pool</artifactId>
			<version>1.6</version>
		</dependency>
		<dependency>
			<groupId>org.apache.commons</groupId>
			<artifactId>commons-pool2</artifactId>
			<version>2.0</version>
			<type>jar</type>
			<scope>compile</scope>
		</dependency>

代码使用如下：

        GenericObjectPoolConfig config = new GenericObjectPoolConfig();
        config.setMaxTotal(100);
        config.setMaxIdle(10);

        List<String> masters = new ArrayList<String>();
        masters.add("master-1");//sentinel.conf文件中的名字
        masters.add("master-2");

        Set<String> sentinels = new HashSet<String>();
        sentinels.add("192.168.248.127:63792");
        sentinels.add("192.168.248.128:63792");

        ShardedJedisSentinelPool pool = new ShardedJedisSentinelPool(masters, sentinels, config, 60000);
        ShardedJedis j = null;
        for (int i = 11; i < 15; i++) {
            try {
                j = pool.getResource();
                j.set("KEY: " + i, "" + i);
                pool.returnResource(j);
            } catch (JedisConnectionException e) {
                e.printStackTrace();
            }
        }
        pool.destroy();

ShardedJedisSentinelPool文件内容如下：

/**
 *   2017/4/30.
 */
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.logging.Logger;
import java.util.regex.Pattern;

import org.apache.commons.pool2.PooledObject;
import org.apache.commons.pool2.PooledObjectFactory;
import org.apache.commons.pool2.impl.DefaultPooledObject;
import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import redis.clients.jedis.*;
import redis.clients.jedis.exceptions.JedisConnectionException;
import redis.clients.util.Hashing;
import redis.clients.util.Pool;

public class ShardedJedisSentinelPool extends Pool<ShardedJedis> {

    public static final int MAX_RETRY_SENTINEL = 10;

    protected final Logger log = Logger.getLogger(getClass().getName());

    protected GenericObjectPoolConfig poolConfig;

    protected int timeout = Protocol.DEFAULT_TIMEOUT;

    private int sentinelRetry = 0;

    protected String password;

    protected int database = Protocol.DEFAULT_DATABASE;

    protected Set<MasterListener> masterListeners = new HashSet<MasterListener>();

    private volatile List<HostAndPort> currentHostMasters;

    public ShardedJedisSentinelPool(List<String> masters, Set<String> sentinels) {
        this(masters, sentinels, new GenericObjectPoolConfig(),
                Protocol.DEFAULT_TIMEOUT, null, Protocol.DEFAULT_DATABASE);
    }

    public ShardedJedisSentinelPool(List<String> masters, Set<String> sentinels, String password) {
        this(masters, sentinels, new GenericObjectPoolConfig(),
                Protocol.DEFAULT_TIMEOUT, password);
    }

    public ShardedJedisSentinelPool(final GenericObjectPoolConfig poolConfig, List<String> masters, Set<String> sentinels) {
        this(masters, sentinels, poolConfig, Protocol.DEFAULT_TIMEOUT, null,
                Protocol.DEFAULT_DATABASE);
    }

    public ShardedJedisSentinelPool(List<String> masters, Set<String> sentinels,
                                    final GenericObjectPoolConfig poolConfig, int timeout,
                                    final String password) {
        this(masters, sentinels, poolConfig, timeout, password,
                Protocol.DEFAULT_DATABASE);
    }

    public ShardedJedisSentinelPool(List<String> masters, Set<String> sentinels,
                                    final GenericObjectPoolConfig poolConfig, final int timeout) {
        this(masters, sentinels, poolConfig, timeout, null,
                Protocol.DEFAULT_DATABASE);
    }

    public ShardedJedisSentinelPool(List<String> masters, Set<String> sentinels,
                                    final GenericObjectPoolConfig poolConfig, final String password) {
        this(masters, sentinels, poolConfig, Protocol.DEFAULT_TIMEOUT,
                password);
    }

    public ShardedJedisSentinelPool(List<String> masters, Set<String> sentinels,
                                    final GenericObjectPoolConfig poolConfig, int timeout,
                                    final String password, final int database) {
        this.poolConfig = poolConfig;
        this.timeout = timeout;
        this.password = password;
        this.database = database;

        List<HostAndPort> masterList = initSentinels(sentinels, masters);
        initPool(masterList);
    }

    public void destroy() {
        for (MasterListener m : masterListeners) {
            m.shutdown();
        }

        super.destroy();
    }

    public List<HostAndPort> getCurrentHostMaster() {
        return currentHostMasters;
    }

    private void initPool(List<HostAndPort> masters) {
        if (!equals(currentHostMasters, masters)) {
            StringBuffer sb = new StringBuffer();
            for (HostAndPort master : masters) {
                sb.append(master.toString());
                sb.append(" ");
            }
            log.info("Created ShardedJedisPool to master at [" + sb.toString() + "]");
            List<JedisShardInfo> shardMasters = makeShardInfoList(masters);
            initPool(poolConfig, new ShardedJedisFactory(shardMasters, Hashing.MURMUR_HASH, null));
            currentHostMasters = masters;
        }
    }

    private boolean equals(List<HostAndPort> currentShardMasters, List<HostAndPort> shardMasters) {
        if (currentShardMasters != null && shardMasters != null) {
            if (currentShardMasters.size() == shardMasters.size()) {
                for (int i = 0; i < currentShardMasters.size(); i++) {
                    if (!currentShardMasters.get(i).equals(shardMasters.get(i))) return false;
                }
                return true;
            }
        }
        return false;
    }

    private List<JedisShardInfo> makeShardInfoList(List<HostAndPort> masters) {
        List<JedisShardInfo> shardMasters = new ArrayList<JedisShardInfo>();
        for (HostAndPort master : masters) {
            JedisShardInfo jedisShardInfo = new JedisShardInfo(master.getHost(), master.getPort(), timeout);
            jedisShardInfo.setPassword(password);

            shardMasters.add(jedisShardInfo);
        }
        return shardMasters;
    }

    private List<HostAndPort> initSentinels(Set<String> sentinels, final List<String> masters) {

        Map<String, HostAndPort> masterMap = new HashMap<String, HostAndPort>();
        List<HostAndPort> shardMasters = new ArrayList<HostAndPort>();

        log.info("Trying to find all master from available Sentinels...");

        for (String masterName : masters) {
            HostAndPort master = null;
            boolean fetched = false;

            while (!fetched && sentinelRetry < MAX_RETRY_SENTINEL) {
                for (String sentinel : sentinels) {
                    final HostAndPort hap = toHostAndPort(Arrays.asList(sentinel.split(":")));

                    log.fine("Connecting to Sentinel " + hap);

                    try {
                        Jedis jedis = new Jedis(hap.getHost(), hap.getPort());
                        master = masterMap.get(masterName);
                        if (master == null) {
                            List<String> hostAndPort = jedis.sentinelGetMasterAddrByName(masterName);
                            if (hostAndPort != null && hostAndPort.size() > 0) {
                                master = toHostAndPort(hostAndPort);
                                log.fine("Found Redis master at " + master);
                                shardMasters.add(master);
                                masterMap.put(masterName, master);
                                fetched = true;
                                jedis.disconnect();
                                break;
                            }
                        }
                    } catch (JedisConnectionException e) {
                        log.warning("Cannot connect to sentinel running @ " + hap + ". Trying next one.");
                    }
                }

                if (null == master) {
                    try {
                        log.severe("All sentinels down, cannot determine where is "
                                + masterName + " master is running... sleeping 1000ms, Will try again.");
                        Thread.sleep(1000);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                    fetched = false;
                    sentinelRetry++;
                }
            }

            // Try MAX_RETRY_SENTINEL times.
            if (!fetched && sentinelRetry >= MAX_RETRY_SENTINEL) {
                log.severe("All sentinels down and try " + MAX_RETRY_SENTINEL + " times, Abort.");
                throw new JedisConnectionException("Cannot connect all sentinels, Abort.");
            }
        }

        // All shards master must been accessed.
        if (masters.size() != 0 && masters.size() == shardMasters.size()) {

            log.info("Starting Sentinel listeners...");
            for (String sentinel : sentinels) {
                final HostAndPort hap = toHostAndPort(Arrays.asList(sentinel.split(":")));
                MasterListener masterListener = new MasterListener(masters, hap.getHost(), hap.getPort());
                masterListeners.add(masterListener);
                masterListener.start();
            }
        }

        return shardMasters;
    }

    private HostAndPort toHostAndPort(List<String> getMasterAddrByNameResult) {
        String host = getMasterAddrByNameResult.get(0);
        int port = Integer.parseInt(getMasterAddrByNameResult.get(1));

        return new HostAndPort(host, port);
    }

    /**
     * PoolableObjectFactory custom impl.
     */
    protected static class ShardedJedisFactory implements PooledObjectFactory<ShardedJedis> {
        private List<JedisShardInfo> shards;
        private Hashing algo;
        private Pattern keyTagPattern;

        public ShardedJedisFactory(List<JedisShardInfo> shards, Hashing algo, Pattern keyTagPattern) {
            this.shards = shards;
            this.algo = algo;
            this.keyTagPattern = keyTagPattern;
        }

        public PooledObject<ShardedJedis> makeObject() throws Exception {
            ShardedJedis jedis = new ShardedJedis(shards, algo, keyTagPattern);
            return new DefaultPooledObject<ShardedJedis>(jedis);
        }

        public void destroyObject(PooledObject<ShardedJedis> pooledShardedJedis) throws Exception {
            final ShardedJedis shardedJedis = pooledShardedJedis.getObject();
            for (Jedis jedis : shardedJedis.getAllShards()) {
                try {
                    try {
                        jedis.quit();
                    } catch (Exception e) {

                    }
                    jedis.disconnect();
                } catch (Exception e) {

                }
            }
        }

        public boolean validateObject(PooledObject<ShardedJedis> pooledShardedJedis) {
            try {
                ShardedJedis jedis = pooledShardedJedis.getObject();
                for (Jedis shard : jedis.getAllShards()) {
                    if (!shard.ping().equals("PONG")) {
                        return false;
                    }
                }
                return true;
            } catch (Exception ex) {
                return false;
            }
        }

        public void activateObject(PooledObject<ShardedJedis> p) throws Exception {

        }

        public void passivateObject(PooledObject<ShardedJedis> p) throws Exception {

        }
    }

    protected class JedisPubSubAdapter extends JedisPubSub {
        @Override
        public void onMessage(String channel, String message) {
        }

        @Override
        public void onPMessage(String pattern, String channel, String message) {
        }

        @Override
        public void onPSubscribe(String pattern, int subscribedChannels) {
        }

        @Override
        public void onPUnsubscribe(String pattern, int subscribedChannels) {
        }

        @Override
        public void onSubscribe(String channel, int subscribedChannels) {
        }

        @Override
        public void onUnsubscribe(String channel, int subscribedChannels) {
        }
    }

    protected class MasterListener extends Thread {

        protected List<String> masters;
        protected String host;
        protected int port;
        protected long subscribeRetryWaitTimeMillis = 5000;
        protected Jedis jedis;
        protected AtomicBoolean running = new AtomicBoolean(false);

        protected MasterListener() {
        }

        public MasterListener(List<String> masters, String host, int port) {
            this.masters = masters;
            this.host = host;
            this.port = port;
        }

        public MasterListener(List<String> masters, String host, int port,
                              long subscribeRetryWaitTimeMillis) {
            this(masters, host, port);
            this.subscribeRetryWaitTimeMillis = subscribeRetryWaitTimeMillis;
        }

        public void run() {

            running.set(true);

            while (running.get()) {

                jedis = new Jedis(host, port);

                try {
                    jedis.subscribe(new JedisPubSubAdapter() {
                        @Override
                        public void onMessage(String channel, String message) {
                            log.fine("Sentinel " + host + ":" + port + " published: " + message + ".");

                            String[] switchMasterMsg = message.split(" ");

                            if (switchMasterMsg.length > 3) {

                                int index = masters.indexOf(switchMasterMsg[0]);
                                if (index >= 0) {
                                    HostAndPort newHostMaster = toHostAndPort(Arrays.asList(switchMasterMsg[3], switchMasterMsg[4]));
                                    List<HostAndPort> newHostMasters = new ArrayList<HostAndPort>();
                                    for (int i = 0; i < masters.size(); i++) {
                                        newHostMasters.add(null);
                                    }
                                    Collections.copy(newHostMasters, currentHostMasters);
                                    newHostMasters.set(index, newHostMaster);

                                    initPool(newHostMasters);
                                } else {
                                    StringBuffer sb = new StringBuffer();
                                    for (String masterName : masters) {
                                        sb.append(masterName);
                                        sb.append(",");
                                    }
                                    log.fine("Ignoring message on +switch-master for master name "
                                            + switchMasterMsg[0]
                                            + ", our monitor master name are ["
                                            + sb + "]");
                                }

                            } else {
                                log.severe("Invalid message received on Sentinel "
                                        + host
                                        + ":"
                                        + port
                                        + " on channel +switch-master: "
                                        + message);
                            }
                        }
                    }, "+switch-master");

                } catch (JedisConnectionException e) {

                    if (running.get()) {
                        log.severe("Lost connection to Sentinel at " + host
                                + ":" + port
                                + ". Sleeping 5000ms and retrying.");
                        try {
                            Thread.sleep(subscribeRetryWaitTimeMillis);
                        } catch (InterruptedException e1) {
                            e1.printStackTrace();
                        }
                    } else {
                        log.fine("Unsubscribing from Sentinel at " + host + ":"
                                + port);
                    }
                }
            }
        }

        public void shutdown() {
            try {
                log.fine("Shutting down listener on " + host + ":" + port);
                running.set(false);
                // This isn't good, the Jedis object is not thread safe
                jedis.disconnect();
            } catch (Exception e) {
                log.severe("Caught exception while shutting down: " + e.getMessage());
            }
        }
    }
}

总结

本文通过现实中遇到的问题，即在Redis数据分片的情况下，在使用Sentinel做HA时，如何做到主从的切换对应用程序透明，通过Jedis的Pub/Sub功能，能同时监控多个分片的主从切换情况，并通过监听到的新地址重新构造连接池，后续从连接池中取得的所有连接都指向新地址。该方案的关键是：使用sentinel做HA，Jedis版本必须2.2.2及以上，所有访问Redis实例的连接都必须从连接池中获取；

该项目的GitHub主页: https://github.com/warmbreeze/sharded-jedis-sentinel-pool

==================================以下有代验证==============================================

连接泄露Bug

循环时发现ShardedJedisSentinelPool连接池有时竟然满了！毕竟默认是8个连接，但为什么会这样？查了下代码，果然是这个原因！这个新扩展的pool没有为getResource()出来的ShardedJedis设置DataSource，所以关闭ShardedJedis时没法正常还回到连接池中。

补丁很简单，就是参照Jedis的ShardedJedisPool，覆写一些资源管理的方法。

@Override
    public ShardedJedis getResource() {
        ShardedJedis jedis = super.getResource();
        jedis.setDataSource(this);
        return jedis;
    }

    @Override
    public void returnBrokenResource(final ShardedJedis resource) {
        if (resource != null) {
            returnBrokenResourceObject(resource);
        }
    }

    @Override
    public void returnResource(final ShardedJedis resource) {
        if (resource != null) {
            resource.resetState();
            returnResourceObject(resource);
        }
    }

java20150326

关注

2
点赞
踩
4

收藏

觉得还不错? 一键收藏
0
评论
基于Redis Sentinel主从切换以及Sharding分片的Redis集群高可用方案

本文主要介绍一种通过Jedis&Sentinel实现Redis集群高可用方案，该方案需要使用Jedis2.2.2及以上版本（强制），Redis2.8及以上版本(可选，Sentinel最早出现在Redis2.4中，Redis2.8中Sentinel更加稳定)，Redis集群是以分片(Sharding)加主从的方式搭建，满足可扩展性的要求； Redis Sentinel介绍Redi
复制链接

扫一扫

专栏目录