【Jedis详解】Sentinel连接创建与关闭

本篇主要讲解Jedis源码中连接Sentinel获取连接池及其关闭的部分，Redis Sentinel的相关内容大家可以参考《Redis运维与开发》这本书，个人觉得是非常好的Redis入门书籍。

 

1. Sentinel连接池初始化

通过Jedis连接Sentinel我们需要先构建JedisSentinelPool，通过它再获取到Jedis实例，然后就可以进行Redis操作，代码大体如下：

JedisSentinelPool pool = new JedisSentinelPool("mymaster", sentinels, new GenericObjectPoolConfig(), 1000);
Jedis jedis = pool.getResource();

这里我们先看下JedisSentinelPool的构建逻辑，下面是其最终调用的构造函数：

public JedisSentinelPool(String masterName, Set<String> sentinels,
    final GenericObjectPoolConfig poolConfig, final int connectionTimeout, 
    final int soTimeout, final String password, final int database, 
    final String clientName, final int sentinelConnectionTimeout, 
    final int sentinelSoTimeout, final String sentinelPassword, 
    final String sentinelClientName) {

   this.poolConfig = poolConfig;
   this.connectionTimeout = connectionTimeout;
   this.soTimeout = soTimeout;
   this.password = password;
   this.database = database;
   this.clientName = clientName;
   this.sentinelConnectionTimeout = sentinelConnectionTimeout;
   this.sentinelSoTimeout = sentinelSoTimeout;
   this.sentinelPassword = sentinelPassword;
   this.sentinelClientName = sentinelClientName;

   HostAndPort master = initSentinels(sentinels, masterName);
   initPool(master);
}

 

其中重点在于initSentinels和initPool这两个方法，下面逐一介绍。

(1) initSentinels(Set<String> sentinels, final String masterName)

private HostAndPort initSentinels(Set<String> sentinels, final String masterName) {

  HostAndPort master = null;
  boolean sentinelAvailable = false;

  log.info("Trying to find master from available Sentinels...");

  for (String sentinel : sentinels) {
    final HostAndPort hap = HostAndPort.parseString(sentinel);

    log.debug("Connecting to Sentinel {}", hap);

    Jedis jedis = null;
    try {
      jedis = new Jedis(hap.getHost(), hap.getPort(), sentinelConnectionTimeout, sentinelSoTimeout);
      if (sentinelPassword != null) {
        jedis.auth(sentinelPassword);
      }
      if (sentinelClientName != null) {
        jedis.clientSetname(sentinelClientName);
      }

      //通过"sentinel get-master-addr-by-name <masterName>"指令获取主节点ip和端口
      List<String> masterAddr = jedis.sentinelGetMasterAddrByName(masterName);

      // connected to sentinel...
      sentinelAvailable = true;

      if (masterAddr == null || masterAddr.size() != 2) {
        log.warn("Can not get master addr, master name: {}. Sentinel: {}", masterName, hap);
        continue;
      }

      master = toHostAndPort(masterAddr);
      log.debug("Found Redis master at {}", master);
      break;
    } catch (JedisException e) {
      // resolves #1036, it should handle JedisException there's another chance
      // of raising JedisDataException
      log.warn(
        "Cannot get master address from sentinel running @ {}. Reason: {}. Trying next one.", hap,
        e.toString());
    } finally {
      if (jedis != null) {
        jedis.close();
      }
    }
  }

  if (master == null) {
    if (sentinelAvailable) {
      // can connect to sentinel, but master name seems to not
      // monitored
      throw new JedisException("Can connect to sentinel, but " + masterName
          + " seems to be not monitored...");
    } else {
      throw new JedisConnectionException("All sentinels down, cannot determine where is "
          + masterName + " master is running...");
    }
  }

  log.info("Redis master running at " + master + ", starting Sentinel listeners...");

  /**
   * 为每个sentinel添加监听器：主要监听主节点IP端口信息变更，信息一旦变更则重新初始化连接池
   * 1. 主动获取：通过"sentinel get-master-addr-by-name <masterName>"指令获取；
   * 2. 消息通知：通过订阅"+switch-master"频道获取。
   */
  for (String sentinel : sentinels) {
    final HostAndPort hap = HostAndPort.parseString(sentinel);
    MasterListener masterListener = new MasterListener(masterName, hap.getHost(), hap.getPort());
    // whether MasterListener threads are alive or not, process can be stopped
    masterListener.setDaemon(true);
    masterListeners.add(masterListener);
    masterListener.start();
  }

  return master;
}

 

该方法主要有两个作用：

1. 通过"sentinel get-master-addr-by-name <masterName>"指令获取主节点ip和端口。

2. 为每个sentinel添加监听器，主要监听主节点IP端口信息变更。

 

我们重点分析第二个作用，每个监听线程所执行的内容：

protected volatile Jedis j;
protected AtomicBoolean running = new AtomicBoolean(false);

@Override
public void run() {

  running.set(true);

  while (running.get()) {

    j = new Jedis(host, port);

    try {
      // double check that it is not being shutdown
      if (!running.get()) {
        break;
      }

      // code for active refresh
      List<String> masterAddr = j.sentinelGetMasterAddrByName(masterName);
      if (masterAddr == null || masterAddr.size() != 2) {
        log.warn("Can not get master addr, master name: {}. Sentinel: {}:{}.", masterName, host, port);
      } else {
        initPool(toHostAndPort(masterAddr));
      }

      //订阅主节点变更频道（+switch-master）
      j.subscribe(new JedisPubSub() {
        @Override
        public void onMessage(String channel, String message) {
          log.debug("Sentinel {}:{} published: {}.", host, port, message);

          String[] switchMasterMsg = message.split(" ");

          //+switch-master事件消息格式:
          //<masterName> <old master ip> <old master port> <new master ip> <new master port>
          if (switchMasterMsg.length > 3) {

            if (masterName.equals(switchMasterMsg[0])) {
              initPool(toHostAndPort(Arrays.asList(switchMasterMsg[3], switchMasterMsg[4])));
            } else {
              log.debug(
                "Ignoring message on +switch-master for master name {}, our master name is {}",
                switchMasterMsg[0], masterName);
            }

          } else {
            log.error(
              "Invalid message received on Sentinel {}:{} on channel +switch-master: {}", host,
              port, message);
          }
        }
      }, "+switch-master");

    } catch (JedisException e) {

      if (running.get()) {
        log.error("Lost connection to Sentinel at {}:{}. Sleeping 5000ms and retrying.", host,
          port, e);
        try {
          Thread.sleep(subscribeRetryWaitTimeMillis);
        } catch (InterruptedException e1) {
          log.error("Sleep interrupted: ", e1);
        }
      } else {
        log.debug("Unsubscribing from Sentinel at {}:{}", host, port);
      }
    } finally {
      j.close();
    }
  }
}

 

在该任务循环中，其主要通过两种方式来发现主节点信息是否变更：

1. 主动获取：通过"sentinel get-master-addr-by-name <masterName>"指令获取；

2. 消息通知：通过订阅"+switch-master"频道获取。

 

当主节点发生变更时，会发布"+switch-master"事件，该事件的消息格式为：

<masterName> <old master ip> <old master port> <new master ip> <new master port>

 

Jeids通过switchMasterMsg信息重新初始化连接池，initPool方法我们下面说明。

（2）initPool(HostAndPort master)

private final Object initPoolLock = new Object();

private void initPool(HostAndPort master) {
  //同步锁保证每次只有一个线程进行初始化  
  synchronized(initPoolLock){
    if (!master.equals(currentHostMaster)) {
      currentHostMaster = master;
      if (factory == null) {
        factory = new JedisFactory(master.getHost(), master.getPort(), connectionTimeout,
            soTimeout, password, database, clientName);
        initPool(poolConfig, factory);
      } else {
        factory.setHostAndPort(currentHostMaster);
        // although we clear the pool, we still have to check the
        // returned object
        // in getResource, this call only clears idle instances, not
        // borrowed instances
        internalPool.clear();
      }

      log.info("Created JedisPool to master at " + master);
    }
  }
}

public void initPool(final GenericObjectPoolConfig poolConfig, PooledObjectFactory<T> factory) {

  if (this.internalPool != null) {
    try {
      closeInternalPool();
    } catch (Exception e) {
    }
  }

  this.internalPool = new GenericObjectPool<T>(factory, poolConfig);
}

Jedis的连接池都是通过GenericObjectPool实现，本文暂不涉及，后续会专门介绍。

 

2. Sentinel获取连接

Jedis通过Sentinel连接池获取连接的代码如下：

Jedis jedis = pool.getResource();

JedisSentinelPool类通过重写父类Pool的getResource方法获取连接，代码如下：

@Override
public Jedis getResource() {
  while (true) {
    Jedis jedis = super.getResource();
    jedis.setDataSource(this);

    // get a reference because it can change concurrently
    final HostAndPort master = currentHostMaster;
    final HostAndPort connection = new HostAndPort(jedis.getClient().getHost(), jedis.getClient()
        .getPort());

    if (master.equals(connection)) {
      // connected to the correct master
      return jedis;
    } else {
      returnBrokenResource(jedis);
    }
  }
}

从上面的代码可以看出，获取到的是主节点的连接，也就是说，读写都在主节点上进行操作，从节点仅仅具有备份及故障转移的功能，并不参与数据操作。

 

3. Sentinel连接池关闭

@Override
public void destroy() {
  for (MasterListener m : masterListeners) {
    m.shutdown();
  }

  super.destroy();
}

该方法主要完成两件事：

（1）关闭所有sentinel的监听器，关闭代码如下：

public void shutdown() {
  try {
    log.debug("Shutting down listener on {}:{}", host, port);
    running.set(false);
    // This isn't good, the Jedis object is not thread safe
    if (j != null) {
      j.disconnect();
    }
  } catch (Exception e) {
    log.error("Caught exception while shutting down: ", e);
  }
}

（2）关闭连接池。

Jedis连接池涉及到GenericObjectPool，这个我们后面再详细分析。