Java尝试连接Sentinel时，抛出Could not get a resource from the pool异常

问题描述

前几天兴致冲冲的学习了哨兵模式，并且能够在阿里云上自己实现了redis-sentinel的一些功能，今天尝试去用java客户端去连接sentinel时发生了一系列意外，启动时便抛出了Could not get a resource from the pool这个异常。无奈去网上搜了一下，无非是bind端口没打开或是防火墙配置问题等，都跟自己的情况不一样。

废话少说，先上代码：

RedisUtil

package com.lamarsan.sentinel.util;

import redis.clients.jedis.*;
import redis.clients.jedis.exceptions.JedisConnectionException;

import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.logging.Logger;

/**
 * className: RedisUtil
 * description: TODO
 *
 * @author hasee
 * @version 1.0
 * @date 2018/12/26 10:57
 */
public class RedisUtil {
    private static final Logger myLogger = Logger.getLogger("com.lamarsan.sentinel.util");

    private static JedisSentinelPool pool = null;

    static {
        try {
            Set<String> sentinels = new HashSet<String>();
            sentinels.add("你的ip:26380");
            sentinels.add("你的ip:26379");
            sentinels.add("你的ip:26381");
            String masterName = "mymaster";
            String password = "你的密码";
            JedisPoolConfig config = new JedisPoolConfig();
            config.setMinIdle(8);
            config.setMaxTotal(100);
            config.setMaxIdle(100);
            config.setMaxWaitMillis(10000);
            pool = new JedisSentinelPool(masterName, sentinels, config, password);
            try {
                pool.getResource();
            } catch (JedisConnectionException e) {
                myLogger.info(e.getMessage());
                e.printStackTrace();
            }
        } catch (Exception e) {
            myLogger.info(e.getMessage());
            e.printStackTrace();
        }
    }

    private static void returnResource(JedisSentinelPool pool, Jedis jedis) {
        if (jedis != null) {
            jedis.close();
        }
    }

    /**
     * <p>通过key获取储存在redis中的value</p>
     * <p>并释放连接</p>
     *
     * @param key
     * @return 成功返回value 失败返回null
     */
    public static String get(String key) {
        Jedis jedis = null;
        String value = null;
        try {
            jedis = pool.getResource();
            value = jedis.get(key);
        } catch (Exception e) {
            if (jedis != null) {
                jedis.close();
            }
            e.printStackTrace();
        } finally {
            returnResource(pool, jedis);
        }
        return value;
    }

    /**
     * <p>向redis存入key和value,并释放连接资源</p>
     * <p>如果key已经存在 则覆盖</p>
     *
     * @param key
     * @param value
     * @return 成功 返回OK 失败返回 0
     */
    public static String set(String key, String value) {
        Jedis jedis = null;
        try {
            jedis = pool.getResource();
            return jedis.set(key, value);
        } catch (Exception e) {
            if (jedis != null) {
                jedis.close();
            }
            e.printStackTrace();
            return "0";
        } finally {
            returnResource(pool, jedis);
        }
    }
    
    .....
}

main

package com.lamarsan.sentinel;

import com.lamarsan.sentinel.util.RedisUtil;

/**
 * className: Test
 * description: TODO
 *
 * @author hasee
 * @version 1.0
 * @date 2019/9/13 15:54
 */
public class Test {
    public static void main(String[] args) throws InterruptedException {
        RedisUtil.setnx("hello", "world");
        while (true) {
            Thread.sleep(10000);
            String result = RedisUtil.get("hello");
            System.out.println(result);
        }
    }
}

解决思路

从代码角度来看，并没有什么配置是有问题的，而且服务器上的配置也没问题，但就是连接时在pool.getResource();抛出连接超时的异常，正当我抓耳挠腮中，索性断个点看看吧！pool的属性：currentHostMaster的ip居然是127.0.0.1！！！！

似乎有了一定的头绪，让我们跟进JedisSentinelPool的源码看看：

public JedisSentinelPool(String masterName, Set<String> sentinels,
      final GenericObjectPoolConfig poolConfig, final String password) {
    this(masterName, sentinels, poolConfig, Protocol.DEFAULT_TIMEOUT, password);
}

连续点击几次this，这都是JedisSentinelPool的构造方法，会进入如下的构造方法：

public JedisSentinelPool(String masterName, Set<String> sentinels,
      final GenericObjectPoolConfig poolConfig, final int connectionTimeout, final int soTimeout,
      final String password, final int database, final String clientName) {
    this.poolConfig = poolConfig;
    this.connectionTimeout = connectionTimeout;
    this.soTimeout = soTimeout;
    this.password = password;
    this.database = database;
    this.clientName = clientName;

    HostAndPort master = initSentinels(sentinels, masterName);
    initPool(master);
  }

通过断点可以发现master的host属性为127.0.0.1。我们进入initSentinels方法一看究竟：

private HostAndPort initSentinels(Set<String> sentinels, final String masterName) {

    HostAndPort master = null;
    boolean sentinelAvailable = false;

    log.info("Trying to find master from available Sentinels...");

    for (String sentinel : sentinels) {
      final HostAndPort hap = HostAndPort.parseString(sentinel);

      log.fine("Connecting to Sentinel " + hap);

      Jedis jedis = null;
      try {
        jedis = new Jedis(hap.getHost(), hap.getPort());

        List<String> masterAddr = jedis.sentinelGetMasterAddrByName(masterName);

        // connected to sentinel...
        sentinelAvailable = true;

        if (masterAddr == null || masterAddr.size() != 2) {
          log.warning("Can not get master addr, master name: " + masterName + ". Sentinel: " + hap
              + ".");
          continue;
        }

        master = toHostAndPort(masterAddr);
        log.fine("Found Redis master at " + master);
        break;
      } catch (JedisException e) {
        // resolves #1036, it should handle JedisException there's another chance
        // of raising JedisDataException
        log.warning("Cannot get master address from sentinel running @ " + hap + ". Reason: " + e
            + ". Trying next one.");
      } finally {
        if (jedis != null) {
          jedis.close();
        }
      }
    }

观察可以发现，master是与masterAddr变量相关，而且通过断点可以得masterAddr的ip为127.0.0.1。

masterAddr是通过sentinelGetMasterAddrByName方法赋值的，让我们进入这个方法：

public List<String> sentinelGetMasterAddrByName(String masterName) {
    client.sentinel(Protocol.SENTINEL_GET_MASTER_ADDR_BY_NAME, masterName);
    final List<Object> reply = client.getObjectMultiBulkReply();
    return BuilderFactory.STRING_LIST.build(reply);
}

进入sentinel方法：

public void sentinel(final String... args) {
    final byte[][] arg = new byte[args.length][];
    for (int i = 0; i < arg.length; i++) {
      arg[i] = SafeEncoder.encode(args[i]);
    }
    sentinel(arg);
}

继续进入sentinel方法：

public void sentinel(final byte[]... args) {
    sendCommand(SENTINEL, args);
}

可以发现，它是发送了一些命令，可是发送命令到哪里呢？让我们继续进入sendCommand方法：

protected Connection sendCommand(final Command cmd, final byte[]... args) {
    try {
      connect();
      Protocol.sendCommand(outputStream, cmd, args);
      pipelinedCommands++;
      return this;
    } catch (JedisConnectionException ex) {
      /*
       * When client send request which formed by invalid protocol, Redis send back error message
       * before close connection. We try to read it to provide reason of failure.
       */
      try {
        String errorMessage = Protocol.readErrorLineIfPossible(inputStream);
        if (errorMessage != null && errorMessage.length() > 0) {
          ex = new JedisConnectionException(errorMessage, ex.getCause());
        }
      } catch (Exception e) {
        /*
         * Catch any IOException or JedisConnectionException occurred from InputStream#read and just
         * ignore. This approach is safe because reading error message is optional and connection
         * will eventually be closed.
         */
      }
      // Any other exceptions related to connection?
      broken = true;
      throw ex;
    }
}

出现了，connect();~~~~，真相就在眼前，让我们进入它！！

public void connect() {
    if (!isConnected()) {
      try {
        socket = new Socket();
        // ->@wjw_add
        socket.setReuseAddress(true);
        socket.setKeepAlive(true); // Will monitor the TCP connection is
        // valid
        socket.setTcpNoDelay(true); // Socket buffer Whetherclosed, to
        // ensure timely delivery of data
        socket.setSoLinger(true, 0); // Control calls close () method,
        // the underlying socket is closed
        // immediately
        // <-@wjw_add

        socket.connect(new InetSocketAddress(host, port), connectionTimeout);
        socket.setSoTimeout(soTimeout);

        if (ssl) {
          if (null == sslSocketFactory) {
            sslSocketFactory = (SSLSocketFactory)SSLSocketFactory.getDefault();
          }
          socket = (SSLSocket) sslSocketFactory.createSocket(socket, host, port, true);
          if (null != sslParameters) {
            ((SSLSocket) socket).setSSLParameters(sslParameters);
          }
          if ((null != hostnameVerifier) &&
              (!hostnameVerifier.verify(host, ((SSLSocket) socket).getSession()))) {
            String message = String.format(
                "The connection to '%s' failed ssl/tls hostname verification.", host);
            throw new JedisConnectionException(message);
          }
        }

        outputStream = new RedisOutputStream(socket.getOutputStream());
        inputStream = new RedisInputStream(socket.getInputStream());
      } catch (IOException ex) {
        broken = true;
        throw new JedisConnectionException(ex);
      }
    }
}

真相大白！！！原来Jedis的底层是通过Socket连接的~接下来就很容易猜到，其通过Socket获取到了服务器的sentinel配置，将其的ip和port填入了masterAddr中，那么我们只需要检查服务器的sentinel配置文件即可。然后我发现了这一句：

sentinel monitor mymaster 127.0.0.1 6380 2

啊，原来是这里！！！在将127.0.0.1的ip改成阿里云的公网ip并重启服务redis-sentinel服务后，java就能正常对redis进行操作了~~~~

总结

当遇到问题时，如果直接搜索不能找到答案，那么尝试阅读源码不失为一种好选择。