Spring OpenFeign源码分析(3) 负载均衡与服务发现

在前文，我们了解到feign实现负载均衡需要两个重要的接口：

1. ILoadBalancer - 管理可用的服务器集合，实现软负载均衡
2. IRule - 实现负载均衡算法

本文重点介绍这两个接口：

1. 主要实现类
2. 注入方式：默认的实现类，可用的实现类
3. 实现类的算法

ILoadBalancer

ILoadBalancer接口

public interface ILoadBalancer {

	public void addServers(List<Server> newServers);

    // 选择一个可用的server
	public Server chooseServer(Object key);

	public void markServerDown(Server server);

	@Deprecated
	public List<Server> getServerList(boolean availableOnly);

    public List<Server> getReachableServers();

	public List<Server> getAllServers();
}

实现类

ILoadBalancer
  |-- AbstractLoadBalancer
    |-- BaseLoadBalancer
      |-- DynamicServerListLoadBalancer
        |-- ZoneAwareLoadBalancer
    |-- NoOpLoadBalancer

默认情况下使用的是ZoneAwareLoadBalancer实现类，是在RibbonClientConfiguration配置了里面装配的。

@Bean
@ConditionalOnMissingBean
public ILoadBalancer ribbonLoadBalancer(IClientConfig config,
		ServerList<Server> serverList, ServerListFilter<Server> serverListFilter,
		IRule rule, IPing ping, ServerListUpdater serverListUpdater) {
	if (this.propertiesFactory.isSet(ILoadBalancer.class, name)) {
		return this.propertiesFactory.get(ILoadBalancer.class, config, name);
	}
	return new ZoneAwareLoadBalancer<>(config, rule, ping, serverList,
			serverListFilter, serverListUpdater);
}

ZoneAwareLoadBalancer类

// 封装zone -> LoadBalancer映射关系
private ConcurrentHashMap<String, BaseLoadBalancer> balancers = 
    new ConcurrentHashMap<String, BaseLoadBalancer>();

public Server chooseServer(Object key) {
    // 当只有一个zone时，直接使用父类chooseServer方法选择server
    if (!ENABLED.get() || getLoadBalancerStats().getAvailableZones().size() <= 1) {
        return super.chooseServer(key);
    }
    Server server = null;
    try {
        // LoadBalancerStats封装zone -> server集关系
        LoadBalancerStats lbStats = getLoadBalancerStats();
        Map<String, ZoneSnapshot> zoneSnapshot = ZoneAvoidanceRule.createSnapshot(lbStats);
        // 获取可用的zone集
        Set<String> availableZones = ZoneAvoidanceRule
            .getAvailableZones(zoneSnapshot, triggeringLoad.get(), triggeringBlackoutPercentage.get());
        if (availableZones != null &&  availableZones.size() < zoneSnapshot.keySet().size()) {
            // 随机一个zone
            String zone = ZoneAvoidanceRule.randomChooseZone(zoneSnapshot, availableZones);
            if (zone != null) {
                // 从balancers获取该zone对应的LoadBalancer
                BaseLoadBalancer zoneLoadBalancer = getLoadBalancer(zone);
                // 用LoadBalancer选择一个可用server
                server = zoneLoadBalancer.chooseServer(key);
            }
        }
    } catch (Exception e) {
        logger.error("Error choosing server using zone aware logic for load balancer={}", name, e);
    }
    if (server != null) {
        return server;
    } else {
        return super.chooseServer(key);
    }
}

LoadBalancerStats类封装着zone -> server集和zone -> ZoneStats关系，如下：

public class LoadBalancerStats implements IClientConfigAware {

    String name;

    // Map<Server,ServerStats> serverStatsMap = new ConcurrentHashMap<Server,ServerStats>();
    // key都是zone
    volatile Map<String, ZoneStats> zoneStatsMap = new ConcurrentHashMap<String, ZoneStats>();
    volatile Map<String, List<? extends Server>> upServerListZoneMap =
        new ConcurrentHashMap<String, List<? extends Server>>();

    // 连接失败数量阈值
    private volatile CachedDynamicIntProperty connectionFailureThreshold;

    // 断路器跳闸超时阈值
    private volatile CachedDynamicIntProperty circuitTrippedTimeoutFactor;

    // 最大断路器跳闸超时时长
    private volatile CachedDynamicIntProperty maxCircuitTrippedTimeout;

    private static final DynamicIntProperty SERVERSTATS_EXPIRE_MINUTES = 
        DynamicPropertyFactory
        	.getInstance().getIntProperty("niws.loadbalancer.serverStats.expire.minutes", 30);

    // 服务器节点 -> 服务器State关系
    private final LoadingCache<Server, ServerStats> serverStatsCache;

    // ...
}

ServerStats维护着服务节点的状态信息，包括：请求数量、失败数量、断路器跳闸状态等，提供判断服务节点可用(断路器是否打开)的方法，在负载均衡选择可用服务节点时会使用到这些状态信息。

ZoneStats类封装当前服务端节点状态：

public class ZoneStats<T extends Server> {

    private final LoadBalancerStats loadBalancerStats;
    private final String zone;
    private static final String PREFIX = "ZoneStats_";
    private final Counter counter;

    final String monitorId;

	// ...
}

看一下super的chooseServer方法：

public Server chooseServer(Object key) {
    if (counter == null) {
        counter = createCounter();
    }
    counter.increment();
    if (rule == null) {
        return null;
    } else {
        try {
            // 使用IRule选择一个server
            // IRule里面封装负载均衡算法
            // 这个IRule是在创建LoadBalancer对象时传递进来的
            // 默认使用ZoneAvoidanceRule
            return rule.choose(key);
        } catch (Exception e) {
            return null;
        }
    }
}

IRule

Interface that defines a “Rule” for a LoadBalancer. A Rule can be thought of as a Strategy for loadbalacing. Well known loadbalancing strategies include Round Robin, Response Time based etc.

IRule接口

public interface IRule{
    /*
     * choose one alive server from lb.allServers or
     * lb.upServers according to key
     */
    public Server choose(Object key);

    public void setLoadBalancer(ILoadBalancer lb);

    public ILoadBalancer getLoadBalancer();    
}

实现类

• AbstractLoadBalancerRule - 抽象类(provides a default implementation for setting and getting load balancer)
• ClientConfigEnabledRoundRobinRule - 内部封装一个RoundRobinRule对象，本质上就是使用RoundRobinRule做轮询负载均衡
• PredicateBasedRule - 使用AbstractServerPredicate做过滤和轮询负载均衡
• ZoneAvoidanceRule - 使用ZoneAvoidancePredicate和AvailabilityPredicate做过滤和轮询负载均衡
• NacosRule - 基于Nacos的NamingService进行服务发现、使用random-weight方式做负载均衡
• RandomRule
• RoundRobinRule

默认使用的是ZoneAvoidanceRule实现，是在RibbonClientConfiguration配置了里面装配的。

@Bean
@ConditionalOnMissingBean
public IRule ribbonRule(IClientConfig config) {
	if (this.propertiesFactory.isSet(IRule.class, name)) {
		return this.propertiesFactory.get(IRule.class, config, name);
	}
	ZoneAvoidanceRule rule = new ZoneAvoidanceRule();
	rule.initWithNiwsConfig(config);
	return rule;
}

ZoneAvoidanceRule类

这个类继承了PredicateBasedRule类，choose方法的实现在父类里面：

// PredicateBasedRule.choose
public Server choose(Object key) {
    ILoadBalancer lb = getLoadBalancer();
    Optional<Server> server = getPredicate().chooseRoundRobinAfterFiltering(lb.getAllServers(), key);
    if (server.isPresent()) {
        return server.get();
    } else {
        return null;
    }       
}

// AbstractServerPredicate.chooseRoundRobinAfterFiltering
public Optional<Server> chooseRoundRobinAfterFiltering(List<Server> servers, Object loadBalancerKey) {
    // 获取有资格的server集
    List<Server> eligible = getEligibleServers(servers, loadBalancerKey);
    if (eligible.size() == 0) {
        return Optional.absent();
    }
    // 轮询
    return Optional.of(eligible.get(incrementAndGetModulo(eligible.size())));
}

ZoneAvoidanceRule类实现了父类的getPredicate方法，返回CompositePredicate对象，封装着ZoneAvoidancePredicate和AvailabilityPredicate用于过滤可用server节点：

private CompositePredicate compositePredicate;

public ZoneAvoidanceRule() {
    super();
    ZoneAvoidancePredicate zonePredicate = new ZoneAvoidancePredicate(this);
    AvailabilityPredicate availabilityPredicate = new AvailabilityPredicate(this);
    compositePredicate = createCompositePredicate(zonePredicate, availabilityPredicate);
}

private CompositePredicate createCompositePredicate(ZoneAvoidancePredicate p1, AvailabilityPredicate p2) {
    return CompositePredicate.withPredicates(p1, p2)
                         .addFallbackPredicate(p2)
                         .addFallbackPredicate(AbstractServerPredicate.alwaysTrue())
                         .build();
}

CompositePredicate是一个代理类，内部使用Predicate链获取可用Server集：

public List<Server> getEligibleServers(List<Server> servers, Object loadBalancerKey) {
    List<Server> result = super.getEligibleServers(servers, loadBalancerKey);
    Iterator<AbstractServerPredicate> i = fallbacks.iterator();
    while (!(result.size() >= minimalFilteredServers && 
             result.size() > (int) (servers.size() * minimalFilteredPercentage))
            && i.hasNext()) {
        AbstractServerPredicate predicate = i.next();
        result = predicate.getEligibleServers(servers, loadBalancerKey);
    }
    return result;
}

ZoneAvoidancePredicate

public boolean apply(PredicateKey input) {
    if (!ENABLED.get()) {
        return true;
    }
    String serverZone = input.getServer().getZone();
    if (serverZone == null) {
        // there is no zone information from the server, we do not want to filter
        // out this server
        return true;
    }
    LoadBalancerStats lbStats = getLBStats();
    if (lbStats == null) {
        // no stats available, do not filter
        return true;
    }
    if (lbStats.getAvailableZones().size() <= 1) {
        // only one zone is available, do not filter
        return true;
    }
    Map<String, ZoneSnapshot> zoneSnapshot = ZoneAvoidanceRule.createSnapshot(lbStats);
    if (!zoneSnapshot.keySet().contains(serverZone)) {
        // The server zone is unknown to the load balancer, do not filter it out 
        return true;
    }
    Set<String> availableZones = ZoneAvoidanceRule
        .getAvailableZones(zoneSnapshot, triggeringLoad.get(), triggeringBlackoutPercentage.get());
    if (availableZones != null) {
        return availableZones.contains(input.getServer().getZone());
    } else {
        return false;
    }
}

AvailabilityPredicate

public boolean apply(PredicateKey input) {
    LoadBalancerStats stats = getLBStats();
    if (stats == null) {
        return true;
    }
    return !shouldSkipServer(stats.getSingleServerStat(input.getServer()));
}

// 判断断路器的状态
private boolean shouldSkipServer(ServerStats stats) {        
    if ((CIRCUIT_BREAKER_FILTERING.get() && stats.isCircuitBreakerTripped()) 
            || stats.getActiveRequestsCount() >= activeConnectionsLimit.get()) {
        return true;
    }
    return false;
}

NacosRule类

public class NacosRule extends AbstractLoadBalancerRule {

	@Autowired
	private NacosDiscoveryProperties nacosDiscoveryProperties;

	@Autowired
	private NacosServiceManager nacosServiceManager;

	@Override
	public Server choose(Object key) {
		try {
			String clusterName = this.nacosDiscoveryProperties.getClusterName();
			String group = this.nacosDiscoveryProperties.getGroup();
			DynamicServerListLoadBalancer loadBalancer = (DynamicServerListLoadBalancer) getLoadBalancer();
			String name = loadBalancer.getName();

            // 使用NamingService服务发现查找Server集
			NamingService namingService = nacosServiceManager
					.getNamingService(nacosDiscoveryProperties.getNacosProperties());
			List<Instance> instances = namingService.selectInstances(name, group, true);
			if (CollectionUtils.isEmpty(instances)) {
				return null;
			}

            // 使用clusterName过滤一下
			List<Instance> instancesToChoose = instances;
			if (StringUtils.isNotBlank(clusterName)) {
				List<Instance> sameClusterInstances = instances.stream()
						.filter(instance -> Objects.equals(clusterName,
								instance.getClusterName()))
						.collect(Collectors.toList());
				if (!CollectionUtils.isEmpty(sameClusterInstances)) {
					instancesToChoose = sameClusterInstances;
				}
			}

            // random-weight算法
			Instance instance = ExtendBalancer.getHostByRandomWeight2(instancesToChoose);

			return new NacosServer(instance);
		} catch (Exception e) {
			return null;
		}
	}

	@Override
	public void initWithNiwsConfig(IClientConfig iClientConfig) {}
}

服务发现

ServerListUpdater

Strategy for DynamicServerListLoadBalancer to use for different ways of doing dynamic server list updates.

用于更新服务节点列表。

public interface ServerListUpdater {

    /**
     * an interface for the updateAction that actually executes a server list update
     */
    public interface UpdateAction {
        void doUpdate();
    }

    /**
     * start the serverList updater with the given update action
     * This call should be idempotent.
     */
    void start(UpdateAction updateAction);

    /**
     * stop the serverList updater. This call should be idempotent
     */
    void stop();

    // 其他方法
}

PollingServerListUpdater实现类：

public synchronized void start(final UpdateAction updateAction) {
    if (isActive.compareAndSet(false, true)) {
        final Runnable wrapperRunnable = new Runnable() {
            @Override
            public void run() {
                if (!isActive.get()) {
                    if (scheduledFuture != null) {
                        scheduledFuture.cancel(true);
                    }
                    return;
                }
                try {
                    updateAction.doUpdate();
                    lastUpdated = System.currentTimeMillis();
                } catch (Exception e) {
                    logger.warn("Failed one update cycle", e);
                }
            }
        };

        // 启动周期调度
        scheduledFuture = getRefreshExecutor().scheduleWithFixedDelay(
                wrapperRunnable,
                initialDelayMs, // 1000
                refreshIntervalMs, // 30 * 1000
                TimeUnit.MILLISECONDS
        );
    }
}

ServerList

Interface that defines the methods sed to obtain the List of Servers.

public interface ServerList<T extends Server> {

    public List<T> getInitialListOfServers();

    /**
     * Return updated list of servers. This is called say every 30 secs
     * (configurable) by the Loadbalancer's Ping cycle
     */
    public List<T> getUpdatedListOfServers();   
}

NacosServerList实现类使用nacos的NamingService做服务发现。

DynamicServerListLoadBalancer

这个实现类里面维护着ServerListUpdater和ServerList对象，启动ServerListUpdater做服务发现：

public class DynamicServerListLoadBalancer<T extends Server> extends BaseLoadBalancer {
    // ...

    volatile ServerList<T> serverListImpl;

    volatile ServerListFilter<T> filter;

    protected final ServerListUpdater.UpdateAction updateAction = new ServerListUpdater.UpdateAction() {
        @Override
        public void doUpdate() {
            updateListOfServers();
        }
    };

    protected volatile ServerListUpdater serverListUpdater;

    // ...

    public void updateListOfServers() {
        List<T> servers = new ArrayList<T>();
        if (serverListImpl != null) {
            servers = serverListImpl.getUpdatedListOfServers();

            if (filter != null) {
                servers = filter.getFilteredListOfServers(servers);
            }
        }
        updateAllServerList(servers);
    }
}

RibbonClientConfiguration

ServerListUpdater是在RibbonClientConfiguration中装配的：

@Bean
@ConditionalOnMissingBean
public ServerListUpdater ribbonServerListUpdater(IClientConfig config) {
	return new PollingServerListUpdater(config);
}

NacosRibbonClientConfiguration

ServerList是在NacosRibbonClientConfiguration中装配的：

@Bean
@ConditionalOnMissingBean
public ServerList<?> ribbonServerList(IClientConfig config,
		NacosDiscoveryProperties nacosDiscoveryProperties) {
	if (this.propertiesFactory.isSet(ServerList.class, config.getClientName())) {
		ServerList serverList = this.propertiesFactory.get(ServerList.class, config,
				config.getClientName());
		return serverList;
	}
	NacosServerList serverList = new NacosServerList(nacosDiscoveryProperties);
	serverList.initWithNiwsConfig(config);
	return serverList;
}