Spring Cloud Ribbon客户端负载均衡(二)

前言

上节我们主要介绍了RestTemplate几种常见的REST请求：Spring Cloud Ribbon客户端负载均衡(一)
本文将通过源码分析来了解Ribbon实现客户端负载均衡的基本原理，从@LoadBalanced注解入手展开

@LoadBalanced

/**
 * Annotation to mark a RestTemplate bean to be configured to use a LoadBalancerClient
 * @author Spencer Gibb
 */
@Target({ ElementType.FIELD, ElementType.PARAMETER, ElementType.METHOD })
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@Qualifier
public @interface LoadBalanced {
}

从@LoadBalanced注解我们可以发现，该注解用来给RestTemplate做标记，以使用负载均衡的客户端（LoadBalancerClient）来配置它。

通过搜索LoadBalancerClient可以发现，这是一个继承ServiceInstanceChooser 接口的接口：

public interface LoadBalancerClient extends ServiceInstanceChooser {
	<T> T execute(String serviceId, LoadBalancerRequest<T> request) throws IOException;
	
	<T> T execute(String serviceId, ServiceInstance serviceInstance, LoadBalancerRequest<T> request) throws IOException;
	
	URI reconstructURI(ServiceInstance instance, URI original);
}

public interface ServiceInstanceChooser {

    /**
     * Choose a ServiceInstance from the LoadBalancer for the specified service
     * @param serviceId the service id to look up the LoadBalancer
     * @return a ServiceInstance that matches the serviceId
     */
    ServiceInstance choose(String serviceId);
}

从上面两个接口中，我们可以通过定义的抽象方法来了解客户端负载均衡器中应具备的几种能力。

• ServiceInstance choose(String serviceId)
  根据传入的服务名serviceId，从负载均衡器中选择对应服务的实例
• T execute(String serviceId, LoadBalancerRequest request)
  使用从负载均衡器中挑选出的服务实例来执行请求内容
• T execute(String serviceId, ServiceInstance serviceInstance, LoadBalancerRequest request)
  作用和上面的方法一样，多了一个serviceInstance参数
• URI reconstructURI(ServiceInstance instance, URI original)
  为系统构建一个合适的host:port形式的URI吗，在分布式系统中，我们使用逻辑上的服务名称作为host来构建URI（替代服务实例的host:port形式）进行请求，比如http://myservice/path/to/service。在该操作的定义中，ServiceInstance对象是带有host和port的具体服务实例，URI对象则是使用逻辑服务名定义为host的URI，而返回的URI则是通过ServiceInstance的服务实例详情拼接出的具体host:post形式的请求地址。

LoadBalancerAutoConfiguration

顺着LoadBalancerClient接口的所属包org.springframework.cloud.client.loadbalancer，可以得出下图所示的关系图

从类的命名上可初步判断LoadBalancerAutoConfiguration为实现客户端负载均衡器的自动化配置类。

@Configuration
@ConditionalOnClass(RestTemplate.class)
@ConditionalOnBean(LoadBalancerClient.class)
@EnableConfigurationProperties(LoadBalancerRetryProperties.class)
public class LoadBalancerAutoConfiguration {

	@LoadBalanced
	@Autowired(required = false)
	private List<RestTemplate> restTemplates = Collections.emptyList();

	@Bean
	public SmartInitializingSingleton loadBalancedRestTemplateInitializerDeprecated(
			final ObjectProvider<List<RestTemplateCustomizer>> restTemplateCustomizers) {
		return () -> restTemplateCustomizers.ifAvailable(customizers -> {
            for (RestTemplate restTemplate : LoadBalancerAutoConfiguration.this.restTemplates) {
                for (RestTemplateCustomizer customizer : customizers) {
                    customizer.customize(restTemplate);
                }
            }
        });
	}

	@Autowired(required = false)
	private List<LoadBalancerRequestTransformer> transformers = Collections.emptyList();

	@Bean
	@ConditionalOnMissingBean
	public LoadBalancerRequestFactory loadBalancerRequestFactory(
			LoadBalancerClient loadBalancerClient) {
		return new LoadBalancerRequestFactory(loadBalancerClient, transformers);
	}

	@Configuration
	@ConditionalOnMissingClass("org.springframework.retry.support.RetryTemplate")
	static class LoadBalancerInterceptorConfig {
		@Bean
		public LoadBalancerInterceptor ribbonInterceptor(
				LoadBalancerClient loadBalancerClient,
				LoadBalancerRequestFactory requestFactory) {
			return new LoadBalancerInterceptor(loadBalancerClient, requestFactory);
		}

		@Bean
		@ConditionalOnMissingBean
		public RestTemplateCustomizer restTemplateCustomizer(
				final LoadBalancerInterceptor loadBalancerInterceptor) {
			return restTemplate -> {
                List<ClientHttpRequestInterceptor> list = new ArrayList<>(
                        restTemplate.getInterceptors());
                list.add(loadBalancerInterceptor);
                restTemplate.setInterceptors(list);
            };
		}
	}

	@Configuration
	@ConditionalOnClass(RetryTemplate.class)
	public static class RetryAutoConfiguration {

		@Bean
		@ConditionalOnMissingBean
		public LoadBalancedRetryFactory loadBalancedRetryFactory() {
			return new LoadBalancedRetryFactory() {};
		}
	}

	@Configuration
	@ConditionalOnClass(RetryTemplate.class)
	public static class RetryInterceptorAutoConfiguration {
		@Bean
		@ConditionalOnMissingBean
		public RetryLoadBalancerInterceptor ribbonInterceptor(
				LoadBalancerClient loadBalancerClient, LoadBalancerRetryProperties properties,
				LoadBalancerRequestFactory requestFactory,
				LoadBalancedRetryFactory loadBalancedRetryFactory) {
			return new RetryLoadBalancerInterceptor(loadBalancerClient, properties,
					requestFactory, loadBalancedRetryFactory);
		}

		@Bean
		@ConditionalOnMissingBean
		public RestTemplateCustomizer restTemplateCustomizer(
				final RetryLoadBalancerInterceptor loadBalancerInterceptor) {
			return restTemplate -> {
                List<ClientHttpRequestInterceptor> list = new ArrayList<>(
                        restTemplate.getInterceptors());
                list.add(loadBalancerInterceptor);
                restTemplate.setInterceptors(list);
            };
		}
	}
}

从LoadBalancerAutoConfiguration类上面的注解可以知道，Ribbon实现的负载均衡自动化配置需要满足下面两个条件：

• @ConditionalOnClass(RestTemplate.class)
  RestTemplate类必须存在于当前工程的环境中
• @ConditionalOnBean(LoadBalancerClient.class)
  在Spring的Bean工程中必须有LoadBalancerClient的实现Bean

在该自动化配置类中，主要做了下面三件事：

• 创建了一个LoadBalancerInterceptor的Bean，用于实现客户端发起请求时进行拦截，以实现客户端负载均衡
• 创建了一个RestTemplateCustomizer的Bean，用于给RestTemplate增加LoadBalancerInterceptor拦截器
• 维护了一个被@LoadBalanced注解修饰的List，并在这里进行初始化，通过调用RestTemplateCustomizer的实例来给需要客户端负载均衡的RestTemplate增加LoadBalancerInterceptor拦截器

LoadBalancerInterceptor拦截器

LoadBalancerInterceptor拦截器是如何将一个普通的RestTemplate变成客户端负载均衡？

public class LoadBalancerInterceptor implements ClientHttpRequestInterceptor {

	private LoadBalancerClient loadBalancer;
	private LoadBalancerRequestFactory requestFactory;

	public LoadBalancerInterceptor(LoadBalancerClient loadBalancer, LoadBalancerRequestFactory requestFactory) {
		this.loadBalancer = loadBalancer;
		this.requestFactory = requestFactory;
	}

	public LoadBalancerInterceptor(LoadBalancerClient loadBalancer) {
		// for backwards compatibility
		this(loadBalancer, new LoadBalancerRequestFactory(loadBalancer));
	}

	@Override
	public ClientHttpResponse intercept(final HttpRequest request, final byte[] body,
			final ClientHttpRequestExecution execution) throws IOException {
		final URI originalUri = request.getURI();
		String serviceName = originalUri.getHost();
		Assert.state(serviceName != null, "Request URI does not contain a valid hostname: " + originalUri);
		return this.loadBalancer.execute(serviceName, requestFactory.createRequest(request, body, execution));
	}
}

可以看到在拦截器中注入了LoadBalancerClient的实现。当一个被@LoadBalanced注解修饰的RestTemplate对象向外发起HTTP请求时，会被LoadBalancerInterceptor类的intercept方法所拦截。在使用RestTemplate时采用了服务名作为host，所以直接从HttpRequest 的URI对象中getHost就能得到服务名，然后调用execute函数根据服务名选择实例并发起请求。

RibbonLoadBalancerClient

LoadBalancerClient其实只是一个抽象的负载均衡器接口，而org.springframework.cloud.netflix.ribbon包下的RibbonLoadBalancerClient才是它的具体实现类。

@Override
	public <T> T execute(String serviceId, LoadBalancerRequest<T> request) throws IOException {
		ILoadBalancer loadBalancer = getLoadBalancer(serviceId);
		Server server = getServer(loadBalancer);
		if (server == null) {
			throw new IllegalStateException("No instances available for " + serviceId);
		}
		RibbonServer ribbonServer = new RibbonServer(serviceId, server, isSecure(server,
				serviceId), serverIntrospector(serviceId).getMetadata(server));

		return execute(serviceId, ribbonServer, request);
	}

	@Override
	public <T> T execute(String serviceId, ServiceInstance serviceInstance, LoadBalancerRequest<T> request) throws IOException {
		Server server = null;
		if(serviceInstance instanceof RibbonServer) {
			server = ((RibbonServer)serviceInstance).getServer();
		}
		if (server == null) {
			throw new IllegalStateException("No instances available for " + serviceId);
		}

		RibbonLoadBalancerContext context = this.clientFactory
				.getLoadBalancerContext(serviceId);
		RibbonStatsRecorder statsRecorder = new RibbonStatsRecorder(context, server);

		try {
			T returnVal = request.apply(serviceInstance);
			statsRecorder.recordStats(returnVal);
			return returnVal;
		}
		// catch IOException and rethrow so RestTemplate behaves correctly
		catch (IOException ex) {
			statsRecorder.recordStats(ex);
			throw ex;
		}
		catch (Exception ex) {
			statsRecorder.recordStats(ex);
			ReflectionUtils.rethrowRuntimeException(ex);
		}
		return null;
	}

可以看到。在execute函数的实现中，第一步做的就是通过getServer根据传入的服务名serviceId去获得具体的服务实例

protected Server getServer(ILoadBalancer loadBalancer) {
		if (loadBalancer == null) {
			return null;
		}
		return loadBalancer.chooseServer("default"); // TODO: better handling of key
	}

通过getServer函数的实现源码，发现这里获取具体服务实例的时候并没有使用LoadBalancerClient接口的choose函数，而是使用了Netflix Ribbon自身的ILoadBalancer接口中定义的chooseServer函数，具体实现则是在ZoneAwareLoadBalancer类。

接下来我们看一下ILoadBalancer 这个接口

public interface ILoadBalancer {
	//向负载均衡器中维护的实例列表增加服务实例
	public void addServers(List<Server> newServers);
	
	//通过某种策略从负载均衡器中选择一个具体的服务实例
	public Server chooseServer(Object key);
	
	//用来标识负载均衡器中的某个具体实例已经停止服务
	public void markServerDown(Server server);
	
	//获取当前正常服务的实例列表
    public List<Server> getReachableServers();
    
	//获取所有已知的服务实例列表，包括正常服务和停止服务的实例
	public List<Server> getAllServers();
}

对ILoadBalancer接口的实现，我们整理出如下所示的结构，可以看到BaseLoadBalancer类实现了基础的负载均衡，而DynamicServerListLoadBalancer和ZoneAwareLoadBalancer在负载均衡的策略上做了一些功能扩展。

回到RibbonLoadBalancerClient的execute函数逻辑，在通过ZoneAwareLoadBalancer的chooseServer函数获取了负载均衡策略分配到的服务实例对象Server之后，将其内容包装成RibbonServer对象，然后使用该对象再回调LoadBalancerRequest的apply(final ServiceInstance instance)函数，向一个实际的具体服务实例发起请求，从而实现一开始以服务名为host的URI请求到host:post形式的实际访问地址的转换。
在**apply(final ServiceInstance instance)**函数中传入的ServiceInstance接口对象是对服务实例的抽象定义。在该接口中暴露了服务治理系统中每个服务实例需要提供的一些基本信息。

public interface ServiceInstance {

	/**
	 * @return the service id as registered.
	 */
	String getServiceId();

	/**
	 * @return the hostname of the registered ServiceInstance
	 */
	String getHost();

	/**
	 * @return the port of the registered ServiceInstance
	 */
	int getPort();

	/**
	 * @return if the port of the registered ServiceInstance is https or not
	 */
	boolean isSecure();

	/**
	 * @return the service uri address
	 */
	URI getUri();

	/**
	 * @return the key value pair metadata associated with the service instance
	 */
	Map<String, String> getMetadata();
	}

之前提到的具体包装Server服务实例的RibbonServer对象就是ServiceInstance接口的实现

public static class RibbonServer implements ServiceInstance {
		private final String serviceId;
		private final Server server;
		private final boolean secure;
		private Map<String, String> metadata;

		public RibbonServer(String serviceId, Server server) {
			this(serviceId, server, false, Collections.<String, String> emptyMap());
		}

		public RibbonServer(String serviceId, Server server, boolean secure,
				Map<String, String> metadata) {
			this.serviceId = serviceId;
			this.server = server;
			this.secure = secure;
			this.metadata = metadata;
		}
		...
	}	

那么apply(final ServiceInstance instance)函数在接收到具体的ServiceInstance 实例后是如果操作的？

public ListenableFuture<ClientHttpResponse> apply(final ServiceInstance instance)
							throws Exception {
						HttpRequest serviceRequest = new ServiceRequestWrapper(request,
								instance, loadBalancer);
						return execution.executeAsync(serviceRequest, body);
					}

从apply的实现中还传入了ServiceRequestWrapper对象，该对象继承了HttpRequestWrapper并重写了getURI函数，重写后的getURI通过调用LoadBalancerClient接口的reconstructURI函数来重新构建一个URI来进行访问。

public class ServiceRequestWrapper extends HttpRequestWrapper {
	private final ServiceInstance instance;
	private final LoadBalancerClient loadBalancer;

	public ServiceRequestWrapper(HttpRequest request, ServiceInstance instance,
								 LoadBalancerClient loadBalancer) {
		super(request);
		this.instance = instance;
		this.loadBalancer = loadBalancer;
	}

	@Override
	public URI getURI() {
		URI uri = this.loadBalancer.reconstructURI(
				this.instance, getRequest().getURI());
		return uri;
	}
}

进入到getURI()函数，可以看到回调了RibbonLoadBalancerClient的reconstructURI(ServiceInstance instance, URI original)函数

@Override
	public URI reconstructURI(ServiceInstance instance, URI original) {
		Assert.notNull(instance, "instance can not be null");
		String serviceId = instance.getServiceId();
		RibbonLoadBalancerContext context = this.clientFactory
				.getLoadBalancerContext(serviceId);

		URI uri;reconstructURI
		Server server;
		if (instance instanceof RibbonServer) {
			RibbonServer ribbonServer = (RibbonServer) instance;
			server = ribbonServer.getServer();
			uri = updateToSecureConnectionIfNeeded(original, ribbonServer);
		} else {
			server = new Server(instance.getScheme(), instance.getHost(), instance.getPort());
			IClientConfig clientConfig = clientFactory.getClientConfig(serviceId);
			ServerIntrospector serverIntrospector = serverIntrospector(serviceId);
			uri = updateToSecureConnectionIfNeeded(original, clientConfig,
					serverIntrospector, server);
		}
		return context.reconstructURIWithServer(server, uri);
	}

从reconstructURI函数中我们可以看到，它通过ServiceInstance实例对象的serviceId，从SpringClientFactory类的clientFactory对象中获取对应的serviceId的负载均衡的上下文RibbonLoadBalancerContext对象。然后根据ServiceInstance中的信息来构建具体服务实例信息的Server对象，并使用RibbonLoadBalancerContext的reconstructURIWithServer(Server server, URI original)函数来构建服务实例的URI。
这里提到的SpringClientFactory类是一个用来创建客户端负载均衡器的工厂类，该工厂类会为每一个不同名的Ribbon客户端生成不同的Spring上下文。RibbonLoadBalancerContext类是LoadBalancerContext的子类，该类用于存储一些被负载均衡器使用的上下文内容和API操作。

再来看一下LoadBalancerContext 的reconstructURIWithServer函数的具体实现。

public class LoadBalancerContext implements IClientConfigAware {
	...
	public URI reconstructURIWithServer(Server server, URI original) {
        String host = server.getHost();
        int port = server.getPort();
        String scheme = server.getScheme();
        
        if (host.equals(original.getHost()) 
                && port == original.getPort()
                && scheme == original.getScheme()) {
            return original;
        }
        if (scheme == null) {
            scheme = original.getScheme();
        }
        if (scheme == null) {
            scheme = deriveSchemeAndPortFromPartialUri(original).first();
        }

        try {
            StringBuilder sb = new StringBuilder();
            sb.append(scheme).append("://");
            if (!Strings.isNullOrEmpty(original.getRawUserInfo())) {
                sb.append(original.getRawUserInfo()).append("@");
            }
            sb.append(host);
            if (port >= 0) {
                sb.append(":").append(port);
            }
            sb.append(original.getRawPath());
            if (!Strings.isNullOrEmpty(original.getRawQuery())) {
                sb.append("?").append(original.getRawQuery());
            }
            if (!Strings.isNullOrEmpty(original.getRawFragment())) {
                sb.append("#").append(original.getRawFragment());
            }
            URI newURI = new URI(sb.toString());
            return newURI;            
        } catch (URISyntaxException e) {
            throw new RuntimeException(e);
        }
    }
}

从reconstructURIWithServer的实现中可以看到，它同reconstructURI的定义类似。只是reconstructURI的第一个参数为Spring Cloud定义的ServiceInstance，而reconstructURIWithServer的第一个参数是Netflix中定义的Server,所以在RibbonLoadBalancerClient实现reconstructURI的时候做了一次转换，使用ServiceInstance的host和port信息构建了一个Server对象来给reconstructURIWithServer使用。reconstructURIWithServer从Server对象中获取host和port信息，然后根据以服务名为host的URI对象original中获取其他请求信息，再进行拼接整合，形成最终要访问的服务实例的具体地址。

小结

Spring Cloud Ribbon中实现客户端负载均衡，通过LoadBalancerInterceptor拦截器对RestTemplate的请求进行拦截，并利用Spring Cloud的负载均衡器LoadBalancerClient将以逻辑服务名为host的URI转换成具体的服务实例地址的过程。同时通过分析LoadBalancerClient的Ribbon实现RibbonLoadBalancerClient，可以知道在使用Ribbon实现负载均衡器的时候，实际使用的还是会Ribbon中定义的ILoadBanlancer接口的实现，自动化配置会采用ZoneAwareLoadBalancer的实例来实现客户端负载均衡。