ServiceComb

ServiceComb-java-chassis

参考资料：

GitHub

官网

如何设计一个优质的微服务框架：Apache ServiceComb 的开放性设计

社区文章

Java-Chassis是ServiceComb提供的Java SDK，其关键特性如下：

• 以契约为基础：通过OpenAPI定义服务契约，服务间通信以契约为基础，实现编程模型与通信模型解耦
• 高性能：基于Vertx实现纯异步内核，对外提供同步和异步两种调用方式，支持高吞吐低时延通信
• 多协议网络通信：支持HTTP/RESTful、TCP/私有协议两种通信方式，可快速切换
• 多编程模型：支持SpringMVC、透明RPC编程模型，支持同步、异步编程模式，遗留系统可低成本接入

1. 在Main方法中调用BeanUtils.init()。
2. 通过pring-boot-starter的方式在Application类上添加@EnableServiceComb注解。

这两种方式都是通过加载classpath:META-INF\spring\*.bean.xml和classpath*:META-INF/spring/scb-core-bean.xml配置文件来初始化ServiceComb中的bean实例，只不过第一种方式是自己通过代码加载，第二种方式是通过SpringBoot加载。

第二种方式中，定义了一个@EnableServiceComb注解，引入了ServiceCombSpringConfiguration配置类，该类中通过ImportResource导入了指定的bean配置文件。

@EnableServiceComb

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Import(ServiceCombSpringConfiguration.class)
public @interface EnableServiceComb {

}

@Configuration
@ImportResource({BeanUtils.DEFAULT_BEAN_CORE_RESOURCE, BeanUtils.DEFAULT_BEAN_NORMAL_RESOURCE})
class ServiceCombSpringConfiguration {
  @Inject
  public void setCseApplicationListener(CseApplicationListener cseApplicationListener) {
    cseApplicationListener.setInitEventClass(ApplicationReadyEvent.class);
  }
}

classpath:META-INF\spring*.bean.xml、classpath:META-INF/spring/scb-core-bean.xml*

两者主要做了两件事

1、开启了spring注解的自动扫描

<context:annotation-config/>
<!-- <context:spring-configured /> -->
<context:component-scan base-package="${scb-scan-package:org.apache.servicecomb}"/>

2、加载了ConfigurationSpringInitializer、CseApplicationListener、GroupExecutor

ConfigurationSpringInitializer：SCBEngine容器启动前对环境变量的操作。

CseApplicationListener：依赖于spring容器负责整个SCBEngine容器的启动。

GroupExecutor：内部用到的线程池。

<!-- initializer doing before any bean -->
<bean class="org.apache.servicecomb.core.ConfigurationSpringInitializer"/>
<!-- initializer after any bean initialized -->
<bean class="org.apache.servicecomb.core.CseApplicationListener"/>

<bean id="cse.executor.groupThreadPool" class="org.apache.servicecomb.core.executor.GroupExecutor"
  init-method="init"/>
<alias name="cse.executor.groupThreadPool" alias="cse.executor.default"/>
<alias name="cse.executor.groupThreadPool" alias="servicecomb.executor.groupThreadPool"/>

ConfigurationSpringInitializer

完成了配置文件的读取，其中包括与spring配置文件的合并和cse.xxx->servivecomb.xxx等转化。

ConfigurationSpringInitializer实现了EnvironmentAware 
在执行后置处理器实例化之前方法 BeanPostProcessor.postProcessBeforeInitialization会回调所有Aware方法，此处会调用EnvironmentAware.setEnvironment
ConfigurationSpringInitializer.setEnvironment()
内部完成了获取yaml文件，创建动态配置对象，Environment对象和ConcurrentCompositeConfiguration的配置同步。

CseApplicationListener

该对象作为程序入口，主要负责拉起整个ServiceComb引擎，会进行所有资源的初始化。该对象中的初始化逻辑分为如下两个步骤：

1. 该类实现了ApplicationContextAware接口，对象创建之后会自动注入ApplicationContext对象。之后会进行HttpClients的加载，以及RegistrationManager和DiscoveryManager的初始化。
2. 该类实现了ApplicationListener接口，可以监听ApplicationEvent事件。在ApplicationContext初始化完成发布ContextRefreshedEvent事件时触发下一个初始化步骤，此时会创建SCBEngine对象，SCBEngine对象负责管理整个引擎中用到的所有资源，调用SCBEngine实例的run()方法会创建ProducerMicroserviceMeta，并依次进行ProducerHandlerManager、ConsumerHandlerManager、FilterChainsManager、ProducerProviderManager、ConsumerProviderManager、TransportManager这些对象的初始化工作，这些Manager分别负责相应资源的管理，这里就不一一详细说明了。初始化完成之后最后还会通过RegistrationManager和DiscoveryManager的run方法将服务注册和发现的流程启动起来。

public class CseApplicationListener
    implements ApplicationListener<ApplicationEvent>, Ordered, ApplicationContextAware {
  private Class<?> initEventClass = ContextRefreshedEvent.class;

  private ApplicationContext applicationContext;

  @Override
  public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
    if (this.applicationContext == applicationContext) {
      // same object. avoid initialize many times.
      return;
    }
    // 将spring容器赋给变量，由后续的onApplicationEvent赋值给SCBEngine
    this.applicationContext = applicationContext;
    // 将spring容器赋给BeanUtils，便于静态方法获取bean，BeanUtils.getBean()
    BeanUtils.setContext(applicationContext);
    // 初始化HttpClients
    HttpClients.load();
    // 初始化对象
    RegistrationManager.INSTANCE.init();
    DiscoveryManager.INSTANCE.init();
  }

  public void setInitEventClass(Class<?> initEventClass) {
    this.initEventClass = initEventClass;
  }

  @Override
  public int getOrder() {
    // should run before default listener, eg: ZuulConfiguration
    return -1000;
  }

  @Override
  public void onApplicationEvent(ApplicationEvent event) {
    if (initEventClass.isInstance(event)) {
      if (applicationContext instanceof AbstractApplicationContext) {
        ((AbstractApplicationContext) applicationContext).registerShutdownHook();
      }

      SCBEngine scbEngine = SCBEngine.getInstance();
      //SCBEngine init first, hence we do not need worry that when other beans need use the
      //producer microserviceMeta, the SCBEngine is not inited.
//        String serviceName = RegistryUtils.getMicroservice().getServiceName();
//        SCBEngine.getInstance().setProducerMicroserviceMeta(new MicroserviceMeta(serviceName).setConsumer(false));
//        SCBEngine.getInstance().setProducerProviderManager(applicationContext.getBean(ProducerProviderManager.class));
//        SCBEngine.getInstance().setConsumerProviderManager(applicationContext.getBean(ConsumerProviderManager.class));
//        SCBEngine.getInstance().setTransportManager(applicationContext.getBean(TransportManager.class));
      scbEngine.setApplicationContext(applicationContext);
      scbEngine.setFilterChainsManager(applicationContext.getBean(FilterChainsManager.class));
      scbEngine.getConsumerProviderManager().getConsumerProviderList()
          .addAll(applicationContext.getBeansOfType(ConsumerProvider.class).values());
      scbEngine.getProducerProviderManager().getProducerProviderList()
          .addAll(applicationContext.getBeansOfType(ProducerProvider.class).values());
      scbEngine.addBootListeners(applicationContext.getBeansOfType(BootListener.class).values());
      // 启动核心方法
      scbEngine.run();
    } else if (event instanceof ContextClosedEvent) {
      if (SCBEngine.getInstance() != null) {
        SCBEngine.getInstance().destroy();
      }
    }
  }
}

BeanUtils

主程序入口

public final class BeanUtils {
  private static final Logger LOGGER = LoggerFactory.getLogger(BeanUtils.class);

  /**
   * 指定Spring扫描的文件
   */
  public static final String DEFAULT_BEAN_CORE_RESOURCE = "classpath*:META-INF/spring/scb-core-bean.xml";

  public static final String DEFAULT_BEAN_NORMAL_RESOURCE = "classpath*:META-INF/spring/*.bean.xml";

  public static final String[] DEFAULT_BEAN_RESOURCE = new String[] {DEFAULT_BEAN_CORE_RESOURCE
      , DEFAULT_BEAN_NORMAL_RESOURCE};

  public static final String SCB_SCAN_PACKAGE = "scb-scan-package";

  private static final String SCB_PACKAGE = "org.apache.servicecomb";

  private static ApplicationContext context;

  private BeanUtils() {
  }

  public static void init() {
    init(DEFAULT_BEAN_RESOURCE);
  }

  public static void init(String... configLocations) {
    // 准备ServiceComb的扫描路径，将其放到System中
    prepareServiceCombScanPackage();
    // xml资源文件
    Set<String> locationSet = new LinkedHashSet<>();
    addBeanLocation(locationSet, DEFAULT_BEAN_RESOURCE);
    addBeanLocation(locationSet, configLocations);
    // 启动Spring容器
    context = new ClassPathXmlApplicationContext(locationSet.toArray(new String[locationSet.size()]));
  }

  public static void addBeanLocation(Set<String> locationSet, String... location) {
    Arrays.stream(location).forEach(loc -> addBeanLocation(locationSet, loc));
  }

  public static void addBeanLocation(Set<String> locationSet, String location) {
    if (location == null) {
      return;
    }

    location = location.trim();
    if (StringUtils.isNotEmpty(location)) {
      locationSet.add(location);
    }
  }

  private static void addItem(Set<String> set, String item) {
    for (String it : set) {
      if (item.startsWith(it)) {
        return;
      }
    }
    set.add(item);
  }

  public static void prepareServiceCombScanPackage() {
    Set<String> scanPackags = new LinkedHashSet<>();
    // add exists settings
    String exists = System.getProperty(SCB_SCAN_PACKAGE);
    if (exists != null) {
      for (String exist : exists.trim().split(",")) {
        if (!exist.isEmpty()) {
          addItem(scanPackags, exist.trim());
        }
      }
    }

    // ensure servicecomb package exist
    addItem(scanPackags, SCB_PACKAGE);

    // add main class package
    for (Class<?> mainClass : new Class<?>[] {JvmUtils.findMainClass(), JvmUtils.findMainClassByStackTrace()}) {
      if (mainClass != null && mainClass.getPackage() != null) {
        String pkg = mainClass.getPackage().getName();
        addItem(scanPackags, pkg);
      }
    }

    // finish
    String scbScanPackages = StringUtils.join(scanPackags, ",");
    System.setProperty(SCB_SCAN_PACKAGE, scbScanPackages);
    LOGGER.info("Scb scan package list: " + scbScanPackages);
  }

  public static ApplicationContext getContext() {
    return context;
  }

  public static void setContext(ApplicationContext applicationContext) {
    context = applicationContext;
  }

  /**
   * 不应该在业务流程中频繁调用，因为内部必然会加一个锁做互斥，会影响并发度
   */
  @SuppressWarnings("unchecked")
  public static <T> T getBean(String name) {
    return (T) context.getBean(name);
  }

  public static <T> Map<String, T> getBeansOfType(Class<T> type) {
    if (context == null) {
      // for some test case
      return Collections.emptyMap();
    }
    return context.getBeansOfType(type);
  }

  public static <T> T getBean(Class<T> type) {
    if (context == null) {
      // for some test case
      return null;
    }
    return context.getBean(type);
  }

  /**
   * Get the implemented class of the given instance
   * @param bean the instance to get implemented class from
   * @return the implemented class (if the checked class is proxied, return the ultimate target class)
   * @see org.springframework.aop.framework.AopProxyUtils#ultimateTargetClass
   */
  public static Class<?> getImplClassFromBean(Object bean) {
    return AopProxyUtils.ultimateTargetClass(bean);
  }

  public static <T extends SPIOrder & SPIEnabled> void addBeans(Class<T> cls, List<T> exists) {
    if (context == null) {
      return;
    }

    for (T instance : exists) {
      context.getAutowireCapableBeanFactory().autowireBean(instance);
    }
    for (T bean : context.getBeansOfType(cls).values()) {
      if (bean.enabled()) {
        exists.add(bean);
      }
    }
    exists.sort(Comparator.comparingInt(SPIOrder::getOrder));
  }
}

SCBEngine.doRun()

启动核心逻辑

  private void doRun() throws Exception {
    status = SCBStatus.STARTING;

    bootListeners.sort(Comparator.comparingInt(BootListener::getOrder));

    // 触发事件，此处为空
    triggerEvent(EventType.BEFORE_HANDLER);
    // 从classpath*:config/cse.handler.xml和handler.xml结尾文件中获取当前环境中配置的Handle,
    // 并获取配置文件中需要添加到Handler链中的handler名字，servicecomb.handler.chain.Consumer.default、servicecomb.handler.chain.Provider.default
    HandlerConfigUtils.init(consumerHandlerManager, producerHandlerManager);
    triggerEvent(EventType.AFTER_HANDLER);

    triggerEvent(EventType.BEFORE_FILTER);
    // 收集分类各种Filter，此时已有spring加载完所有的Filter。
    filterChainsManager.init();
    triggerEvent(EventType.AFTER_FILTER);

    // 创建producerMicroserviceMeta，将Handler和Filter赋值给producerMicroserviceMeta
    createProducerMicroserviceMeta();

    triggerEvent(EventType.BEFORE_PRODUCER_PROVIDER);
    // 完成契约的扫描rest和pojo,RestProducers和PojoProducers实现了BeanPostProcessor,
    // RestProducers将类上添加了@RestSchema和@RestContronller的
    producerProviderManager.init();
    triggerEvent(EventType.AFTER_PRODUCER_PROVIDER);

    triggerEvent(EventType.BEFORE_CONSUMER_PROVIDER);
    // 此处为空方法
    consumerProviderManager.init();
    triggerEvent(EventType.AFTER_CONSUMER_PROVIDER);

    triggerEvent(EventType.BEFORE_TRANSPORT);
    // 核心方法，拉起整个scb的通信容器，包括客户端和服务端，创建对应verticle绑定对应端口,启动rest和highway服务
    transportManager.init(this);
    triggerEvent(EventType.AFTER_TRANSPORT);

    triggerEvent(EventType.BEFORE_REGISTRY);

    triggerAfterRegistryEvent();

    RegistrationManager.INSTANCE.run();
    DiscoveryManager.INSTANCE.run();

    shutdownHook = new Thread(this::destroyForShutdownHook);
    Runtime.getRuntime().addShutdownHook(shutdownHook);
  }

BootListener

启动监听类，覆盖servicecomb整个生命周期，穿插在SCBEngine.doRun()整个方法中，提供了前置和后置方法。

BootEvent内置SCBEngine对象，BootListener.onBootEvent时将BootEvent传入。

public interface BootListener {
  enum EventType {
    BEFORE_HANDLER,
    AFTER_HANDLER,
    BEFORE_FILTER,
    AFTER_FILTER,
    BEFORE_PRODUCER_PROVIDER,
    AFTER_PRODUCER_PROVIDER,
    BEFORE_CONSUMER_PROVIDER,
    AFTER_CONSUMER_PROVIDER,
    BEFORE_TRANSPORT,
    AFTER_TRANSPORT,
    BEFORE_REGISTRY,
    AFTER_REGISTRY,
    BEFORE_CLOSE,
    AFTER_CLOSE
  }

  class BootEvent {
    private SCBEngine scbEngine;

    private EventType eventType;

    public SCBEngine getScbEngine() {
      return scbEngine;
    }

    public void setScbEngine(SCBEngine scbEngine) {
      this.scbEngine = scbEngine;
    }

    public EventType getEventType() {
      return eventType;
    }

    public void setEventType(EventType eventType) {
      this.eventType = eventType;
    }
  }

  default int getOrder() {
    return 0;
  }

  default void onBootEvent(BootEvent event) {
    switch (event.eventType) {
      case BEFORE_HANDLER:
        onBeforeHandler(event);
        return;
      case AFTER_HANDLER:
        onAfterHandler(event);
        return;
      case BEFORE_FILTER:
        onBeforeFilter(event);
        return;
      case AFTER_FILTER:
        onAfterFilter(event);
        return;
      case BEFORE_PRODUCER_PROVIDER:
        onBeforeProducerProvider(event);
        return;
      case AFTER_PRODUCER_PROVIDER:
        onAfterProducerProvider(event);
        return;
      case BEFORE_CONSUMER_PROVIDER:
        onBeforeConsumerProvider(event);
        return;
      case AFTER_CONSUMER_PROVIDER:
        onAfterConsumerProvider(event);
        return;
      case BEFORE_TRANSPORT:
        onBeforeTransport(event);
        return;
      case AFTER_TRANSPORT:
        onAfterTransport(event);
        return;
      case BEFORE_REGISTRY:
        onBeforeRegistry(event);
        return;
      case AFTER_REGISTRY:
        onAfterRegistry(event);
        return;
      case BEFORE_CLOSE:
        onBeforeClose(event);
        return;
      case AFTER_CLOSE:
        onAfterClose(event);
        return;
      default:
        throw new IllegalStateException("unknown boot event type: " + event.eventType);
    }
  }

  default void onBeforeHandler(BootEvent event) {}

  default void onAfterHandler(BootEvent event) {}

  default void onBeforeFilter(BootEvent event) {}

  default void onAfterFilter(BootEvent event) {}

  default void onBeforeProducerProvider(BootEvent event) {}

  default void onAfterProducerProvider(BootEvent event) {}

  default void onBeforeConsumerProvider(BootEvent event) {}

  default void onAfterConsumerProvider(BootEvent event) {}

  default void onBeforeTransport(BootEvent event) {}

  default void onAfterTransport(BootEvent event) {}

  default void onBeforeRegistry(BootEvent event) {}

  default void onAfterRegistry(BootEvent event) {}

  default void onBeforeClose(BootEvent event) {}

  default void onAfterClose(BootEvent event) {}
}

ProducerProvider

PojoProducerProvider和RestProducerProvider是其子类，init 方法主要获取ProducerMeta 用于存放符合规范的接口类，内部有三个属性。

PojoProducerProvider和RestProducerProvider分别有自己获取ProducerMeta的策略，分别是PojoProducers和RestProducers，其两者都实现了BeanPostProcessor，利用其postProcessAfterInitialization接口完成aop后的逻辑，将其存在内部变量producerMetaList中，ProducerProvider的init方法就是获取此属性的值。

public interface ProducerProvider {
  /**
   * 获取ProducerMetas
   * ProducerMeta由PojoProducers或者RestProducers的后置处理器方法生成
   * @return
   */
  List<ProducerMeta> init();

  String getName();
}

public class ProducerMeta {
    // schema注解的值或者类的全限名
  private String schemaId;
	// 对应的类实例
  private Object instance;
	// 对应的接口，pojo时用
  private Class<?> schemaInterface;
}

PojoProducers

/**
 * 筛选出添加了@RpcSchema注解的类，且此类必须实现接口，将其添加到producerMetaList
 */
@Component
public class PojoProducers implements BeanPostProcessor {
  private List<ProducerMeta> producerMetas = new ArrayList<>();

  public synchronized void registerPojoProducer(PojoProducerMeta pojoProducer) {
    producerMetas.add(pojoProducer);
  }

  public List<ProducerMeta> getProducerMetas() {
    return producerMetas;
  }

  @Override
  public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException {
    return bean;
  }

  @Override
  public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
    processProvider(beanName, bean);
    return bean;
  }

  protected void processProvider(String beanName, Object bean) {
    // aop后，新的实例的父类可能是原class，也可能只是个proxy，父类不是原class
    // 所以，需要先取出原class，再取标注
    Class<?> beanCls = BeanUtils.getImplClassFromBean(bean);
    if (beanCls == null) {
      return;
    }
    RpcSchema rpcSchema = beanCls.getAnnotation(RpcSchema.class);
    if (rpcSchema == null) {
      return;
    }

    String schemaId = rpcSchema.schemaId();
    if (StringUtils.isEmpty(schemaId)) {
      Class<?>[] intfs = beanCls.getInterfaces();
      if (intfs.length == 1) {
        schemaId = intfs[0].getName();
      } else {
        throw new Error("Must be schemaId or implements only one interface");
      }
    }

    PojoProducerMeta pojoProducerMeta = new PojoProducerMeta();
    pojoProducerMeta.setSchemaId(schemaId);
    pojoProducerMeta.setSchemaInterface(rpcSchema.schemaInterface());
    pojoProducerMeta.setInstance(bean);

    registerPojoProducer(pojoProducerMeta);
  }
}

RestProducers

/**
 * 筛选出添加了@RestSchema和@RestController注解的类，将其添加到producerMetaList
 */
@Component
public class RestProducers implements BeanPostProcessor {
  /**
   * 用于存放符合规范的ProducerMeta
   */
  private List<ProducerMeta> producerMetaList = new ArrayList<>();
  /**
   * 标识添加了RestController的类也可以添加到ProducerMeta
   * 此处有一个前提条件servicecomb.provider.rest.scanRestController，此配置项必须不为false
   */
  @SuppressWarnings("unchecked")
  private Class<? extends Annotation> restControllerCls = (Class<? extends Annotation>) ReflectUtils
      .getClassByName("org.springframework.web.bind.annotation.RestController");

  private boolean scanRestController = restControllerCls != null &&
      DynamicPropertyFactory.getInstance().getBooleanProperty(RestConst.PROVIDER_SCAN_REST_CONTROLLER, true).get();

  public List<ProducerMeta> getProducerMetaList() {
    return producerMetaList;
  }

  @Override
  public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException {
    return bean;
  }

  @Override
  public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
    processProvider(beanName, bean);

    return bean;
  }

  protected void processProvider(String beanName, Object bean) {
    // aop后，新的实例的父类可能是原class，也可能只是个proxy，父类不是原class
    // 所以，需要先取出原class，再取标注
    Class<?> beanCls = BeanUtils.getImplClassFromBean(bean);
    if (beanCls == null) {
      return;
    }
    // 处理添加了RestSchema的类
    RestSchema restSchema = beanCls.getAnnotation(RestSchema.class);
    if (restSchema != null) {
      ProducerMeta producerMeta = new ProducerMeta(restSchema.schemaId(), bean);
      producerMeta.setSchemaInterface(restSchema.schemaInterface());
      producerMetaList.add(producerMeta);
      return;
    }
    // 处理添加了RestController的类并且servicecomb.provider.rest.scanRestController此配置项不能设置为false
    if (scanRestController && beanCls.getAnnotation(restControllerCls) != null) {
      ProducerMeta producerMeta = new ProducerMeta(beanCls.getName(), bean);
      producerMetaList.add(producerMeta);
    }
  }
}

Vert.x

servicecomb客户端和服务端都是基于vertx进行非阻塞的异步通信

Transport

通信模型，由TransportManager管理

ServletRestTransport：运行于WEB容器的通信模型。

VertxRestTransport：运行于HTTP之上的通信模型，不依赖WEB容器。

HighwayTransport：提供高性能的私有通信协议，仅用于java之间互通，TCP协议。

VertxRestTransport

和HighwayTransport很相似，但是对应的verticle中有好很多可扩展的逻辑，如dispatcher、全局失败过滤器、HttpServerExceptionHandler等

public class VertxRestTransport extends AbstractTransport {
  private static final Logger LOGGER = LoggerFactory.getLogger(VertxRestTransport.class);

  private RestTransportClient restClient;

  @Override
  public String getName() {
    return Const.RESTFUL;
  }

  @Override
  public int getOrder() {
    return -1000;
  }

  @Override
  public boolean canInit() {
    setListenAddressWithoutSchema(TransportConfig.getAddress());

    URIEndpointObject ep = (URIEndpointObject) getEndpoint().getAddress();
    if (ep == null) {
      return true;
    }

    if (!NetUtils.canTcpListen(ep.getSocketAddress().getAddress(), ep.getPort())) {
      LOGGER.warn(
          "Can not start VertxRestTransport, the port:{} may have been occupied. You can ignore this message if you are using a web container like tomcat.",
          ep.getPort());
      return false;
    }

    return true;
  }

  @Override
  public boolean init() throws Exception {
    restClient = RestTransportClientManager.INSTANCE.getRestClient();

    // 部署transport server
    // 默认开启EventLoop线程模型，默认部署当前核心数个，最大8个
    DeploymentOptions options = new DeploymentOptions().setInstances(TransportConfig.getThreadCount());
    SimpleJsonObject json = new SimpleJsonObject();
    json.put(ENDPOINT_KEY, getEndpoint());
    json.put(RestTransportClient.class.getName(), restClient);
    options.setConfig(json);
    options.setWorkerPoolName("pool-worker-transport-rest");
    options.setWorkerPoolSize(VertxOptions.DEFAULT_WORKER_POOL_SIZE);
    // 部署服务 ,传入Verticle
    return VertxUtils.blockDeploy(transportVertx, TransportConfig.getRestServerVerticle(), options);
  }

  @Override
  public void send(Invocation invocation, AsyncResponse asyncResp) throws Exception {
    restClient.send(invocation, asyncResp);
  }
}

RestServerVerticle

Vertx核心类，继承自AbstractVerticle，添加HTTP Dispatcher逻辑，创建HTTPServer，并开启端口监听。

1、设置是否打印请求log

2、处理是否允许跨域

3、匹配对应分发器，重要方法，收集所有VertxHttpDispatcher，排序完后对Router进行添加Handle

4、全局Rest失败的处理器，内置了默认逻辑，如果提供了GlobalRestFailureHandler，将使用自己提供的。

5、获取所有HttpServerExceptionHandler，异常后依次获取执行其HttpServerExceptionHandler.handle()。

public class RestServerVerticle extends AbstractVerticle {
  private static final Logger LOGGER = LoggerFactory.getLogger(RestServerVerticle.class);

  private static final String SSL_KEY = "rest.provider";

  private Endpoint endpoint;

  private URIEndpointObject endpointObject;

  @Override
  public void init(Vertx vertx, Context context) {
    super.init(vertx, context);
    this.endpoint = (Endpoint) context.config().getValue(AbstractTransport.ENDPOINT_KEY);
    this.endpointObject = (URIEndpointObject) endpoint.getAddress();
  }

  /**
   * Verticle核心方法
   * @param startFuture
   * @throws Exception
   */
  @SuppressWarnings("deprecation")
  // TODO: vert.x 3.8.3 does not update startListen to promise, so we keep use deprecated API now. update in newer version.
  @Override
  public void start(Future<Void> startFuture) throws Exception {
    try {
      super.start();
      // 如果本地未配置地址，则表示不必监听，只需要作为客户端使用即可
      if (endpointObject == null) {
        LOGGER.warn("rest listen address is not configured, will not start.");
        startFuture.complete();
        return;
      }
      Router mainRouter = Router.router(vertx);
      // 设置是否打印请求log
      mountAccessLogHandler(mainRouter);
      // 处理是否允许跨域
      mountCorsHandler(mainRouter);
      // 匹配对应分发器
      // 重要方法
      // 收集所有VertxHttpDispatcher，排序完后对Router进行添加Handle
      initDispatcher(mainRouter);
      //全局Rest失败的处理器，内置了默认逻辑，如果提供了GlobalRestFailureHandler，将使用自己提供的。
      mountGlobalRestFailureHandler(mainRouter);

      HttpServer httpServer = createHttpServer();
      httpServer.requestHandler(mainRouter);
      httpServer.connectionHandler(connection -> {
        DefaultHttpServerMetrics serverMetrics = (DefaultHttpServerMetrics) ((ConnectionBase) connection).metrics();
        DefaultServerEndpointMetric endpointMetric = serverMetrics.getEndpointMetric();
        long connectedCount = endpointMetric.getCurrentConnectionCount();
        int connectionLimit = DynamicPropertyFactory.getInstance()
            .getIntProperty("servicecomb.rest.server.connection-limit", Integer.MAX_VALUE).get();
        if (connectedCount > connectionLimit) {
          connection.close();
          endpointMetric.onRejectByConnectionLimit();
        }
      });
      List<HttpServerExceptionHandler> httpServerExceptionHandlers =
          SPIServiceUtils.getAllService(HttpServerExceptionHandler.class);
      httpServer.exceptionHandler(e -> {
        if (e instanceof ClosedChannelException) {
          // This is quite normal in between browser and ege, so do not print out.
          LOGGER.debug("Unexpected error in server.{}", ExceptionUtils.getExceptionMessageWithoutTrace(e));
        } else {
          LOGGER.error("Unexpected error in server.{}", ExceptionUtils.getExceptionMessageWithoutTrace(e));
        }
        // 扩展点
        httpServerExceptionHandlers.forEach(httpServerExceptionHandler -> {
          httpServerExceptionHandler.handle(e);
        });
      });
      // 开启端口监听
      startListen(httpServer, startFuture);
    } catch (Throwable e) {
      // vert.x got some states that not print error and execute call back in VertexUtils.blockDeploy, we add a log our self.
      LOGGER.error("", e);
      throw e;
    }
  }

  /**
   * 全局Rest失败的处理器，内置了默认逻辑，如果提供了GlobalRestFailureHandler，将使用自己提供的。
   * @param mainRouter
   */
  private void mountGlobalRestFailureHandler(Router mainRouter) {
    GlobalRestFailureHandler globalRestFailureHandler =
        SPIServiceUtils.getPriorityHighestService(GlobalRestFailureHandler.class);
    Handler<RoutingContext> failureHandler = null == globalRestFailureHandler ?
        ctx -> {
          if (ctx.response().closed() || ctx.response().ended()) {
            // response has been sent, do nothing
            LOGGER.error("get a failure with closed response", ctx.failure());
            return;
          }
          HttpServerResponse response = ctx.response();
          if (ctx.failure() instanceof InvocationException) {
            // ServiceComb defined exception
            InvocationException exception = (InvocationException) ctx.failure();
            response.setStatusCode(exception.getStatusCode());
            response.setStatusMessage(exception.getReasonPhrase());
            response.end(exception.getErrorData().toString());
            return;
          }

          LOGGER.error("unexpected failure happened", ctx.failure());
          try {
            // unknown exception
            CommonExceptionData unknownError = new CommonExceptionData("unknown error");
            ctx.response().setStatusCode(500).putHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON)
                .end(RestObjectMapperFactory.getRestObjectMapper().writeValueAsString(unknownError));
          } catch (Exception e) {
            LOGGER.error("failed to send error response!", e);
          }
        }
        : globalRestFailureHandler;

    mainRouter.route()
        // this handler does nothing, just ensure the failure handler can catch exception
        .handler(RoutingContext::next)
        .failureHandler(failureHandler);
  }

  /**
   * 处理日志
   * @param mainRouter
   */
  private void mountAccessLogHandler(Router mainRouter) {
    if (!AccessLogConfig.INSTANCE.isServerLogEnabled()) {
      return;
    }
    LOGGER.info("access log enabled, pattern = {}", AccessLogConfig.INSTANCE.getServerLogPattern());
    mainRouter.route().handler(context -> {
      ServerAccessLogEvent accessLogEvent = new ServerAccessLogEvent()
          .setRoutingContext(context)
          .setMilliStartTime(System.currentTimeMillis())
          .setLocalAddress(LocalHostAccessItem.getLocalAddress(context));
      context.response().endHandler(event ->
          EventManager.post(accessLogEvent.setMilliEndTime(System.currentTimeMillis())));
      context.next();
    });
  }

  /**
   * Support CORS
   */
  void mountCorsHandler(Router mainRouter) {
    if (!TransportConfig.isCorsEnabled()) {
      return;
    }

    CorsHandler corsHandler = getCorsHandler(TransportConfig.getCorsAllowedOrigin());
    // Access-Control-Allow-Credentials
    corsHandler.allowCredentials(TransportConfig.isCorsAllowCredentials());
    // Access-Control-Allow-Headers
    corsHandler.allowedHeaders(TransportConfig.getCorsAllowedHeaders());
    // Access-Control-Allow-Methods
    Set<String> allowedMethods = TransportConfig.getCorsAllowedMethods();
    for (String method : allowedMethods) {
      corsHandler.allowedMethod(HttpMethod.valueOf(method));
    }
    // Access-Control-Expose-Headers
    corsHandler.exposedHeaders(TransportConfig.getCorsExposedHeaders());
    // Access-Control-Max-Age
    int maxAge = TransportConfig.getCorsMaxAge();
    if (maxAge >= 0) {
      corsHandler.maxAgeSeconds(maxAge);
    }

    LOGGER.info("mount CorsHandler");
    mainRouter.route().handler(corsHandler);
  }

  private CorsHandler getCorsHandler(String corsAllowedOrigin) {
    return CorsHandler.create(corsAllowedOrigin);
  }

  private void initDispatcher(Router mainRouter) {
    // 获取所有的VertxHttpDispatcher
    List<VertxHttpDispatcher> dispatchers = SPIServiceUtils.loadSortedService(VertxHttpDispatcher.class);
    BeanUtils.addBeans(VertxHttpDispatcher.class, dispatchers);
    
    for (VertxHttpDispatcher dispatcher : dispatchers) {
      if (dispatcher.enabled()) {
        dispatcher.init(mainRouter);
      }
    }
  }

  @SuppressWarnings("deprecation")
  // TODO: vert.x 3.8.3 does not update startListen to promise, so we keep use deprecated API now. update in newer version.
  private void startListen(HttpServer server, Future<Void> startFuture) {
    server.listen(endpointObject.getPort(), endpointObject.getHostOrIp(), ar -> {
      if (ar.succeeded()) {
        LOGGER.info("rest listen success. address={}:{}",
            endpointObject.getHostOrIp(),
            ar.result().actualPort());
        startFuture.complete();
        return;
      }

      String msg = String.format("rest listen failed, address=%s:%d",
          endpointObject.getHostOrIp(),
          endpointObject.getPort());
      LOGGER.error(msg, ar.cause());
      startFuture.fail(ar.cause());
    });
  }

  private HttpServer createHttpServer() {
    HttpServerOptions serverOptions = createDefaultHttpServerOptions();
    return vertx.createHttpServer(serverOptions);
  }

  private HttpServerOptions createDefaultHttpServerOptions() {
    HttpServerOptions serverOptions = new HttpServerOptions();
    serverOptions.setUsePooledBuffers(true);
    serverOptions.setIdleTimeout(TransportConfig.getConnectionIdleTimeoutInSeconds());
    serverOptions.setCompressionSupported(TransportConfig.getCompressed());
    serverOptions.setMaxHeaderSize(TransportConfig.getMaxHeaderSize());
    serverOptions.setMaxInitialLineLength(TransportConfig.getMaxInitialLineLength());
    if (endpointObject.isHttp2Enabled()) {
      serverOptions.setUseAlpn(TransportConfig.getUseAlpn())
          .setInitialSettings(new Http2Settings().setMaxConcurrentStreams(TransportConfig.getMaxConcurrentStreams()));
    }
    if (endpointObject.isSslEnabled()) {
      SSLOptionFactory factory =
          SSLOptionFactory.createSSLOptionFactory(SSL_KEY, null);
      SSLOption sslOption;
      if (factory == null) {
        sslOption = SSLOption.buildFromYaml(SSL_KEY);
      } else {
        sslOption = factory.createSSLOption();
      }
      SSLCustom sslCustom = SSLCustom.createSSLCustom(sslOption.getSslCustomClass());
      VertxTLSBuilder.buildNetServerOptions(sslOption, sslCustom, serverOptions);
    }

    return serverOptions;
  }
}

VertxHttpDispatcher

分发接口，具体实现类由SPI加载（可自由扩展），由AbstractVerticle的start()加载

public interface VertxHttpDispatcher extends SPIOrder, SPIEnabled {
  @Override
  default boolean enabled() {
    return true;
  }
  void init(Router router);
}

VertxRestDispatcher

服务端拦截请求的最后一个Dispatcher，可根据此类自定义添加VertxHttpDispatcher

/**
 * 最后一个VertxHttpDispatcher，默认不拦截任何路径
 */
public class VertxRestDispatcher extends AbstractVertxHttpDispatcher {
  private static final Logger LOGGER = LoggerFactory.getLogger(VertxRestDispatcher.class);

  private static final String KEY_ORDER = "servicecomb.http.dispatcher.rest.order";

  private static final String KEY_ENABLED = "servicecomb.http.dispatcher.rest.enabled";

  private static final String KEY_PATTERN = "servicecomb.http.dispatcher.rest.pattern";

  private Transport transport;

  private MicroserviceMeta microserviceMeta;

  @Override
  public int getOrder() {
    return DynamicPropertyFactory.getInstance().getIntProperty(KEY_ORDER, Integer.MAX_VALUE).get();
  }

  @Override
  public boolean enabled() {
    return DynamicPropertyFactory.getInstance().getBooleanProperty(KEY_ENABLED, true).get();
  }

  @Override
  public void init(Router router) {
    // cookies handler are enabled by default start from 3.8.3
    // 获取匹配路径，默认为null,即不拦截任何路径
    String pattern = DynamicPropertyFactory.getInstance().getStringProperty(KEY_PATTERN, null).get();
    if (pattern == null) {
      router.route().handler(createBodyHandler());
      router.route().failureHandler(this::failureHandler).handler(this::onRequest);
    } else {
      router.routeWithRegex(pattern).handler(createBodyHandler());
      router.routeWithRegex(pattern).failureHandler(this::failureHandler).handler(this::onRequest);
    }
  }

  protected void failureHandler(RoutingContext context) {
    LOGGER.error("http server failed.", context.failure());

    AbstractRestInvocation restProducerInvocation = context.get(RestConst.REST_PRODUCER_INVOCATION);
    Throwable e = context.failure();
    if (e instanceof ErrorDataDecoderException) {
      Throwable cause = e.getCause();
      if (cause instanceof InvocationException) {
        e = cause;
      }
    }

    // only when unexpected exception happens, it will run into here.
    // the connection should be closed.
    handleFailureAndClose(context, restProducerInvocation, e);
  }

  /**
   * Try to find out the failure information and send it in response.
   */
  private void handleFailureAndClose(RoutingContext context, AbstractRestInvocation restProducerInvocation,
      Throwable e) {
    if (null != restProducerInvocation) {
      // if there is restProducerInvocation, let it send exception in response. The exception is allowed to be null.
      sendFailResponseByInvocation(context, restProducerInvocation, e);
      return;
    }

    if (null != e) {
      // if there exists exception, try to send this exception by RoutingContext
      sendExceptionByRoutingContext(context, e);
      return;
    }

    // if there is no exception, the response is determined by status code.
    sendFailureRespDeterminedByStatus(context);
  }

  /**
   * Try to determine response by status code, and send response.
   */
  private void sendFailureRespDeterminedByStatus(RoutingContext context) {
    Family statusFamily = Family.familyOf(context.statusCode());
    if (Family.CLIENT_ERROR.equals(statusFamily) || Family.SERVER_ERROR.equals(statusFamily) || Family.OTHER
        .equals(statusFamily)) {
      context.response().putHeader(HttpHeaders.CONTENT_TYPE, MediaType.WILDCARD)
          .setStatusCode(context.statusCode()).end();
    } else {
      // it seems the status code is not set properly
      context.response().putHeader(HttpHeaders.CONTENT_TYPE, MediaType.WILDCARD)
          .setStatusCode(Status.INTERNAL_SERVER_ERROR.getStatusCode())
          .setStatusMessage(Status.INTERNAL_SERVER_ERROR.getReasonPhrase())
          .end(wrapResponseBody(Status.INTERNAL_SERVER_ERROR.getReasonPhrase()));
    }
    context.response().close();
  }

  /**
   * Use routingContext to send failure information in throwable.
   */
  private void sendExceptionByRoutingContext(RoutingContext context, Throwable e) {
    if (e instanceof InvocationException) {
      InvocationException invocationException = (InvocationException) e;
      context.response().putHeader(HttpHeaders.CONTENT_TYPE, MediaType.WILDCARD)
          .setStatusCode(invocationException.getStatusCode()).setStatusMessage(invocationException.getReasonPhrase())
          .end(wrapResponseBody(invocationException.getReasonPhrase()));
    } else {
      context.response().putHeader(HttpHeaders.CONTENT_TYPE, MediaType.WILDCARD)
          .setStatusCode(Status.INTERNAL_SERVER_ERROR.getStatusCode()).end(wrapResponseBody(e.getMessage()));
    }
    context.response().close();
  }

  /**
   * Consumer will treat the response body as json by default, so it's necessary to wrap response body as Json string
   * to avoid deserialization error.
   *
   * @param message response body
   * @return response body wrapped as Json string
   */
  String wrapResponseBody(String message) {
    if (isValidJson(message)) {
      return message;
    }

    JsonObject jsonObject = new JsonObject();
    jsonObject.put("message", message);

    return jsonObject.toString();
  }

  /**
   * Check if the message is a valid Json string.
   * @param message the message to be checked.
   * @return true if message is a valid Json string, otherwise false.
   */
  private boolean isValidJson(String message) {
    try {
      new JsonObject(message);
    } catch (Exception ignored) {
      return false;
    }
    return true;
  }

  /**
   * Use restProducerInvocation to send failure message. The throwable is allowed to be null.
   */
  private void sendFailResponseByInvocation(RoutingContext context, AbstractRestInvocation restProducerInvocation,
      Throwable e) {
    restProducerInvocation.sendFailResponse(e);
    context.response().close();
  }

  /**
   * 拦截成功请求的处理方法，首先判断servicecomb.filter-chains.enabled是否为true
   * @param context
   */
  protected void onRequest(RoutingContext context) {
    if (transport == null) {
      transport = SCBEngine.getInstance().getTransportManager().findTransport(Const.RESTFUL);
      microserviceMeta = SCBEngine.getInstance().getProducerMicroserviceMeta();
    }
    HttpServletRequestEx requestEx = new VertxServerRequestToHttpServletRequest(context);
    HttpServletResponseEx responseEx = new VertxServerResponseToHttpServletResponse(context.response());

    // 如果为true,由CompletableFuture执行，不执行HttpServerFilter的拦截方法。
    // 使用CompletableFuture操作，forkjoin线程模型需自行控制并发安全
    if (SCBEngine.getInstance().isFilterChainEnabled()) {
      InvocationCreator creator = new RestVertxProducerInvocationCreator(context,
          microserviceMeta, transport.getEndpoint(),
          requestEx, responseEx);
      new RestProducerInvocationFlow(creator, requestEx, responseEx)
          .run();
      return;
    }
    // 正常执行
    VertxRestInvocation vertxRestInvocation = new VertxRestInvocation();
    context.put(RestConst.REST_PRODUCER_INVOCATION, vertxRestInvocation);
    // 最终在AbstractRestInvocation.scheduleInvocation()方法内执行主要逻辑
    vertxRestInvocation.invoke(transport, requestEx, responseEx, httpServerFilters);
  }
}

RestClientInvocation

基于http协议的Rest客户端核心类，执行客户端请求如CseTemplate,其中包括执行HttpClientFilter的回调方法。

public class RestClientInvocation {
  private static final Logger LOGGER = LoggerFactory.getLogger(RestClientInvocation.class);

  private static final String[] INTERNAL_HEADERS = new String[] {
      org.apache.servicecomb.core.Const.CSE_CONTEXT,
      org.apache.servicecomb.core.Const.TARGET_MICROSERVICE
  };

  private HttpClientWithContext httpClientWithContext;

  private Invocation invocation;

  private RestOperationMeta restOperationMeta;

  private AsyncResponse asyncResp;

  private List<HttpClientFilter> httpClientFilters;

  private HttpClientRequest clientRequest;

  private HttpClientResponse clientResponse;

  private Handler<Throwable> throwableHandler = e -> fail((ConnectionBase) clientRequest.connection(), e);

  private boolean alreadyFailed = false;

  public RestClientInvocation(HttpClientWithContext httpClientWithContext, List<HttpClientFilter> httpClientFilters) {
    this.httpClientWithContext = httpClientWithContext;
    this.httpClientFilters = httpClientFilters;
  }

  public void invoke(Invocation invocation, AsyncResponse asyncResp) throws Exception {
    this.invocation = invocation;
    this.asyncResp = asyncResp;

    OperationMeta operationMeta = invocation.getOperationMeta();
    restOperationMeta = operationMeta.getExtData(RestConst.SWAGGER_REST_OPERATION);

    String path = this.createRequestPath(restOperationMeta);
    IpPort ipPort = (IpPort) invocation.getEndpoint().getAddress();

    createRequest(ipPort, path);
    clientRequest.putHeader(org.apache.servicecomb.core.Const.TARGET_MICROSERVICE, invocation.getMicroserviceName());
    RestClientRequestImpl restClientRequest =
        new RestClientRequestImpl(clientRequest, httpClientWithContext.context(), asyncResp, throwableHandler);
    invocation.getHandlerContext().put(RestConst.INVOCATION_HANDLER_REQUESTCLIENT, restClientRequest);

    Buffer requestBodyBuffer = restClientRequest.getBodyBuffer();
    HttpServletRequestEx requestEx = new VertxClientRequestToHttpServletRequest(clientRequest, requestBodyBuffer);
    invocation.getInvocationStageTrace().startClientFiltersRequest();
    for (HttpClientFilter filter : httpClientFilters) {
      if (filter.enabled()) {
        filter.beforeSendRequest(invocation, requestEx);
      }
    }

    clientRequest.exceptionHandler(e -> {
      invocation.getTraceIdLogger()
          .error(LOGGER, "Failed to send request, alreadyFailed:{}, local:{}, remote:{}, message={}.",
              alreadyFailed, getLocalAddress(), ipPort.getSocketAddress(),
              ExceptionUtils.getExceptionMessageWithoutTrace(e));
      throwableHandler.handle(e);
    });

    // 从业务线程转移到网络线程中去发送
    invocation.getInvocationStageTrace().startSend();
    httpClientWithContext.runOnContext(httpClient -> {
      clientRequest.setTimeout(operationMeta.getConfig().getMsRequestTimeout());
      processServiceCombHeaders(invocation, operationMeta);
      try {
        restClientRequest.end();
      } catch (Throwable e) {
        invocation.getTraceIdLogger().error(LOGGER,
            "send http request failed, alreadyFailed:{}, local:{}, remote: {}, message={}.",
            alreadyFailed,
            getLocalAddress(), ipPort
            , ExceptionUtils.getExceptionMessageWithoutTrace(e));
        fail((ConnectionBase) clientRequest.connection(), e);
      }
    });
  }

  /**
   * If this is a 3rd party invocation, ServiceComb related headers should be removed by default to hide inner
   * implementation. Otherwise, the InvocationContext will be set into the request headers.
   *
   * @see OperationConfig#isClientRequestHeaderFilterEnabled()
   * @param invocation  invocation determines whether this is an invocation to 3rd party services
   * @param operationMeta operationMeta determines whether to remove certain headers and which headers should be removed
   */
  private void processServiceCombHeaders(Invocation invocation, OperationMeta operationMeta) {
    if (invocation.isThirdPartyInvocation() && operationMeta.getConfig().isClientRequestHeaderFilterEnabled()) {
      for (String internalHeaderName : INTERNAL_HEADERS) {
        clientRequest.headers().remove(internalHeaderName);
      }
      return;
    }
    this.setCseContext();
  }

  private String getLocalAddress() {
    HttpConnection connection = clientRequest.connection();
    if (connection == null) {
      return "not connected";
    }
    SocketAddress socketAddress = connection.localAddress();
    return socketAddress != null ? socketAddress.toString() : "not connected";
  }

  private HttpMethod getMethod() {
    OperationMeta operationMeta = invocation.getOperationMeta();
    RestOperationMeta swaggerRestOperation = operationMeta.getExtData(RestConst.SWAGGER_REST_OPERATION);
    String method = swaggerRestOperation.getHttpMethod();
    return HttpMethod.valueOf(method);
  }

  @SuppressWarnings("deprecation")
  void createRequest(IpPort ipPort, String path) {
    URIEndpointObject endpoint = (URIEndpointObject) invocation.getEndpoint().getAddress();
    RequestOptions requestOptions = new RequestOptions();
    requestOptions.setHost(ipPort.getHostOrIp())
        .setPort(ipPort.getPort())
        .setSsl(endpoint.isSslEnabled())
        .setURI(path);

    HttpMethod method = getMethod();
    invocation.getTraceIdLogger()
        .debug(LOGGER, "Sending request by rest, method={}, qualifiedName={}, path={}, endpoint={}.",
            method,
            invocation.getMicroserviceQualifiedName(),
            path,
            invocation.getEndpoint().getEndpoint());
    clientRequest = httpClientWithContext.getHttpClient().request(method, requestOptions, this::handleResponse);
  }

  protected void handleResponse(HttpClientResponse httpClientResponse) {
    this.clientResponse = httpClientResponse;

    if (HttpStatus.isSuccess(clientResponse.statusCode()) && restOperationMeta.isDownloadFile()) {
      ReadStreamPart part = new ReadStreamPart(httpClientWithContext.context(), httpClientResponse);
      invocation.getHandlerContext().put(RestConst.READ_STREAM_PART, part);
      processResponseBody(null);
      return;
    }

    httpClientResponse.exceptionHandler(e -> {
      invocation.getTraceIdLogger().error(LOGGER, "Failed to receive response, local:{}, remote:{}, message={}.",
          getLocalAddress(), httpClientResponse.netSocket().remoteAddress(),
          ExceptionUtils.getExceptionMessageWithoutTrace(e));
      fail((ConnectionBase) clientRequest.connection(), e);
    });

    clientResponse.bodyHandler(this::processResponseBody);
  }

  /**
   * after this method, connection will be recycled to connection pool
   * @param responseBuf response body buffer, when download, responseBuf is null, because download data by ReadStreamPart
   */
  protected void processResponseBody(Buffer responseBuf) {
    DefaultHttpSocketMetric httpSocketMetric = (DefaultHttpSocketMetric) ((ConnectionBase) clientRequest.connection())
        .metric();
    invocation.getInvocationStageTrace().finishGetConnection(httpSocketMetric.getRequestBeginTime());
    invocation.getInvocationStageTrace().finishWriteToBuffer(httpSocketMetric.getRequestEndTime());
    invocation.getInvocationStageTrace().finishReceiveResponse();

    invocation.getResponseExecutor().execute(() -> {
      try {
        invocation.getInvocationStageTrace().startClientFiltersResponse();
        HttpServletResponseEx responseEx =
            new VertxClientResponseToHttpServletResponse(clientResponse, responseBuf);
        for (HttpClientFilter filter : httpClientFilters) {
          if (filter.enabled()) {
            Response response = filter.afterReceiveResponse(invocation, responseEx);
            if (response != null) {
              complete(response);
              return;
            }
          }
        }
      } catch (Throwable e) {
        // already collection time from httpSocketMetric
        // and connection maybe already belongs to other invocation in this time
        // so set connection to null
        fail(null, e);
      }
    });
  }

  protected void complete(Response response) {
    invocation.getInvocationStageTrace().finishClientFiltersResponse();
    asyncResp.complete(response);
  }

  protected void fail(ConnectionBase connection, Throwable e) {
    if (alreadyFailed) {
      return;
    }

    alreadyFailed = true;

    InvocationStageTrace stageTrace = invocation.getInvocationStageTrace();
    // connection maybe null when exception happens such as ssl handshake failure
    if (connection != null) {
      DefaultHttpSocketMetric httpSocketMetric = (DefaultHttpSocketMetric) connection.metric();
      stageTrace.finishGetConnection(httpSocketMetric.getRequestBeginTime());
      stageTrace.finishWriteToBuffer(httpSocketMetric.getRequestEndTime());
    }

    // even failed and did not received response, still set time for it
    // that will help to know the real timeout time
    if (stageTrace.getFinishReceiveResponse() == 0) {
      stageTrace.finishReceiveResponse();
    }
    if (stageTrace.getStartClientFiltersResponse() == 0) {
      stageTrace.startClientFiltersResponse();
    }

    stageTrace.finishClientFiltersResponse();

    try {
      if (e instanceof TimeoutException) {
        // give an accurate cause for timeout exception
        //   The timeout period of 30000ms has been exceeded while executing GET /xxx for server 1.1.1.1:8080
        // should not copy the message to invocationException to avoid leak server ip address
        LOGGER.info("Request timeout, Details: {}.", e.getMessage());
        asyncResp.consumerFail(new InvocationException(Status.REQUEST_TIMEOUT,
            new CommonExceptionData("Request Timeout.")));
        return;
      }
      asyncResp.fail(invocation.getInvocationType(), e);
    } catch (Throwable e1) {
      invocation.getTraceIdLogger().error(LOGGER, "failed to invoke asyncResp, message={}"
          , ExceptionUtils.getExceptionMessageWithoutTrace(e));
    }
  }

  protected void setCseContext() {
    try {
      String cseContext = JsonUtils.writeValueAsString(invocation.getContext());
      clientRequest.putHeader(org.apache.servicecomb.core.Const.CSE_CONTEXT, cseContext);
    } catch (Throwable e) {
      invocation.getTraceIdLogger().error(LOGGER, "Failed to encode and set cseContext, message={}."
          , ExceptionUtils.getExceptionMessageWithoutTrace(e));
    }
  }

  protected String createRequestPath(RestOperationMeta swaggerRestOperation) throws Exception {
    URIEndpointObject address = (URIEndpointObject) invocation.getEndpoint().getAddress();
    String urlPrefix = address.getFirst(DefinitionConst.URL_PREFIX);

    String path = (String) invocation.getHandlerContext().get(RestConst.REST_CLIENT_REQUEST_PATH);
    if (path == null) {
      path = swaggerRestOperation.getPathBuilder().createRequestPath(invocation.getSwaggerArguments());
    }

    if (StringUtils.isEmpty(urlPrefix) || path.startsWith(urlPrefix)) {
      return path;
    }

    return urlPrefix + path;
  }
}

HighwayClient

基于TCP的客户端核心类，无可用扩展点。

public class HighwayClient {
  private static final Logger LOGGER = LoggerFactory.getLogger(HighwayClient.class);

  private static final String SSL_KEY = "highway.consumer";

  private ClientPoolManager<HighwayClientConnectionPool> clientMgr;

  public void init(Vertx vertx) throws Exception {
    TcpClientConfig normalConfig = createTcpClientConfig();
    normalConfig.setSsl(false);

    TcpClientConfig sslConfig = createTcpClientConfig();
    sslConfig.setSsl(true);

    clientMgr = new ClientPoolManager<>(vertx, new HighwayClientPoolFactory(normalConfig, sslConfig));

    DeploymentOptions deployOptions = VertxUtils.createClientDeployOptions(clientMgr,
        HighwayConfig.getClientThreadCount());
    VertxUtils.blockDeploy(vertx, ClientVerticle.class, deployOptions);
  }

  private TcpClientConfig createTcpClientConfig() {
    TcpClientConfig tcpClientConfig = new TcpClientConfig();
    // global request timeout to be login timeout
    tcpClientConfig.setMsLoginTimeout(DynamicPropertyFactory.getInstance()
        .getLongProperty("servicecomb.request.timeout", TcpClientConfig.DEFAULT_LOGIN_TIMEOUT).get());

    SSLOptionFactory factory =
        SSLOptionFactory.createSSLOptionFactory(SSL_KEY, null);
    SSLOption sslOption;
    if (factory == null) {
      sslOption = SSLOption.buildFromYaml(SSL_KEY);
    } else {
      sslOption = factory.createSSLOption();
    }
    SSLCustom sslCustom = SSLCustom.createSSLCustom(sslOption.getSslCustomClass());
    VertxTLSBuilder.buildClientOptionsBase(sslOption, sslCustom, tcpClientConfig);

    return tcpClientConfig;
  }

  public void send(Invocation invocation, AsyncResponse asyncResp) throws Exception {
    invocation.getInvocationStageTrace().startClientFiltersRequest();
    invocation.getInvocationStageTrace().startSend();

    HighwayClientConnection tcpClient = findClientPool(invocation);

    OperationProtobuf operationProtobuf = ProtobufManager.getOrCreateOperation(invocation);
    HighwayClientPackage clientPackage = createClientPackage(invocation, operationProtobuf);

    tcpClient.send(clientPackage, ar -> {
      invocation.getInvocationStageTrace().finishWriteToBuffer(clientPackage.getFinishWriteToBuffer());
      invocation.getInvocationStageTrace().finishReceiveResponse();
      // 此时是在网络线程中，转换线程
      invocation.getResponseExecutor().execute(() -> {
        invocation.getInvocationStageTrace().startClientFiltersResponse();
        if (ar.failed()) {
          // 只会是本地异常
          invocation.getInvocationStageTrace().finishClientFiltersResponse();
          if (ar.cause() instanceof TimeoutException) {
            // give an accurate cause for timeout exception
            //   The timeout period of 30000ms has been exceeded while executing GET /xxx for server 1.1.1.1:8080
            // should not copy the message to invocationException to avoid leak server ip address
            LOGGER.info("Request timeout, Details: {}.", ar.cause().getMessage());

            asyncResp.consumerFail(new InvocationException(Status.REQUEST_TIMEOUT,
                new CommonExceptionData("Request Timeout.")));
            return;
          }
          asyncResp.consumerFail(ar.cause());
          return;
        }

        // 处理应答
        try {
          Response response =
              HighwayCodec.decodeResponse(invocation,
                  operationProtobuf,
                  ar.result());
          invocation.getInvocationStageTrace().finishClientFiltersResponse();
          asyncResp.complete(response);
        } catch (Throwable e) {
          invocation.getInvocationStageTrace().finishClientFiltersResponse();
          asyncResp.consumerFail(e);
        }
      });
    });
  }

  public HighwayClientPackage createClientPackage(Invocation invocation, OperationProtobuf operationProtobuf) {
    long msRequestTimeout = invocation.getOperationMeta().getConfig().getMsRequestTimeout();
    return new HighwayClientPackage(invocation, operationProtobuf, msRequestTimeout);
  }

  public HighwayClientConnection findClientPool(Invocation invocation) {
    HighwayClientConnection tcpClient = clientMgr.findClientPool(invocation.isSync())
        .findOrCreateClient(invocation.getEndpoint().getEndpoint());

    invocation.getInvocationStageTrace().finishGetConnection(System.nanoTime());

    return tcpClient;
  }
}

HttpClientFilter

字面意思客户端过滤器，提供了排序和是否启用功能，前置和后置回调方法。

由SPI获取相关类（可自由扩展），即：META-INF.services下面的文件org.apache.servicecomb.common.rest.filter.HttpClientFilter

在RestTransportClient.send()中的RestClientInvocation.invoke()完成HttpClientFilters的遍历执行

invocation对象可自由处理

public interface HttpClientFilter {
  default boolean enabled() {
    return true;
  }

  int getOrder();

  void beforeSendRequest(Invocation invocation, HttpServletRequestEx requestEx);

  // if finished, then return a none null response
  // if return a null response, then sdk will call next filter.afterReceive
  Response afterReceiveResponse(Invocation invocation, HttpServletResponseEx responseEx);
}

HttpServerFilter

字面意思，服务端过滤器，提供了排序和是否启用功能，前置和后置回调方法。

public interface HttpServerFilter {
  int getOrder();

  default boolean enabled() {
    return true;
  }

  default boolean needCacheRequest(OperationMeta operationMeta) {
    return false;
  }

  /**
   * @return if finished, then return a none null response<br>
   * if return a null response, then sdk will call next filter.afterReceiveRequest
   */
  Response afterReceiveRequest(Invocation invocation, HttpServletRequestEx requestEx);

  /**
   * @param invocation maybe null
   */
  default CompletableFuture<Void> beforeSendResponseAsync(Invocation invocation, HttpServletResponseEx responseEx) {
    CompletableFuture<Void> future = new CompletableFuture<>();
    try {
      beforeSendResponse(invocation, responseEx);
      future.complete(null);
    } catch (Throwable e) {
      future.completeExceptionally(e);
    }
    return future;
  }

  /**
   * @param invocation maybe null
   */
  default void beforeSendResponse(Invocation invocation, HttpServletResponseEx responseEx) {

  }
}

HttpServerExceptionHandler

AbstractRestInvocation

包含了服务端执行请求的主要逻辑

public abstract class AbstractRestInvocation {
  private static final Logger LOGGER = LoggerFactory.getLogger(AbstractRestInvocation.class);

  public static final String UNKNOWN_OPERATION_ID = "UNKNOWN_OPERATION";

  protected long start;

  protected RestOperationMeta restOperationMeta;

  protected Invocation invocation;

  protected HttpServletRequestEx requestEx;

  protected HttpServletResponseEx responseEx;

  protected ProduceProcessor produceProcessor;

  protected List<HttpServerFilter> httpServerFilters = Collections.emptyList();

  public AbstractRestInvocation() {
    this.start = System.nanoTime();
  }

  public void setHttpServerFilters(List<HttpServerFilter> httpServerFilters) {
    this.httpServerFilters = httpServerFilters;
  }

  protected void findRestOperation(MicroserviceMeta microserviceMeta) {
    ServicePathManager servicePathManager = ServicePathManager.getServicePathManager(microserviceMeta);
    if (servicePathManager == null) {
      LOGGER.error("No schema defined for {}:{}.", microserviceMeta.getAppId(), microserviceMeta.getMicroserviceName());
      throw new InvocationException(Status.NOT_FOUND, Status.NOT_FOUND.getReasonPhrase());
    }

    OperationLocator locator = locateOperation(servicePathManager);
    requestEx.setAttribute(RestConst.PATH_PARAMETERS, locator.getPathVarMap());
    this.restOperationMeta = locator.getOperation();
  }

  protected void initProduceProcessor() {
    produceProcessor = restOperationMeta.ensureFindProduceProcessor(requestEx);
    if (produceProcessor == null) {
      LOGGER.error("Accept {} is not supported, operation={}.", requestEx.getHeader(HttpHeaders.ACCEPT),
          restOperationMeta.getOperationMeta().getMicroserviceQualifiedName());
      String msg = String.format("Accept %s is not supported", requestEx.getHeader(HttpHeaders.ACCEPT));
      throw new InvocationException(Status.NOT_ACCEPTABLE, msg);
    }
  }

  protected void setContext() throws Exception {
    String strCseContext = requestEx.getHeader(Const.CSE_CONTEXT);
    if (StringUtils.isEmpty(strCseContext)) {
      return;
    }

    @SuppressWarnings("unchecked")
    Map<String, String> cseContext =
        JsonUtils.readValue(strCseContext.getBytes(StandardCharsets.UTF_8), Map.class);
    invocation.mergeContext(cseContext);
  }

  public String getContext(String key) {
    if (null == invocation || null == invocation.getContext()) {
      return null;
    }

    return invocation.getContext(key);
  }

  protected void scheduleInvocation() {
    try {
      createInvocation();
    } catch (Throwable e) {
      sendFailResponse(e);
      return;
    }

    invocation.onStart(requestEx, start);
    invocation.getInvocationStageTrace().startSchedule();
    OperationMeta operationMeta = restOperationMeta.getOperationMeta();

    try {
      this.setContext();
    } catch (Exception e) {
      LOGGER.error("failed to set invocation context", e);
      sendFailResponse(e);
      return;
    }
    // 找到ProviderQpsFlowControlHandler执行
    // 如果找到就执行HttpServerFilter.beforeSendResponseAsync()
    Holder<Boolean> qpsFlowControlReject = checkQpsFlowControl(operationMeta);
    if (qpsFlowControlReject.value) {
      return;
    }

    try {
      operationMeta.getExecutor().execute(() -> {
        synchronized (this.requestEx) {
          try {
            if (isInQueueTimeout()) {
              throw new InvocationException(Status.INTERNAL_SERVER_ERROR, "Timeout when processing the request.");
            }
            if (requestEx.getAttribute(RestConst.REST_REQUEST) != requestEx) {
              // already timeout
              // in this time, request maybe recycled and reused by web container, do not use requestEx
              LOGGER.error("Rest request already timeout, abandon execute, method {}, operation {}.",
                  operationMeta.getHttpMethod(),
                  operationMeta.getMicroserviceQualifiedName());
              return;
            }

            runOnExecutor();
          } catch (Throwable e) {
            LOGGER.error("rest server onRequest error", e);
            sendFailResponse(e);
          }
        }
      });
    } catch (Throwable e) {
      LOGGER.error("failed to schedule invocation, message={}, executor={}.", e.getMessage(), e.getClass().getName());
      sendFailResponse(e);
    }
  }

  private Holder<Boolean> checkQpsFlowControl(OperationMeta operationMeta) {
    Holder<Boolean> qpsFlowControlReject = new Holder<>(false);
    @SuppressWarnings("deprecation")
    Handler providerQpsFlowControlHandler = operationMeta.getProviderQpsFlowControlHandler();
    if (null != providerQpsFlowControlHandler) {
      try {
        providerQpsFlowControlHandler.handle(invocation, response -> {
          qpsFlowControlReject.value = true;
          produceProcessor = ProduceProcessorManager.INSTANCE.findDefaultJsonProcessor();
          sendResponse(response);
        });
      } catch (Throwable e) {
        LOGGER.error("failed to execute ProviderQpsFlowControlHandler", e);
        qpsFlowControlReject.value = true;
        sendFailResponse(e);
      }
    }
    return qpsFlowControlReject;
  }

  private boolean isInQueueTimeout() {
    return System.nanoTime() - invocation.getInvocationStageTrace().getStart() >
        invocation.getOperationMeta().getConfig().getNanoRestRequestWaitInPoolTimeout();
  }

  protected void runOnExecutor() {
    invocation.onExecuteStart();

    invoke();
  }

  protected abstract OperationLocator locateOperation(ServicePathManager servicePathManager);

  // create a invocation without args setted
  protected abstract void createInvocation();

  public void invoke() {
    try {
      // 执行filter.afterReceiveRequest()
      Response response = prepareInvoke();
      if (response != null) {
        sendResponseQuietly(response);
        return;
      }

      doInvoke();
    } catch (Throwable e) {
      LOGGER.error("unknown rest exception.", e);
      sendFailResponse(e);
    }
  }

  protected Response prepareInvoke() throws Throwable {
    this.initProduceProcessor();
    invocation.getHandlerContext().put(RestConst.REST_REQUEST, requestEx);

    invocation.getInvocationStageTrace().startServerFiltersRequest();
    for (HttpServerFilter filter : httpServerFilters) {
      if (filter.enabled()) {
        Response response = filter.afterReceiveRequest(invocation, requestEx);
        if (response != null) {
          return response;
        }
      }
    }

    return null;
  }

  protected void doInvoke() throws Throwable {
    invocation.getInvocationStageTrace().startHandlersRequest();
    invocation.next(resp -> sendResponseQuietly(resp));
  }

  public void sendFailResponse(Throwable throwable) {
    if (produceProcessor == null) {
      produceProcessor = ProduceProcessorManager.INSTANCE.findDefaultProcessor();
    }

    Response response = Response.createProducerFail(throwable);
    sendResponseQuietly(response);
  }

  protected void sendResponseQuietly(Response response) {
    if (invocation != null) {
      invocation.getInvocationStageTrace().finishHandlersResponse();
    }
    try {
      sendResponse(response);
    } catch (Throwable e) {
      LOGGER.error("Failed to send rest response, operation:{}, request uri:{}",
          getMicroserviceQualifiedName(), requestEx.getRequestURI(), e);
    }
  }

  @SuppressWarnings("deprecation")
  protected void sendResponse(Response response) {
    RestServerCodecFilter.copyHeadersToHttpResponse(response.getHeaders(), responseEx);

    responseEx.setStatus(response.getStatusCode(), response.getReasonPhrase());
    responseEx.setAttribute(RestConst.INVOCATION_HANDLER_RESPONSE, response);
    responseEx.setAttribute(RestConst.INVOCATION_HANDLER_PROCESSOR, produceProcessor);
    // 完成HttpServerFilter.beforeSendResponseAsync()
    executeHttpServerFilters(response);
  }

  protected void executeHttpServerFilters(Response response) {
    HttpServerFilterBeforeSendResponseExecutor exec =
        new HttpServerFilterBeforeSendResponseExecutor(httpServerFilters, invocation, responseEx);
    CompletableFuture<Void> future = exec.run();
    future.whenComplete((v, e) -> {
      if (invocation != null) {
        invocation.getInvocationStageTrace().finishServerFiltersResponse();
      }

      onExecuteHttpServerFiltersFinish(response, e);
    });
  }

  protected void onExecuteHttpServerFiltersFinish(Response response, Throwable e) {
    if (e != null) {
      LOGGER.error("Failed to execute HttpServerFilters, operation:{}, request uri:{}",
          getMicroserviceQualifiedName(), requestEx.getRequestURI(), e);
    }

    try {
      responseEx.flushBuffer();
    } catch (Throwable flushException) {
      LOGGER.error("Failed to flush rest response, operation:{}, request uri:{}",
          getMicroserviceQualifiedName(), requestEx.getRequestURI(), flushException);
    }

    try {
      requestEx.getAsyncContext().complete();
    } catch (Throwable completeException) {
      LOGGER.error("Failed to complete async rest response, operation:{}, request uri:{}",
          getMicroserviceQualifiedName(), requestEx.getRequestURI(), completeException);
    }

    // if failed to locate path, then will not create invocation
    // TODO: statistics this case
    if (invocation != null) {
      invocation.onFinish(response);
    }
  }

  private String getMicroserviceQualifiedName() {
    return null == invocation ? UNKNOWN_OPERATION_ID : invocation.getMicroserviceQualifiedName();
  }
}

客户端

RestTemplateWrapper

继承了RestTemplate，内部重写了大部分RestTemplate的方法，将其在RestTemplateWrapper中完成筛选AcceptableRestTemplate（是一个抽象类，添加了对url的过滤判断），最终在AcceptableRestTemplate的实现类中完成了最终的网络请求。

/**
 * 用于同时支持cse调用和非cse调用
 */
// TODO: 2017/7/1 what we want to expose is RestOperations instead, since some RestTemplate methods are not to be called by users
class RestTemplateWrapper extends RestTemplate {
  private final List<AcceptableRestTemplate> acceptableRestTemplates = new ArrayList<>();

  final RestTemplate defaultRestTemplate = new RestTemplate();

  RestTemplateWrapper() {
     // 添加cse默认的CseRestTemplate
    acceptableRestTemplates.add(new CseRestTemplate());
  }

  void addAcceptableRestTemplate(int index, AcceptableRestTemplate restTemplate) {
    acceptableRestTemplates.add(index, restTemplate);
  }

  RestTemplate getRestTemplate(String url) {
    for (AcceptableRestTemplate template : acceptableRestTemplates) {
      if (template.isAcceptable(url)) {
        return template;
      }
    }
    return defaultRestTemplate;
  }

  RestTemplate getRestTemplate(URI uri) {
    for (AcceptableRestTemplate template : acceptableRestTemplates) {
      if (template.isAcceptable(uri)) {
        return template;
      }
    }
    return defaultRestTemplate;
  }

  @Override
  public <T> T getForObject(String url, Class<T> responseType, Object... urlVariables) throws RestClientException {
    return getRestTemplate(url).getForObject(url, responseType, urlVariables);
  }
  // 此处省略对个重写方法
  // .....
}

public abstract class AcceptableRestTemplate extends RestTemplate {
  public abstract boolean isAcceptable(String uri);

  public abstract boolean isAcceptable(URI uri);
}

CseRestTemplate

只有cse://或servicecomb://开头的url才会用到CseRestTemplate，如果正常的http://会直接使用原生的RestTemplate进行发送，是不会经过ServiceComb内部流程处理的。Invocation中的处理链是由用户在配置文件中自定义的，可随意扩展，加载之后会在消费者端的处理链尾端增加一个TransportClientHandler，提供端的处理链尾端会增加一个ProducerOperationHandler。

public class CseRestTemplate extends AcceptableRestTemplate {
  /**
   * 完成了消息转换器的、请求工厂、和handler的添加
   */
  public CseRestTemplate() {
    setMessageConverters(Arrays.asList(new CseHttpMessageConverter()));
    setRequestFactory(new CseClientHttpRequestFactory());
    setUriTemplateHandler(new CseUriTemplateHandler());
  }

  // GET

  @Override
  @Nullable
  public <T> T getForObject(String url, Class<T> responseType, Object... uriVariables) throws RestClientException {
    RequestCallback requestCallback = acceptHeaderRequestCallback(responseType);
    CseHttpMessageConverterExtractor<T> responseExtractor =
        new CseHttpMessageConverterExtractor<>();
    return execute(url, HttpMethod.GET, requestCallback, responseExtractor, uriVariables);
  }
  // 此处省略对个重写方法
  // .....

  /**
   * 判断是否cse://或servicecomb://开头
   * @param uri
   * @return
   */
  @Override
  public boolean isAcceptable(String uri) {
    return uri.startsWith(RestConst.URI_PREFIX) || uri.startsWith(RestConst.URI_PREFIX_NEW);
  }

  @Override
  public boolean isAcceptable(URI uri) {
    return RestConst.SCHEME.equals(uri.getScheme()) || RestConst.SCHEME_NEW.equals(uri.getScheme());
  }
}

然后会调用InvokerUtils.innerSyncInvoke方法进行请求处理，接着创建一个Executor，这里的SyncResponseExecutor会直接使用业务线程进行请求处理。然后就是依次调用处理链中的每一个handler进行处理，比如LoadbalanceHandler会进行负载均衡选出请求目标地址等，该请求最终会通过TransportClientHandler选择对应的Transport发送到网络中去。

public static Response innerSyncInvoke(Invocation invocation) {
  try {
    if (isInEventLoop()) {
      throw new IllegalStateException("Can not execute sync logic in event loop. ");
    }
    invocation.onStart(null, System.nanoTime());
    // 创建线程池
    SyncResponseExecutor respExecutor = new SyncResponseExecutor();
    invocation.setResponseExecutor(respExecutor);

    invocation.getInvocationStageTrace().startHandlersRequest();
    invocation.next(respExecutor::setResponse);

    Response response = respExecutor.waitResponse();
    invocation.getInvocationStageTrace().finishHandlersResponse();
    invocation.onFinish(response);
    return response;
  } catch (Throwable e) {
    String msg =
        String.format("invoke failed, %s", invocation.getOperationMeta().getMicroserviceQualifiedName());
    LOGGER.error(msg, e);
    LOGGER.error("invocation type: {}, handler chain: {}", invocation.getInvocationType(),
        invocation.getHandlerChain());

    Response response = Response.createConsumerFail(e);
    invocation.onFinish(response);
    return response;
  }
}

  private void send(Invocation invocation, AsyncResponse asyncResp, LoadBalancer chosenLB) throws Exception {
    long time = System.currentTimeMillis();
    // 挑选出一个可用服务，内部包括挑选规则，默认采用轮训，从highway和rest所有服务中获取
    ServiceCombServer server = chosenLB.chooseServer(invocation);
    if (null == server) {
      asyncResp.consumerFail(new InvocationException(Status.INTERNAL_SERVER_ERROR, "No available address found."));
      return;
    }
    chosenLB.getLoadBalancerStats().incrementNumRequests(server);
    // 将endpoint赋值给invocation，将由TransportClientHandler完成请求的发送
    invocation.setEndpoint(server.getEndpoint());
    // 传递给下一个Handler
    invocation.next(resp -> {
      // this stats is for WeightedResponseTimeRule
      chosenLB.getLoadBalancerStats().noteResponseTime(server, (System.currentTimeMillis() - time));
      if (isFailedResponse(resp)) {
        chosenLB.getLoadBalancerStats().incrementSuccessiveConnectionFailureCount(server);
        ServiceCombLoadBalancerStats.INSTANCE.markFailure(server);
      } else {
        chosenLB.getLoadBalancerStats().incrementActiveRequestsCount(server);
        ServiceCombLoadBalancerStats.INSTANCE.markSuccess(server);
      }
      asyncResp.handle(resp);
    });
  }

TransportClientHandler

public class TransportClientHandler implements Handler {

  private static final Logger log = LoggerFactory.getLogger(TransportClientHandler.class);

  public static final TransportClientHandler INSTANCE = new TransportClientHandler();

  @Override
  public void handle(Invocation invocation, AsyncResponse asyncResp) throws Exception {
    Transport transport = invocation.getTransport();

    log.debug(
        "Sending request {} to {}",
        invocation.getMicroserviceQualifiedName(),
        invocation.getEndpoint().getEndpoint());

    transport.send(invocation, asyncResp);
  }
}

服务端

RestTransport收到请求之后会由之前提到的VertxRestDispatcher进行分派处理。先将Vertx的context封装为HttpServletRequest和HttpServletResponse，然后创建一个VertxRestInvocation进行后续实际的处理流程。

上面已经将VertxRestDispatcher代码贴出，将直接走到onRequest()方法中去，下面会调用VertxRestInvocation.invoke()方法

RestProducerInvocation

这里主要进行了两个步骤，先是调用findRestOperation方法根据HTTP URI定位到对应的OperationMeta，然后调用scheduleInvocation方法进行后续的请求处理。

public void invoke(Transport transport, HttpServletRequestEx requestEx, HttpServletResponseEx responseEx,
    List<HttpServerFilter> httpServerFilters) {
  this.transport = transport;
  this.requestEx = requestEx;
  this.responseEx = responseEx;
  this.httpServerFilters = httpServerFilters;
  requestEx.setAttribute(RestConst.REST_REQUEST, requestEx);

  try {
    // 寻找接口契约，如果找到匹配的契约信息，将信息保存在当前类的全局变量中，供scheduleInvocation执行方式时使用
    findRestOperation();
  } catch (InvocationException e) {
    sendFailResponse(e);
    return;
  }
  // 真正执行方法
  scheduleInvocation();
}

protected void findRestOperation(MicroserviceMeta microserviceMeta) {
  // 会将静态路径和动态路径的Operation进行预处理，可以加速后续的Operation查询定位。
  ServicePathManager servicePathManager = ServicePathManager.getServicePathManager(microserviceMeta);
  if (servicePathManager == null) {
    LOGGER.error("No schema defined for {}:{}.", microserviceMeta.getAppId(), microserviceMeta.getMicroserviceName());
    throw new InvocationException(Status.NOT_FOUND, Status.NOT_FOUND.getReasonPhrase());
  }

  OperationLocator locator = locateOperation(servicePathManager);
  requestEx.setAttribute(RestConst.PATH_PARAMETERS, locator.getPathVarMap());
  this.restOperationMeta = locator.getOperation();
}

OperationLocator

从path和http method定位到具体的operation，在上面的locateOperation方法中会去调用servicePathManager的producerLocateOperation方法，创建一个OperationLocator对象进行请求定位，定位逻辑的代码如下，逻辑是先匹配静态路径，然后再匹配动态路径，这里的动态路径已经提前初始化成正则表达式了，直接使用正则匹配即可。

/**
 * 从path和http method定位到具体的operation
 */
public class OperationLocator {
  private static final Logger LOGGER = LoggerFactory.getLogger(OperationLocator.class);

  private static final String SLASH = "/";

  protected RestOperationMeta operation;

  protected Map<String, String> pathVarMap = new HashMap<>();

  protected boolean resourceFound = false;

  public RestOperationMeta getOperation() {
    return this.operation;
  }

  public Map<String, String> getPathVarMap() {
    return this.pathVarMap;
  }

  // 先在静态路径operation list中查找；如果找不到，则在动态路径operation list中查找
  public void locate(String microserviceName, String path, String httpMethod, MicroservicePaths microservicePaths) {
    // 在静态路径中查找
    operation = locateStaticPathOperation(path, httpMethod, microservicePaths.getStaticPathOperationMap());
    if (operation != null) {
      // 全部定位完成
      return;
    }

    // 在动态路径中查找
    operation = locateDynamicPathOperation(path, microservicePaths.getDynamicPathOperationList(), httpMethod);
    if (operation != null) {
      return;
    }

    Status status = Status.NOT_FOUND;
    if (resourceFound) {
      status = Status.METHOD_NOT_ALLOWED;
    }
    LOGGER.error("locate path failed, status:{}, http method:{}, path:{}, microserviceName:{}",
        status,
        httpMethod,
        path,
        microserviceName);
    throw new InvocationException(status, status.getReasonPhrase());
  }

  protected RestOperationMeta locateStaticPathOperation(String path, String httpMethod,
      Map<String, OperationGroup> staticPathOperations) {
    OperationGroup group = staticPathOperations.get(path);
    if (group == null) {
      return null;
    }

    resourceFound = true;
    return group.findValue(httpMethod);
  }

  protected RestOperationMeta locateDynamicPathOperation(String path, Collection<RestOperationMeta> resourceList,
      String httpMethod) {
    for (RestOperationMeta resource : resourceList) {
      String remainPath = resource.getAbsolutePathRegExp().match(path, pathVarMap);
      // 刚好匹配，不多也不少
      if ("".equals(remainPath)) {
        resourceFound = true;
        if (checkHttpMethod(resource, httpMethod)) {
          return resource;
        }
      }
    }
    return null;
  }

  protected boolean checkHttpMethod(RestOperationMeta operation, String httpMethod) {
    return operation.getHttpMethod().equals(httpMethod);
  }

  // TODO: almost always change path, this make performance lower.
  // Path: /a/b/c -> /a/b/c/
  static String getStandardPath(String path) {
    if (path.length() > 0 && !path.endsWith(SLASH)) {
      path += SLASH;
    }
    return path;
  }
}

AbstractRestInvocation.scheduleInvocation

在该方法中先创建了Invocation对象，这是贯穿ServiceComb上下层的核心对象。然后就是使用之前定位到的OperationMeta进行实际的请求处理。OperationMeta在初始化的时候会设置一个executor线程池，如果用户有自己为微服务、接口、方法配置独立的线程池则使用用户配置的，否则使用共享的默认GroupExecutor线程池，如果该Operation是异步方法则依然在原来的EventLoop线程中执行后续逻辑。

protected void scheduleInvocation() {
  try {
    // 创建Invocation，根据tranport和OperationMeta
    createInvocation();
  } catch (Throwable e) {
    sendFailResponse(e);
    return;
  }

  invocation.onStart(requestEx, start);
  invocation.getInvocationStageTrace().startSchedule();
  OperationMeta operationMeta = restOperationMeta.getOperationMeta();

  try {
    this.setContext();
  } catch (Exception e) {
    LOGGER.error("failed to set invocation context", e);
    sendFailResponse(e);
    return;
  }
  // 找到ProviderQpsFlowControlHandler执行
  // 如果找到就执行HttpServerFilter.beforeSendResponseAsync()
  Holder<Boolean> qpsFlowControlReject = checkQpsFlowControl(operationMeta);
  if (qpsFlowControlReject.value) {
    return;
  }

  try {
    // 使用线程池完成执行后面逻辑，最终在内部通过反射完成方法的调用
    operationMeta.getExecutor().execute(() -> {
      synchronized (this.requestEx) {
        try {
          if (isInQueueTimeout()) {
            throw new InvocationException(Status.INTERNAL_SERVER_ERROR, "Timeout when processing the request.");
          }
          if (requestEx.getAttribute(RestConst.REST_REQUEST) != requestEx) {
            // already timeout
            // in this time, request maybe recycled and reused by web container, do not use requestEx
            LOGGER.error("Rest request already timeout, abandon execute, method {}, operation {}.",
                operationMeta.getHttpMethod(),
                operationMeta.getMicroserviceQualifiedName());
            return;
          }

          runOnExecutor();
        } catch (Throwable e) {
          LOGGER.error("rest server onRequest error", e);
          sendFailResponse(e);
        }
      }
    });
  } catch (Throwable e) {
    LOGGER.error("failed to schedule invocation, message={}, executor={}.", e.getMessage(), e.getClass().getName());
    sendFailResponse(e);
  }
}

ProducerOperationHandler.doInvoke

进入ProducerOperationHandler中去处理，我们跳过其他非核心代码直接进入最后的doInvoke方法，这里使用到了反射机制调用实际业务实现类的相应方法获取到最后的结果，再将结果设置回Response对象中去

public Response doInvoke(Invocation invocation, SwaggerProducerOperation producerOperation) {
  Response response;
  try {
    invocation.onBusinessMethodStart();

    Object[] args = invocation.toProducerArguments();
    // 执行可扩展方法，由SPI加载
    for (ProducerInvokeExtension producerInvokeExtension : producerOperation.getProducerInvokeExtenstionList()) {
      producerInvokeExtension.beforeMethodInvoke(invocation, producerOperation, args);
    }

    // 反射执行具体的方法
    Object result = producerOperation.getProducerMethod().invoke(producerOperation.getProducerInstance(), args);
    response = producerOperation.getResponseMapper().mapResponse(invocation.getStatus(), result);

    invocation.onBusinessMethodFinish();
    invocation.onBusinessFinish();
  } catch (Throwable e) {
    if (shouldPrintErrorLog(e)) {
      invocation.getTraceIdLogger().error(LOGGER, "unexpected error operation={}, message={}",
          invocation.getInvocationQualifiedName(),
          org.apache.servicecomb.foundation.common.utils.ExceptionUtils.getExceptionMessageWithoutTrace(e));
    }
    invocation.onBusinessMethodFinish();
    invocation.onBusinessFinish();
    response = processException(invocation, e);
  }
  return response;
}

servicecomb://127.0.0.1/xxx ==cse://127.0.0.1/xx

# Vertx EnvetLoop线程模型的服务线程数
# 下面两个配置都都没设置时，读取当前机器的核心数上线为8
servicecomb.rest.server.verticle-count(servicecomb.rest.server.thread-count)=8

# highway通信模型下绑定的地址
servicecomb.highway.address=xxx
# highway通信模型下的EnvetLoop程模型的服务线程数，下面两个配置都都没设置时，读取当前机器的核心数上线为8
servicecomb.highway.client.verticle-count(servicecomb.highway.client.thread-count)=8

# 服务端SERVER_LOG_ENABLED，默认为false，Vertx Handle处理请求时打印日志
servicecomb.accesslog.enabled=true

# 客户端CLIENT_LOG_ENABLED，默认为false，Vertx Handle处理请求时打印日志
servicecomb.accesslog.request.enabled=true

# 是否允许跨域，SERVICECOMB_CORS_CONFIG_BASE，默认false。
servicecomb.cors.enabled=false

# 用于VertxRestDispatcher，VertxHttpDispatcher.getOrder()用于升序排序，默认为Integer.MAX_VALUE，即最后一个VertxHttpDispatcher
servicecomb.http.dispatcher.rest.order=Integer.MAX_VALUE

# VertxRestDispatcher最后一个VertxHttpDispatcher，默认开启此VertxHttpDispatcher
servicecomb.http.dispatcher.rest.enabled=true

# VertxRestDispatcher 用于router.routeWithRegex(String)方法的入参，默认为null，即不拦截任何路径
servicecomb.http.dispatcher.rest.pattern=

# VertxRestDispatcher ,默认为false,使用线程池处理逻辑，true时由CompletableFuture执行，不执行HttpServerFilter的拦截方法。
servicecomb.filter-chains.enabled=false

# TCP客户端请求时间，默认为30s
servicecomb.request.timeout=30000

# 扫描加了@RestController的类 默认为true
servicecomb.provider.rest.scanRestController=true

# 客户端根据契约信息去request获取参数异常时打印日志，默认false不打印
servicecomb.codec.printErrorMessage=false

# 服务端Handler配置，可配置多个用,分开，标识Provider的默认Handler,如果
# servicecomb.handler.chain.Provider.service.xxx
servicecomb.handler.chain.Provider.default=

# 客户端Handler配置，可配置多个用,分开 ，标识Consumer的默认Handler,如果servicecomb.handler.chain.Consumer.service.xxx为空时采用此配置
servicecomb.handler.chain.Consumer.default=

# 指定服务的服务端Handler配置
servicecomb.handler.chain.Provider.service.xxx=

# 指定服务的客户端Handler配置
servicecomb.handler.chain.Consumer.service.xxx=

# 客户端过滤器链，需添加在spring容器中
servicecomb.filter-chains.consumer=xx

# 客户端过滤器链，需添加在spring容器中
servicecomb.filter-chains.provider=xx

接收到请求之后的扩展方法，由SPI加载，执行可扩展方法，由SPI加载，如果返回Response将不再执行后边HttpServerFilter，不再执行后边的Handler链，直接执行sendResponse
Response HttpServerFilter.afterReceiveRequest(Invocation invocation, HttpServletRequestEx requestEx)

核心方法，套娃式执行Handler链，由cse.handler.xml加载
Handler.handle(Invocation invocation, AsyncResponse asyncResp)

服务端真正执行某个方法前执行，由SPI加载，生产者方法调用扩展，在调用实际方法之前处理所需的验证/检查
ProducerInvokeExtension.beforeMethodInvoke(SwaggerInvocation invocation, SwaggerProducerOperation producerOperation,
      Object[] args)

服务端返回响应之前的方法
HttpServerFilter.beforeSendResponse(Invocation invocation, HttpServletResponseEx responseEx)

ServerListFilterExt

自定义扩展类，由SPI加载，在执行RuleExt.choose(List servers, Invocation invocation)前执行，过滤从Invocation 获取的服务列表，将结果最终传给RuleExt。

/**
 *  Base interface for server list filters.
 *
 *  Robin ServerListFilter can not support invocation based filter strategies, so we create a new one to
 *  support this.
 */
public interface ServerListFilterExt {
  default int getOrder() {
    return 0;
  }

  default boolean enabled() {
    return true;
  }

  default void setLoadBalancer(LoadBalancer loadBalancer) {
  }

  List<ServiceCombServer> getFilteredListOfServers(List<ServiceCombServer> servers, Invocation invocation);
}