Hystrix Dashboard监控源码分析

前言

本篇博文，从源码层次，分析Hystrix Dashboard如何进行Hystrix监控，进而加深Hystrix Dashboard理解。

一、开启Hystrix Dashboard

1、添加依赖

项目中，添加如下依赖。由于项目中，集成了swagger-ui，且swagger-ui使用了高版本的guava，与hystrix所依赖guava冲突，需要移除hystrix所使用的的低版本guava。

		<dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
            <version>2.1.3.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-feign</artifactId>
            <version>1.4.5.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
            <version>2.1.3.RELEASE</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.guava</groupId>
                    <artifactId>guava</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- Hystrix Dashboard监控，所需配置信息 -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
            <version>2.1.3.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-hystrix-dashboard</artifactId>
            <version>1.4.5.RELEASE</version>
        </dependency>
        <!-- Hystrix Dashboard监控，所需配置信息 -->

2、配置项

配置文件中，添加如下配置项

management.endpoints.web.exposure.include=hystrix.stream,turbine.stream

3、配置类

在配置类或启动类上，通过EnableHystrixDashboard注解，开启Hystrix Dashboard监控。

@Configuration
@EnableCircuitBreaker
@EnableHystrixDashboard
@EnableTurbine
public class FeignCloudConfiguration {

}

4、成品展示

启动项目，在浏览器中，输入http://localhost:10001/hystrix。展示如下界面

在中间地址栏，输入http://localhost:10001/actuator/hystrix.stream。

在刚开始的时候，页面一直处理loading状态，这个是因为，没有相关服务被调用的原因。当任意调用一个微服务时，此时能正常显示Hystrix监控数据。

二、Hystrix Dashboard源码分析

项目启动时，控制台，存在如下内容

2021-05-12 20:45:40.813 INFO 10180 — [ main] c.netflix.config.DynamicPropertyFactory : DynamicPropertyFactory is initialized with configuration sources: com.netflix.config.ConcurrentCompositeConfiguration@21bd128b
2021-05-12 20:45:43.970 INFO 10180 — [ main] o.s.b.a.e.web.ServletEndpointRegistrar : Registered ‘/actuator/hystrix.stream’ to hystrix.stream-actuator-endpoint
2021-05-12 20:45:44.746 INFO 10180 — [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 1 endpoint(s) beneath base path ‘/actuator’

也就是，/actuator/hystrix.stream请求路径，被映射到hystrix.stream-actuator-endpoint端点。该端点，通过ServletEndpointRegistrar完成注册操作。注册动作，对应的源码如下所示

@Override
	public void onStartup(ServletContext servletContext) throws ServletException {
		this.servletEndpoints
				.forEach((servletEndpoint) -> register(servletContext, servletEndpoint));
	}

	private void register(ServletContext servletContext,
			ExposableServletEndpoint endpoint) {
		String name = endpoint.getEndpointId().toLowerCaseString() + "-actuator-endpoint";
		String path = this.basePath + "/" + endpoint.getRootPath();
		String urlMapping = path.endsWith("/") ? path + "*" : path + "/*";
		EndpointServlet endpointServlet = endpoint.getEndpointServlet();
		Dynamic registration = servletContext.addServlet(name,
				endpointServlet.getServlet());
		registration.addMapping(urlMapping);
		registration.setInitParameters(endpointServlet.getInitParameters());
		logger.info("Registered '" + path + "' to " + name);
	}

1、ServletEndpointRegistrar装配

ServletEndpointManagementContextConfiguration配置类，存在WebMvcServletEndpointManagementContextConfiguration内部类，正是这个内部类完成ServletEndpointRegistrar 的装配。

	@Configuration
	@ConditionalOnClass(DispatcherServlet.class)
	public static class WebMvcServletEndpointManagementContextConfiguration {

		private final ApplicationContext context;

		public WebMvcServletEndpointManagementContextConfiguration(ApplicationContext context) {
			this.context = context;
		}

		@Bean
		public ServletEndpointRegistrar servletEndpointRegistrar(WebEndpointProperties properties,
				ServletEndpointsSupplier servletEndpointsSupplier) {
			DispatcherServletPath dispatcherServletPath = this.context.getBean(DispatcherServletPath.class);
			return new ServletEndpointRegistrar(dispatcherServletPath.getRelativePath(properties.getBasePath()),
					servletEndpointsSupplier.getEndpoints());
		}
	}

ServletEndpointRegistrar 构造方法中的第一个参数，用于指定请求的basePath，其值对应于WebEndpointProperties.basePath属性，默认为/actuator。

2、端点注册

ServletEndpointRegistrar构造方法的第二个参数，通过ServletEndpointDiscoverer.getEndpoints收集待注册端点信息。

	@Override
	public final Collection<E> getEndpoints() {
		if (this.endpoints == null) {
			this.endpoints = discoverEndpoints();
		}
		return this.endpoints;
	}

	private Collection<E> discoverEndpoints() {
		Collection<EndpointBean> endpointBeans = createEndpointBeans();
		addExtensionBeans(endpointBeans);
		return convertToEndpoints(endpointBeans);
	}
	
	private Collection<EndpointBean> createEndpointBeans() {
		Map<EndpointId, EndpointBean> byId = new LinkedHashMap<>();
		String[] beanNames = BeanFactoryUtils.beanNamesForAnnotationIncludingAncestors(
				this.applicationContext, Endpoint.class);
		for (String beanName : beanNames) {
			if (!ScopedProxyUtils.isScopedTarget(beanName)) {
				EndpointBean endpointBean = createEndpointBean(beanName);
				EndpointBean previous = byId.putIfAbsent(endpointBean.getId(),
						endpointBean);
				Assert.state(previous == null,
						() -> "Found two endpoints with the id '" + endpointBean.getId()
								+ "': '" + endpointBean.getBeanName() + "' and '"
								+ previous.getBeanName() + "'");
			}
		}
		return byId.values();
	}

createEndpointBeans方法，首先收集Spring上下文环境中，存在EndPoint注解标注的bean，然后构建成EndpointBean对象。该对象中，有一个比较重要的属性，即filter属性，后面的过滤动作，是通过该filter完成。

	private Class<?> getFilter(Class<?> type) {
		AnnotationAttributes attributes = AnnotatedElementUtils.getMergedAnnotationAttributes(type, FilteredEndpoint.class);
		if (attributes == null) {
			return null;
		}
		return attributes.getClass("value");
	}

也就是，通过获取Bean上的FilteredEndpoint注解，拿到该Bean所使用的的filter。

	private Collection<E> convertToEndpoints(Collection<EndpointBean> endpointBeans) {
		Set<E> endpoints = new LinkedHashSet<>();
		for (EndpointBean endpointBean : endpointBeans) {
			if (isEndpointExposed(endpointBean)) {
				endpoints.add(convertToEndpoint(endpointBean));
			}
		}
		return Collections.unmodifiableSet(endpoints);
	}
	
    private boolean isEndpointExposed(EndpointBean endpointBean) {
		return isFilterMatch(endpointBean.getFilter(), endpointBean)
				&& !isEndpointFiltered(endpointBean)
				&& isEndpointExposed(endpointBean.getBean());
	}

convertToEndpoints方法，使用EndPoint注解标记的Bean，通过isEndpointExposed方法，进行过滤。通过过滤操作后，收集并返回，用于后期的注册动作。

private boolean isFilterMatch(Class<?> filter, EndpointBean endpointBean) {
		if (!isEndpointExposed(endpointBean.getBean())) {
			return false;
		}
		if (filter == null) {
			return true;
		}
		E endpoint = getFilterEndpoint(endpointBean);
		Class<?> generic = ResolvableType.forClass(EndpointFilter.class, filter)
				.resolveGeneric(0);
		if (generic == null || generic.isInstance(endpoint)) {
			EndpointFilter<E> instance = (EndpointFilter<E>) BeanUtils
					.instantiateClass(filter);
			return isFilterMatch(instance, endpoint);
		}
		return false;

	}

isFilterMatch方法，首先通过isEndpointExposed方法，判断endPointBean是否使用ServletEndpoint注解标记（该方法被ServletEndpointDiscoverer实现类覆写），然后实例化bean初始化时，所获取的filter属性，该属性须为EndpointFilter实例。然后，通过其match方法，完成匹配操作。
Endpoint收集完成后，我们在回到register方法，此时通过this.basePath + “/” + id构成请求路径，用于映射Endpoint所使用的Servlet。

private void register(ServletContext servletContext,
			ExposableServletEndpoint endpoint) {
		String name = endpoint.getEndpointId().toLowerCaseString() + "-actuator-endpoint";
		String path = this.basePath + "/" + endpoint.getRootPath();
		String urlMapping = path.endsWith("/") ? path + "*" : path + "/*";
		EndpointServlet endpointServlet = endpoint.getEndpointServlet();
		Dynamic registration = servletContext.addServlet(name,
				endpointServlet.getServlet());
		registration.addMapping(urlMapping);
		registration.setInitParameters(endpointServlet.getInitParameters());
		logger.info("Registered '" + path + "' to " + name);
	}

3、hystrix.stream端点分析

hystrix.stream所使用的的端点为HystrixStreamEndpoint，其中ServletEndpoint为组合注解，包含Endpoint和FilteredEndpoint两个注解。其中，不难发现hystrix.stream对应HystrixMetricsStreamServlet Servlet。

@ServletEndpoint(id = "hystrix.stream")
public class HystrixStreamEndpoint implements Supplier<EndpointServlet> {

    private final Map<String, String> initParameters;

    public HystrixStreamEndpoint(Map<String, String> initParameters) {
        this.initParameters = initParameters;
    }

    @Override
    public EndpointServlet get() {
        return new EndpointServlet(HystrixMetricsStreamServlet.class)
                .withInitParameters(this.initParameters);
    }
}

4、hystrix.stream请求分析

	@Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        if (isDestroyed) {
            response.sendError(503, "Service has been shut down.");
        } else {
            handleRequest(request, response);
        }
    }

	private void handleRequest(HttpServletRequest request, final HttpServletResponse response) throws ServletException, IOException {
        final AtomicBoolean moreDataWillBeSent = new AtomicBoolean(true);
        Subscription sampleSubscription = null;

        /* ensure we aren't allowing more connections than we want */
        int numberConnections = incrementAndGetCurrentConcurrentConnections();
        try {
            int maxNumberConnectionsAllowed = getMaxNumberConcurrentConnectionsAllowed(); //may change at runtime, so look this up for each request
            if (numberConnections > maxNumberConnectionsAllowed) {
                response.sendError(503, "MaxConcurrentConnections reached: " + maxNumberConnectionsAllowed);
            } else {
                /* initialize response */
                response.setHeader("Content-Type", "text/event-stream;charset=UTF-8");
                response.setHeader("Cache-Control", "no-cache, no-store, max-age=0, must-revalidate");
                response.setHeader("Pragma", "no-cache");

                final PrintWriter writer = response.getWriter();

                //since the sample stream is based on Observable.interval, events will get published on an RxComputation thread
                //since writing to the servlet response is blocking, use the Rx IO thread for the write that occurs in the onNext
                sampleSubscription = sampleStream
                        .observeOn(Schedulers.io())
                        .subscribe(new Subscriber<String>() {
                            @Override
                            public void onCompleted() {
                                logger.error("HystrixSampleSseServlet: ({}) received unexpected OnCompleted from sample stream", getClass().getSimpleName());
                                moreDataWillBeSent.set(false);
                            }

                            @Override
                            public void onError(Throwable e) {
                                moreDataWillBeSent.set(false);
                            }

                            @Override
                            public void onNext(String sampleDataAsString) {
                                if (sampleDataAsString != null) {
                                    writer.print("data: " + sampleDataAsString + "\n\n");
                                    // explicitly check for client disconnect - PrintWriter does not throw exceptions
                                    if (writer.checkError()) {
                                        moreDataWillBeSent.set(false);
                                    }
                                    writer.flush();
                                }
                            }
                        });

                while (moreDataWillBeSent.get() && !isDestroyed) {
                    try {
                        Thread.sleep(pausePollerThreadDelayInMs);
                        //in case stream has not started emitting yet, catch any clients which connect/disconnect before emits start
                        writer.print("ping: \n\n");
                        // explicitly check for client disconnect - PrintWriter does not throw exceptions
                        if (writer.checkError()) {
                            moreDataWillBeSent.set(false);
                        }
                        writer.flush();
                    } catch (InterruptedException e) {
                        moreDataWillBeSent.set(false);
                    }
                }
            }
        } finally {
            decrementCurrentConcurrentConnections();
            if (sampleSubscription != null && !sampleSubscription.isUnsubscribed()) {
                sampleSubscription.unsubscribe();
            }
        }
    }

get请求时，通过设置Content-Type为text/event-stream，基于数据流方式，接收服务端推送的数据。然后通过Observable，接收待发送监控数据后，将其设置为data部分，推送给前端展示。对应数据内容如下图所示

而整个监控数据的源头，对应于如下代码

this.singleSource = Observable.interval(delayInMs, TimeUnit.MILLISECONDS)
                .map(new Func1<Long, DashboardData>() {
                    @Override
                    public DashboardData call(Long timestamp) {
                        return new DashboardData(
                                HystrixCommandMetrics.getInstances(),
                                HystrixThreadPoolMetrics.getInstances(),
                                HystrixCollapserMetrics.getInstances()
                        );
                    }
                })
                .doOnSubscribe(new Action0() {
                    @Override
                    public void call() {
                        isSourceCurrentlySubscribed.set(true);
                    }
                })
                .doOnUnsubscribe(new Action0() {
                    @Override
                    public void call() {
                        isSourceCurrentlySubscribed.set(false);
                    }
                })
                .share()
                .onBackpressureDrop();

关于监控数据具体细节，会在后续博文中，继续介绍。

总结

1、hystrix.stream请求，通过HystrixMetricsStreamServlet完成请求操作。
2、HystrixMetricsStreamServlet通过HystrixStreamEndpoint 完成请求路径的映射与绑定。
3、HystrixStreamEndpoint 由于其所使用的EndPoint注解，被ServletEndpointRegistrar
收集，并注册。