Spring Cloud Hystrix 源码系列：Metrics 收集

最新推荐文章于 2024-07-13 19:19:36 发布

布道

最新推荐文章于 2024-07-13 19:19:36 发布

阅读量1.2k

点赞数

分类专栏： spring源码玩转spring cloud 文章标签： Spring Cloud Hystrix Hystrix Metrics

本文链接：https://blog.csdn.net/alex_xfboy/article/details/87989958

版权

spring源码同时被 2 个专栏收录

47 篇文章 18 订阅

订阅专栏

玩转spring cloud

34 篇文章 11 订阅

订阅专栏

在 Hystrix Command 执行过程(开始执行、结束执行、异常、超时)时会不断发出各类事件，通过收集这些数据，提供给消费者。如断路器、Hystrix Dashboard可以统计分析这些数据，从而完成特定的功能。本文基于hystrix-core 1.5.18（近年来几乎很少更新，建议升级）。

1. rxjava

1.3 Subject

2. Metrics 收集流程

2.1 HystrixCommandMetrics

2.2 HystrixThreadEventStream

2.3 HystrixEventStream

2.4 HystrixPlugins

2.5 HystrixEventNotifier

2.6 HystrixMetricsPublisher

2.7 HystrixConcurrencyStrategy

2.8 HystrixCommandExecutionHook

如果你hystrix-core是1.4.26（也是非常受欢迎的版本），那源码就没有上图那么复杂（没有HystrixEventStream相关概念）

1. rxjava

Observable.share()属于连接操作，组合操作。

Observable.publish( )，将一个Observable转换为一个ConnectableObservable（可连接的Observable）
ConnectableObservable.refCount( )，让一个ConnectableObservable表现得像一个普通的Observable

ConnectableObservable与普通的Observable差不多，除了这一点：ConnectableObservable在被订阅时并不开始发射数据，只有在它的connect()被调用时才开始。用这种方法，你可以等所有的潜在订阅者都订阅了这个Observable之后才开始发射数据

public class Observable<T> {
    public final Observable<T> share() {
        return this.publish().refCount();
    }
    public final ConnectableObservable<T> publish() {
        return OperatorPublish.create(this);
    }
    public Observable<T> refCount() {
        return unsafeCreate(new OnSubscribeRefCount(this));
    }
}

1.3 Subject

Hystrix 基于 RxJava，本文涉及到 Subject 概念，这里提一下 rx.subjects.Subject。Subject 继承Observable，因此可作为被观察者、数据源，也就是一个数据发射器；实现了接口 Observer，因此可作为观察者，可以订阅其他Observable，处理Observable发射出的数据。因此，Subject既可以发射数据，也可以接收数据。

public abstract class Subject<T, R> extends Observable<R> implements Observer<T> {
}

2. Metrics 收集流程

2.1 HystrixCommandMetrics

每个Command的构造器中会获取一个HystrixCommandMetrics工具，用来记录metrics，也就是说，每个CommandKey会拥有一个对应的HystrixCommandMetrics工具。下面是利用HystrixCommandMetrics工具发射各种的事件。

abstract class AbstractCommand<R> implements HystrixInvokableInfo<R>, HystrixObservable<R> {
    protected final HystrixCommandMetrics metrics;
    private static HystrixCommandMetrics initMetrics(HystrixCommandMetrics fromConstructor, HystrixCommandGroupKey groupKey,
                                                     HystrixThreadPoolKey threadPoolKey, HystrixCommandKey commandKey,
                                                     HystrixCommandProperties properties) {
        if (fromConstructor == null) {
            return HystrixCommandMetrics.getInstance(commandKey, groupKey, threadPoolKey, properties);
        } else {
            return fromConstructor;
        }
    }
    protected AbstractCommand(HystrixCommandGroupKey group, HystrixCommandKey key, HystrixThreadPoolKey threadPoolKey, HystrixCircuitBreaker circuitBreaker, HystrixThreadPool threadPool,
            HystrixCommandProperties.Setter commandPropertiesDefaults, HystrixThreadPoolProperties.Setter threadPoolPropertiesDefaults,
            HystrixCommandMetrics metrics, TryableSemaphore fallbackSemaphore, TryableSemaphore executionSemaphore,
            HystrixPropertiesStrategy propertiesStrategy, HystrixCommandExecutionHook executionHook) {
        ...
        this.metrics = initMetrics(metrics, this.commandGroup, this.threadPoolKey, this.commandKey, this.properties);
        //Strategies from plugins
        this.eventNotifier = HystrixPlugins.getInstance().getEventNotifier();
        HystrixMetricsPublisherFactory.createOrRetrievePublisherForCommand(this.commandKey, this.commandGroup, this.metrics, this.circuitBreaker, this.properties);
        ...
    }
}
/**
 * Used by {@link HystrixCommand} to record metrics.
 */
public class HystrixCommandMetrics extends HystrixMetrics {
    // 存储HystrixCommandMetrics的数据
    // String is HystrixCommandKey.name() (we can't use HystrixCommandKey directly as we can't guarantee it implements hashcode/equals correctly)
    private static final ConcurrentHashMap<String, HystrixCommandMetrics> metrics = new ConcurrentHashMap<String, HystrixCommandMetrics>();
    //根据Command构建metrics
    public static HystrixCommandMetrics getInstance(HystrixCommandKey key, HystrixCommandGroupKey commandGroup, HystrixThreadPoolKey threadPoolKey, HystrixCommandProperties properties) {
        // attempt to retrieve from cache first
        HystrixCommandMetrics commandMetrics = metrics.get(key.name());
        if (commandMetrics != null) {
            return commandMetrics;
        } else {
            synchronized (HystrixCommandMetrics.class) {
                HystrixCommandMetrics existingMetrics = metrics.get(key.name());
                if (existingMetrics != null) {
                    return existingMetrics;
                } else {
                    HystrixThreadPoolKey nonNullThreadPoolKey;
                    if (threadPoolKey == null) {
                        nonNullThreadPoolKey = HystrixThreadPoolKey.Factory.asKey(commandGroup.name());
                    } else {
                        nonNullThreadPoolKey = threadPoolKey;
                    }
                    HystrixCommandMetrics newCommandMetrics = new HystrixCommandMetrics(key, commandGroup, nonNullThreadPoolKey, properties, HystrixPlugins.getInstance().getEventNotifier());
                    metrics.putIfAbsent(key.name(), newCommandMetrics);
                    return newCommandMetrics;
                }
            }
        }
    }
    //计数、发事件
    void markCommandStart(HystrixCommandKey commandKey, HystrixThreadPoolKey threadPoolKey, HystrixCommandProperties.ExecutionIsolationStrategy isolationStrategy) {
        int currentCount = concurrentExecutionCount.incrementAndGet();
        HystrixThreadEventStream.getInstance().commandExecutionStarted(commandKey, threadPoolKey, isolationStrategy, currentCount);
    }
    //计数、发事件
    void markCommandDone(ExecutionResult executionResult, HystrixCommandKey commandKey, HystrixThreadPoolKey threadPoolKey, boolean executionStarted) {
        HystrixThreadEventStream.getInstance().executionDone(executionResult, commandKey, threadPoolKey);
        if (executionStarted) {
            concurrentExecutionCount.decrementAndGet();
        }
    }
    /**
     * 获取一个请求的快照（错误数和错误百分比）
     * This metrics should measure the actual health of a {@link HystrixCommand}.  For that reason, the following are included:
     * <p><ul>
     * <li>{@link HystrixEventType#SUCCESS}
     * <li>{@link HystrixEventType#FAILURE}
     * <li>{@link HystrixEventType#TIMEOUT}
     * <li>{@link HystrixEventType#THREAD_POOL_REJECTED}
     * <li>{@link HystrixEventType#SEMAPHORE_REJECTED}
     * </ul><p>
     * The following are not included in either attempts/failures:
     * <p><ul>
     * <li>{@link HystrixEventType#BAD_REQUEST} - this event denotes bad arguments to the command and not a problem with the command
     * <li>{@link HystrixEventType#SHORT_CIRCUITED} - this event measures a health problem in the past, not a problem with the current state
     * <li>{@link HystrixEventType#CANCELLED} - this event denotes a user-cancelled command.  It's not known if it would have been a success or failure, so it shouldn't count for either
     * <li>All Fallback metrics
     * <li>{@link HystrixEventType#EMIT} - this event is not a terminal state for the command
     * <li>{@link HystrixEventType#COLLAPSED} - this event is about the batching process, not the command execution
     */
    public HealthCounts getHealthCounts() {
        return healthCountsStream.getLatest();
    }
}

2.2 HystrixThreadEventStream

HystrixCommandMetrics主要是通过HystrixThreadEventStream工作的！

那HystrixThreadEventStream是什么呢？

每个线程拥有自己的HystrixThreadEventStream（从ThreadLocal中获取对象），它包含了很多Subject<事件,事件>，用来接收和发射数据。

下面源码中HystrixCommandCompletion其实是事件，由于writeOnlyCommandCompletionSubject绑定了数据处理者(下面的writeCommandCompletionsToShardedStreams这个Action1)。

你可以这样理解（文章头部的图）：

AbstractCommand中通过调用HystrixCommandMetrics的markCommandDone开始请求
每个线程都会构建一个HystrixThreadEventStream来处理处理该请求
HystrixThreadEventStream通过内部的writeOnlyCommandCompletionSubject先接受上游的请求，然后将其构建成为一个事件（HystrixCommandCompletion），通过onNext把事件转发到writeCommandCompletionsToShardedStreams
writeCommandCompletionsToShardedStreams本质是个Action，但它内部通过HystrixCommandCompletionStream.write（这个后续也会讲）写入到了一个新的writeOnlySubject（Subject）并且转发事件（onNext）

这里思考一个问题：writeOnlySubject转发了事件HystrixCommandCompletion，那谁是接盘侠呢？

public class HystrixThreadEventStream {
    private final long threadId;
    private final String threadName;
    //用来接收和发射CommandStartSubject
    private final Subject<HystrixCommandExecutionStarted, HystrixCommandExecutionStarted> writeOnlyCommandStartSubject;
    //用来接收和发射CommandCompletionSubject
    private final Subject<HystrixCommandCompletion, HystrixCommandCompletion> writeOnlyCommandCompletionSubject;
    ///每个线程的HystrixThreadEventStream
    private static final ThreadLocal<HystrixThreadEventStream> threadLocalStreams = new ThreadLocal<HystrixThreadEventStream>() {
        @Override
        protected HystrixThreadEventStream initialValue() {
            return new HystrixThreadEventStream(Thread.currentThread());
        }
    };
    HystrixThreadEventStream(Thread thread) {
        //创建为一个数据发射器
        writeOnlyCommandCompletionSubject = PublishSubject.create();
        //绑定发射数据时的处理者
        writeOnlyCommandCompletionSubject
                .onBackpressureBuffer()
                .doOnNext(writeCommandCompletionsToShardedStreams)
                .unsafeSubscribe(Subscribers.empty());
    }
    //可执行的实体,和Runnable很像
    private static final Action1<HystrixCommandCompletion> writeCommandCompletionsToShardedStreams = new Action1<HystrixCommandCompletion>() {
        // 当接收到数据时, 又将数据发送给了command级别的处理者
        @Override
        public void call(HystrixCommandCompletion commandCompletion) {
            // 获取CommandKey对应的HystrixCommandCompletionStream
            HystrixCommandCompletionStream commandStream = HystrixCommandCompletionStream.getInstance(commandCompletion.getCommandKey());
            // 写入数据
            commandStream.write(commandCompletion);

            if (commandCompletion.isExecutedInThread() || commandCompletion.isResponseThreadPoolRejected()) {
                HystrixThreadPoolCompletionStream threadPoolStream = HystrixThreadPoolCompletionStream.getInstance(commandCompletion.getThreadPoolKey());
                threadPoolStream.write(commandCompletion);
            }
        }
    };
    //HystrixCommandMetrics.markCommandDone发射 标记命令结束事件
    public void executionDone(ExecutionResult executionResult, HystrixCommandKey commandKey, HystrixThreadPoolKey threadPoolKey) {
        HystrixCommandCompletion event = HystrixCommandCompletion.from(executionResult, commandKey, threadPoolKey);
        writeOnlyCommandCompletionSubject.onNext(event);
    }
}
public class HystrixCommandCompletion extends HystrixCommandEvent {
}

2.3 HystrixEventStream

上面讲过HystrixThreadEventStream构建事件并转发事件，HystrixCommandCompletionStream就是专门用于接受并处理各种事件（HystrixCommandEvent）的，准确的讲是承上启下，它属于 HystrixEventStream 的一种。它有几个子类：

HystrixCommandCompletionStream
HystrixCommandStartStream
HystrixThreadPoolCompletionStream
HystrixThreadPoolStartStream
HystrixCollapserEventStream

接着上面源码继续分析HystrixCommandCompletionStream，可以接收数据，并将数据提供给其他消费者。上面提的那个问题HystrixCommandCompletionStream的接盘侠是BucketedCounterStream（桶统计流，这个老厉害了，在熔断器章节中会细讲）

public class HystrixCommandCompletionStream {
    // 一个用于接收和发射结束事件的Subject
    private final Subject<HystrixCommandCompletion, HystrixCommandCompletion> writeOnlySubject;
    // 一个Observable，将接收到的数据作为数据源发射给其他消费者
    private final Observable<HystrixCommandCompletion> readOnlyStream;
    //创建
    public static HystrixCommandCompletionStream getInstance(HystrixCommandKey commandKey) {
        HystrixCommandCompletionStream initialStream = streams.get(commandKey.name());
        if (initialStream != null) {
            return initialStream;
        } else {
            synchronized (HystrixCommandCompletionStream.class) {
                HystrixCommandCompletionStream existingStream = streams.get(commandKey.name());
                if (existingStream == null) {
                    HystrixCommandCompletionStream newStream = new HystrixCommandCompletionStream(commandKey);
                    streams.putIfAbsent(commandKey.name(), newStream);
                    return newStream;
                } else {
                    return existingStream;
                }
            }
        }
    }
    // 提供了接收数据的方法,其他工具(如HystrixThreadEventStream)可以将数据写进来
    public void write(HystrixCommandCompletion event) {
        writeOnlySubject.onNext(event);
    }
    HystrixCommandCompletionStream(final HystrixCommandKey commandKey) {
        this.commandKey = commandKey;

        this.writeOnlySubject = new SerializedSubject<HystrixCommandCompletion, HystrixCommandCompletion>(PublishSubject.<HystrixCommandCompletion>create());
        this.readOnlyStream = writeOnlySubject.share();//这个很厉害啦，observe就可以工作
    }
    // 实现HystrixEventStream的observe(方法), 其他消费者可以利用observe()拿到这个数据源，然后订阅它，处理它发射的所有数据
    @Override
    public Observable<HystrixCommandCompletion> observe() {
        return readOnlyStream;
    }
}

2.4 HystrixPlugins

您可以通过实现插件来修改Hystrix的行为，或者向其添加其他行为。您可以通过 HystrixPlugins 注册这些插件。然后，hystrix将把它们应用到所有HystrixCommand、HystrixObservableCommand和 HystrixCollapser实现中，覆盖所有其他实现。

public class HystrixPlugins {    
    final AtomicReference<HystrixEventNotifier> notifier = new AtomicReference<HystrixEventNotifier>();
    final AtomicReference<HystrixConcurrencyStrategy> concurrencyStrategy = new AtomicReference<HystrixConcurrencyStrategy>();
    final AtomicReference<HystrixMetricsPublisher> metricsPublisher = new AtomicReference<HystrixMetricsPublisher>();
    final AtomicReference<HystrixPropertiesStrategy> propertiesFactory = new AtomicReference<HystrixPropertiesStrategy>();
    final AtomicReference<HystrixCommandExecutionHook> commandExecutionHook = new AtomicReference<HystrixCommandExecutionHook>(); 
}

2.5 HystrixEventNotifier

在执行 HystrixCommand和 HystrixObservableCommand期间发生的事件在 HystrixEventNotifier上触发，以提供警报和统计信息收集的机会。

public abstract class HystrixEventNotifier {
    /**
     * 为触发的每个事件调用
     */
    public void markEvent(HystrixEventType eventType, HystrixCommandKey key) {
        // do nothing
    }
    /**
     * 使用线程隔离之后调用执行一个命令，如果一个命令被拒绝,短路等，将不会被调用
     * <p>
     * Will not get called if a command is rejected, short-circuited etc.
     */
    public void markCommandExecution(HystrixCommandKey key, ExecutionIsolationStrategy isolationStrategy, int duration, List<HystrixEventType> eventsDuringExecution) {
        // do nothing
    }
}

2.6 HystrixMetricsPublisher

每一个metrics实例将被捕捉，同时请求 HystrixMetricsPublisher 的实现类并初始化它，这使实现类有机会接收metrics数据对象，并启动一个后台进程来处理metrics，默认实现类不会在任何地方发布它们。如果您希望使用Servo，它是一个内存系统，支持各种检索数据的机制，如通过轮询器或JMX，请参阅它

public interface HystrixMetricsPublisherCommand {
    public void initialize();
}
public class HystrixMetricsPublisherFactory {
    HystrixMetricsPublisherCommand getPublisherForCommand(HystrixCommandKey commandKey, HystrixCommandGroupKey commandOwner, HystrixCommandMetrics metrics, HystrixCircuitBreaker circuitBreaker, HystrixCommandProperties properties) {
        // attempt to retrieve from cache first
    }
    HystrixMetricsPublisherThreadPool getPublisherForThreadPool(HystrixThreadPoolKey threadPoolKey, HystrixThreadPoolMetrics metrics, HystrixThreadPoolProperties properties) {
        // attempt to retrieve from cache first
        HystrixMetricsPublisherThreadPool publisher = threadPoolPublishers.get(threadPoolKey.name());
    }
    HystrixMetricsPublisherCollapser getPublisherForCollapser(HystrixCollapserKey collapserKey, HystrixCollapserMetrics metrics, HystrixCollapserProperties properties) {
        // attempt to retrieve from cache first
        HystrixMetricsPublisherCollapser publisher = collapserPublishers.get(collapserKey.name());
    }
}

2.7 HystrixConcurrencyStrategy

hystrix使用 ThreadLocal、Callable、Runnable、ThreadPoolExecutor 和 BlockingQueue 的实现作为其线程隔离和请求级范围功能的一部分。默认情况下，Hystrix的实现是“开箱即用”的，可以使用以下方法实现 HystrixConcurrencyStrategy 类：

getThreadPool() 和getBlockingQueue() 方法是注入您选择的实现的直接选项，
wrapCallable() 方法允许您修饰hystrix执行的每个 Callable的执行。这对于依赖 ThreadLocal状态实现应用程序功能的系统来说是必不可少的。包装 Callable 可以根据需要从父线程捕获状态并将其复制到子线程。
getRequestVariable() 方法需要一个 HystrixRequestVariable<T> 的实现，该实现的功能与 ThreadLocal类似，但作用域限于请求-在请求中的所有线程上都可用。一般来说，只需将 HystrixRequestContext与自己的HystrixRequestVariable默认实现一起使用就可以了。

2.8 HystrixCommandExecutionHook

一个HystrixCommandExecutionHook实现使您能够访问 HystrixInvokable（Hystrixcommand或 Hystrixobserveblecommand）的执行生命周期，这样您就可以注入行为、日志记录、重写响应、更改线程状态等。您可以通过重写以下一个或多个钩子来实现这一点：

`HystrixCommandExecutionHook`	when Hystrix calls the method
`onStart`	before the `HystrixInvokable`begins executing
`onEmit`	whenever the `HystrixInvokable`emits a value
`onError`	if the `HystrixInvokable` fails with an exception
`onSuccess`	if the `HystrixInvokable`completes successfully
`onThreadStart`	at the start of thread execution if the `HystrixInvokable` is a `HystrixCommand` executed using the `THREADExecutionIsolationStrategy`
`onThreadComplete`	at the completion of thread execution if the `HystrixInvokable`is a `HystrixCommand` executed using the `THREADExecutionIsolationStrategy`
`onExecutionStart`	when the user-defined execution method in the `HystrixInvokable`begins
`onExecutionEmit`	whenever the user-defined execution method in the `HystrixInvokable` emits a value
`onExecutionError`	when the user-defined execution method in the `HystrixInvokable`fails with an exception
`onExecutionSuccess`	when the user-defined execution method in the `HystrixInvokable`completes successfully
`onFallbackStart`	if the `HystrixInvokable` attempts to call the fallback method
`onFallbackEmit`	whenever the fallback method in the `HystrixInvokable` emits a value
`onFallbackError`	if the fallback method in the `HystrixInvokable` fails with an exception or does not exist when a call attempt is made
`onFallbackSuccess`	if the fallback method in the `HystrixInvokable` completes successfully
`onCacheHit`	if the response to the `HystrixInvokable` is found in the `HystrixRequestCache`