在 Hystrix Command 执行过程(开始执行、结束执行、异常、超时)时会不断发出各类事件,通过收集这些数据,提供给消费者。如断路器、Hystrix Dashboard可以统计分析这些数据,从而完成特定的功能。本文基于hystrix-core 1.5.18(近年来几乎很少更新,建议升级)。
目录
2.7 HystrixConcurrencyStrategy
2.8 HystrixCommandExecutionHook
如果你hystrix-core是1.4.26(也是非常受欢迎的版本),那源码就没有上图那么复杂(没有HystrixEventStream相关概念)
1. rxjava
1.2 Observable.share
Observable.share()属于连接操作,组合操作。
- Observable.publish( ),将一个Observable转换为一个ConnectableObservable(可连接的Observable)
- ConnectableObservable.refCount( ),让一个ConnectableObservable表现得像一个普通的Observable
ConnectableObservable与普通的Observable差不多,除了这一点:ConnectableObservable在被订阅时并不开始发射数据,只有在它的connect()
被调用时才开始。用这种方法,你可以等所有的潜在订阅者都订阅了这个Observable之后才开始发射数据
public class Observable<T> {
public final Observable<T> share() {
return this.publish().refCount();
}
public final ConnectableObservable<T> publish() {
return OperatorPublish.create(this);
}
public Observable<T> refCount() {
return unsafeCreate(new OnSubscribeRefCount(this));
}
}
1.3 Subject
Hystrix 基于 RxJava,本文涉及到 Subject 概念,这里提一下 rx.subjects.Subject。Subject 继承Observable,因此可作为被观察者、数据源,也就是一个数据发射器;实现了接口 Observer,因此可作为观察者,可以订阅其他Observable,处理Observable发射出的数据。因此,Subject既可以发射数据,也可以接收数据。
public abstract class Subject<T, R> extends Observable<R> implements Observer<T> {
}
2. Metrics 收集流程
2.1 HystrixCommandMetrics
每个Command的构造器中会获取一个HystrixCommandMetrics工具,用来记录metrics,也就是说,每个CommandKey会拥有一个对应的HystrixCommandMetrics工具。下面是利用HystrixCommandMetrics工具发射 各种 的事件。
abstract class AbstractCommand<R> implements HystrixInvokableInfo<R>, HystrixObservable<R> {
protected final HystrixCommandMetrics metrics;
private static HystrixCommandMetrics initMetrics(HystrixCommandMetrics fromConstructor, HystrixCommandGroupKey groupKey,
HystrixThreadPoolKey threadPoolKey, HystrixCommandKey commandKey,
HystrixCommandProperties properties) {
if (fromConstructor == null) {
return HystrixCommandMetrics.getInstance(commandKey, groupKey, threadPoolKey, properties);
} else {
return fromConstructor;
}
}
protected AbstractCommand(HystrixCommandGroupKey group, HystrixCommandKey key, HystrixThreadPoolKey threadPoolKey, HystrixCircuitBreaker circuitBreaker, HystrixThreadPool threadPool,
HystrixCommandProperties.Setter commandPropertiesDefaults, HystrixThreadPoolProperties.Setter threadPoolPropertiesDefaults,
HystrixCommandMetrics metrics, TryableSemaphore fallbackSemaphore, TryableSemaphore executionSemaphore,
HystrixPropertiesStrategy propertiesStrategy, HystrixCommandExecutionHook executionHook) {
...
this.metrics = initMetrics(metrics, this.commandGroup, this.threadPoolKey, this.commandKey, this.properties);
//Strategies from plugins
this.eventNotifier = HystrixPlugins.getInstance().getEventNotifier();
HystrixMetricsPublisherFactory.createOrRetrievePublisherForCommand(this.commandKey, this.commandGroup, this.metrics, this.circuitBreaker, this.properties);
...
}
}
/**
* Used by {@link HystrixCommand} to record metrics.
*/
public class HystrixCommandMetrics extends HystrixMetrics {
// 存储HystrixCommandMetrics的数据
// String is HystrixCommandKey.name() (we can't use HystrixCommandKey directly as we can't guarantee it implements hashcode/equals correctly)
private static final ConcurrentHashMap<String, HystrixCommandMetrics> metrics = new ConcurrentHashMap<String, HystrixCommandMetrics>();
//根据Command构建metrics
public static HystrixCommandMetrics getInstance(HystrixCommandKey key, HystrixCommandGroupKey commandGroup, HystrixThreadPoolKey threadPoolKey, HystrixCommandProperties properties) {
// attempt to retrieve from cache first
HystrixCommandMetrics commandMetrics = metrics.get(key.name());
if (commandMetrics != null) {
return commandMetrics;
} else {
synchronized (HystrixCommandMetrics.class) {
HystrixCommandMetrics existingMetrics = metrics.get(key.name());
if (existingMetrics != null) {
return existingMetrics;
} else {
HystrixThreadPoolKey nonNullThreadPoolKey;
if (threadPoolKey == null) {
nonNullThreadPoolKey = HystrixThreadPoolKey.Factory.asKey(commandGroup.name());
} else {
nonNullThreadPoolKey = threadPoolKey;
}
HystrixCommandMetrics newCommandMetrics = new HystrixCommandMetrics(key, commandGroup, nonNullThreadPoolKey, properties, HystrixPlugins.getInstance().getEventNotifier());
metrics.putIfAbsent(key.name(), newCommandMetrics);
return newCommandMetrics;
}
}
}
}
//计数、发事件
void markCommandStart(HystrixCommandKey commandKey, HystrixThreadPoolKey threadPoolKey, HystrixCommandProperties.ExecutionIsolationStrategy isolationStrategy) {
int currentCount = concurrentExecutionCount.incrementAndGet();
HystrixThreadEventStream.getInstance().commandExecutionStarted(commandKey, threadPoolKey, isolationStrategy, currentCount);
}
//计数、发事件
void markCommandDone(ExecutionResult executionResult, HystrixCommandKey commandKey, HystrixThreadPoolKey threadPoolKey, boolean executionStarted) {
HystrixThreadEventStream.getInstance().executionDone(executionResult, commandKey, threadPoolKey);
if (executionStarted) {
concurrentExecutionCount.decrementAndGet();
}
}
/**
* 获取一个请求的快照(错误数和错误百分比)
* This metrics should measure the actual health of a {@link HystrixCommand}. For that reason, the following are included:
* <p><ul>
* <li>{@link HystrixEventType#SUCCESS}
* <li>{@link HystrixEventType#FAILURE}
* <li>{@link HystrixEventType#TIMEOUT}
* <li>{@link HystrixEventType#THREAD_POOL_REJECTED}
* <li>{@link HystrixEventType#SEMAPHORE_REJECTED}
* </ul><p>
* The following are not included in either attempts/failures:
* <p><ul>
* <li>{@link HystrixEventType#BAD_REQUEST} - this event denotes bad arguments to the command and not a problem with the command
* <li>{@link HystrixEventType#SHORT_CIRCUITED} - this event measures a health problem in the past, not a problem with the current state
* <li>{@link HystrixEventType#CANCELLED} - this event denotes a user-cancelled command. It's not known if it would have been a success or failure, so it shouldn't count for either
* <li>All Fallback metrics
* <li>{@link HystrixEventType#EMIT} - this event is not a terminal state for the command
* <li>{@link HystrixEventType#COLLAPSED} - this event is about the batching process, not the command execution
*/
public HealthCounts getHealthCounts() {
return healthCountsStream.getLatest();
}
}
2.2 HystrixThreadEventStream
HystrixCommandMetrics主要是通过HystrixThreadEventStream工作的!
那HystrixThreadEventStream是什么呢?
每个线程拥有自己的HystrixThreadEventStream(从ThreadLocal中获取对象),它包含了很多Subject<事件,事件>,用来接收和发射数据。
下面源码中HystrixCommandCompletion其实是事件,由于writeOnlyCommandCompletionSubject绑定了数据处理者(下面的writeCommandCompletionsToShardedStreams这个Action1)。
你可以这样理解(文章头部的图):
- AbstractCommand中通过调用HystrixCommandMetrics的markCommandDone开始请求
- 每个线程都会构建一个HystrixThreadEventStream来处理处理该请求
- HystrixThreadEventStream通过内部的writeOnlyCommandCompletionSubject先接受上游的请求,然后将其构建成为一个事件(HystrixCommandCompletion),通过onNext把事件转发到writeCommandCompletionsToShardedStreams
- writeCommandCompletionsToShardedStreams本质是个Action,但它内部通过HystrixCommandCompletionStream.write(这个后续也会讲)写入到了一个新的writeOnlySubject(Subject)并且转发事件(onNext)
这里思考一个问题:writeOnlySubject转发了事件HystrixCommandCompletion,那谁是接盘侠呢?
public class HystrixThreadEventStream {
private final long threadId;
private final String threadName;
//用来接收和发射CommandStartSubject
private final Subject<HystrixCommandExecutionStarted, HystrixCommandExecutionStarted> writeOnlyCommandStartSubject;
//用来接收和发射CommandCompletionSubject
private final Subject<HystrixCommandCompletion, HystrixCommandCompletion> writeOnlyCommandCompletionSubject;
///每个线程的HystrixThreadEventStream
private static final ThreadLocal<HystrixThreadEventStream> threadLocalStreams = new ThreadLocal<HystrixThreadEventStream>() {
@Override
protected HystrixThreadEventStream initialValue() {
return new HystrixThreadEventStream(Thread.currentThread());
}
};
HystrixThreadEventStream(Thread thread) {
//创建为一个数据发射器
writeOnlyCommandCompletionSubject = PublishSubject.create();
//绑定发射数据时的处理者
writeOnlyCommandCompletionSubject
.onBackpressureBuffer()
.doOnNext(writeCommandCompletionsToShardedStreams)
.unsafeSubscribe(Subscribers.empty());
}
//可执行的实体,和Runnable很像
private static final Action1<HystrixCommandCompletion> writeCommandCompletionsToShardedStreams = new Action1<HystrixCommandCompletion>() {
// 当接收到数据时, 又将数据发送给了command级别的处理者
@Override
public void call(HystrixCommandCompletion commandCompletion) {
// 获取CommandKey对应的HystrixCommandCompletionStream
HystrixCommandCompletionStream commandStream = HystrixCommandCompletionStream.getInstance(commandCompletion.getCommandKey());
// 写入数据
commandStream.write(commandCompletion);
if (commandCompletion.isExecutedInThread() || commandCompletion.isResponseThreadPoolRejected()) {
HystrixThreadPoolCompletionStream threadPoolStream = HystrixThreadPoolCompletionStream.getInstance(commandCompletion.getThreadPoolKey());
threadPoolStream.write(commandCompletion);
}
}
};
//HystrixCommandMetrics.markCommandDone发射 标记命令结束事件
public void executionDone(ExecutionResult executionResult, HystrixCommandKey commandKey, HystrixThreadPoolKey threadPoolKey) {
HystrixCommandCompletion event = HystrixCommandCompletion.from(executionResult, commandKey, threadPoolKey);
writeOnlyCommandCompletionSubject.onNext(event);
}
}
public class HystrixCommandCompletion extends HystrixCommandEvent {
}
2.3 HystrixEventStream
上面讲过HystrixThreadEventStream构建事件并转发事件,HystrixCommandCompletionStream就是专门用于接受并处理各种事件(HystrixCommandEvent)的,准确的讲是承上启下,它属于 HystrixEventStream 的一种。它有几个子类:
- HystrixCommandCompletionStream
- HystrixCommandStartStream
- HystrixThreadPoolCompletionStream
- HystrixThreadPoolStartStream
- HystrixCollapserEventStream
接着上面源码继续分析HystrixCommandCompletionStream,可以接收数据,并将数据提供给其他消费者。上面提的那个问题HystrixCommandCompletionStream的接盘侠是BucketedCounterStream(桶统计流,这个老厉害了,在熔断器章节中会细讲)
public class HystrixCommandCompletionStream {
// 一个用于接收和发射结束事件的Subject
private final Subject<HystrixCommandCompletion, HystrixCommandCompletion> writeOnlySubject;
// 一个Observable,将接收到的数据作为数据源发射给其他消费者
private final Observable<HystrixCommandCompletion> readOnlyStream;
//创建
public static HystrixCommandCompletionStream getInstance(HystrixCommandKey commandKey) {
HystrixCommandCompletionStream initialStream = streams.get(commandKey.name());
if (initialStream != null) {
return initialStream;
} else {
synchronized (HystrixCommandCompletionStream.class) {
HystrixCommandCompletionStream existingStream = streams.get(commandKey.name());
if (existingStream == null) {
HystrixCommandCompletionStream newStream = new HystrixCommandCompletionStream(commandKey);
streams.putIfAbsent(commandKey.name(), newStream);
return newStream;
} else {
return existingStream;
}
}
}
}
// 提供了接收数据的方法,其他工具(如HystrixThreadEventStream)可以将数据写进来
public void write(HystrixCommandCompletion event) {
writeOnlySubject.onNext(event);
}
HystrixCommandCompletionStream(final HystrixCommandKey commandKey) {
this.commandKey = commandKey;
this.writeOnlySubject = new SerializedSubject<HystrixCommandCompletion, HystrixCommandCompletion>(PublishSubject.<HystrixCommandCompletion>create());
this.readOnlyStream = writeOnlySubject.share();//这个很厉害啦,observe就可以工作
}
// 实现HystrixEventStream的observe(方法), 其他消费者可以利用observe()拿到这个数据源,然后订阅它,处理它发射的所有数据
@Override
public Observable<HystrixCommandCompletion> observe() {
return readOnlyStream;
}
}
2.4 HystrixPlugins
您可以通过实现插件来修改Hystrix的行为,或者向其添加其他行为。您可以通过 HystrixPlugins 注册这些插件。然后,hystrix将把它们应用到所有HystrixCommand
、HystrixObservableCommand
和 HystrixCollapser
实现中,覆盖所有其他实现。
public class HystrixPlugins {
final AtomicReference<HystrixEventNotifier> notifier = new AtomicReference<HystrixEventNotifier>();
final AtomicReference<HystrixConcurrencyStrategy> concurrencyStrategy = new AtomicReference<HystrixConcurrencyStrategy>();
final AtomicReference<HystrixMetricsPublisher> metricsPublisher = new AtomicReference<HystrixMetricsPublisher>();
final AtomicReference<HystrixPropertiesStrategy> propertiesFactory = new AtomicReference<HystrixPropertiesStrategy>();
final AtomicReference<HystrixCommandExecutionHook> commandExecutionHook = new AtomicReference<HystrixCommandExecutionHook>();
}
2.5 HystrixEventNotifier
在执行 HystrixCommand
和 HystrixObservableCommand
期间发生的事件在 HystrixEventNotifier上触发,以提供警报和统计信息收集的机会。
public abstract class HystrixEventNotifier {
/**
* 为触发的每个事件调用
*/
public void markEvent(HystrixEventType eventType, HystrixCommandKey key) {
// do nothing
}
/**
* 使用线程隔离之后调用执行一个命令,如果一个命令被拒绝,短路等,将不会被调用
* <p>
* Will not get called if a command is rejected, short-circuited etc.
*/
public void markCommandExecution(HystrixCommandKey key, ExecutionIsolationStrategy isolationStrategy, int duration, List<HystrixEventType> eventsDuringExecution) {
// do nothing
}
}
2.6 HystrixMetricsPublisher
每一个metrics实例将被捕捉,同时请求 HystrixMetricsPublisher
的实现类并初始化它,这使实现类有机会接收metrics数据对象,并启动一个后台进程来处理metrics,默认实现类不会在任何地方发布它们。如果您希望使用Servo,它是一个内存系统,支持各种检索数据的机制,如通过轮询器或JMX,请参阅它
public interface HystrixMetricsPublisherCommand {
public void initialize();
}
public class HystrixMetricsPublisherFactory {
HystrixMetricsPublisherCommand getPublisherForCommand(HystrixCommandKey commandKey, HystrixCommandGroupKey commandOwner, HystrixCommandMetrics metrics, HystrixCircuitBreaker circuitBreaker, HystrixCommandProperties properties) {
// attempt to retrieve from cache first
}
HystrixMetricsPublisherThreadPool getPublisherForThreadPool(HystrixThreadPoolKey threadPoolKey, HystrixThreadPoolMetrics metrics, HystrixThreadPoolProperties properties) {
// attempt to retrieve from cache first
HystrixMetricsPublisherThreadPool publisher = threadPoolPublishers.get(threadPoolKey.name());
}
HystrixMetricsPublisherCollapser getPublisherForCollapser(HystrixCollapserKey collapserKey, HystrixCollapserMetrics metrics, HystrixCollapserProperties properties) {
// attempt to retrieve from cache first
HystrixMetricsPublisherCollapser publisher = collapserPublishers.get(collapserKey.name());
}
}
2.7 HystrixConcurrencyStrategy
hystrix使用 ThreadLocal、Callable、Runnable、ThreadPoolExecutor 和 BlockingQueue 的实现作为其线程隔离和请求级范围功能的一部分。默认情况下,Hystrix的实现是“开箱即用”的,可以使用以下方法实现 HystrixConcurrencyStrategy
类:
- getThreadPool() 和getBlockingQueue()
方法
是注入您选择的实现的直接选项, - wrapCallable() 方法允许您修饰hystrix执行的每个 Callable
的执行
。这对于依赖 ThreadLocal状态实现应用程序功能的系统来说是必不可少的。包装 Callable 可以根据需要从父线程捕获状态并将其复制到子线程。 - getRequestVariable() 方法需要一个
HystrixRequestVariable<T>
的实现,该实现的功能与 ThreadLocal类似,但作用域限于请求-在请求中的所有线程上都可用。一般来说,只需将HystrixRequestContext
与自己的HystrixRequestVariable
默认实现一起使用就可以了。
2.8 HystrixCommandExecutionHook
一个HystrixCommandExecutionHook
实现使您能够访问 HystrixInvokable(Hystrixcommand或 Hystrixobserveblecommand)的执行生命周期,这样您就可以注入行为、日志记录、重写响应、更改线程状态等。您可以通过重写以下一个或多个钩子来实现这一点:
HystrixCommandExecutionHook | when Hystrix calls the method |
---|---|
onStart | before the HystrixInvokable begins executing |
onEmit | whenever the HystrixInvokable emits a value |
onError | if the HystrixInvokable fails with an exception |
onSuccess | if the HystrixInvokable completes successfully |
onThreadStart | at the start of thread execution if the HystrixInvokable is a HystrixCommand executed using the THREAD ExecutionIsolationStrategy |
onThreadComplete | at the completion of thread execution if the HystrixInvokable is a HystrixCommand executed using the THREAD ExecutionIsolationStrategy |
onExecutionStart | when the user-defined execution method in the HystrixInvokable begins |
onExecutionEmit | whenever the user-defined execution method in the HystrixInvokable emits a value |
onExecutionError | when the user-defined execution method in the HystrixInvokable fails with an exception |
onExecutionSuccess | when the user-defined execution method in the HystrixInvokable completes successfully |
onFallbackStart | if the HystrixInvokable attempts to call the fallback method |
onFallbackEmit | whenever the fallback method in the HystrixInvokable emits a value |
onFallbackError | if the fallback method in the HystrixInvokable fails with an exception or does not exist when a call attempt is made |
onFallbackSuccess | if the fallback method in the HystrixInvokable completes successfully |
onCacheHit | if the response to the HystrixInvokable is found in the HystrixRequestCache |
在下一篇中会着重分享“熔断器”。