转载:https://www.sczyh30.com/posts/Microservice/netflix-hystrix-how-it-works-summary/
Hystrix中的Circuit Breaker的实现比较明了。整个HystrixCircuitBreaker
接口一共有三个方法和三个静态类:
其中allowRequest()
方法表示是否允许指令执行,isOpen()
方法表示断路器是否为开启状态,markSuccess()
用于将断路器关闭。
Factory
静态类相当于Circuit Breaker Factory,用于获取相应的HystrixCircuitBreaker
。我们来看一下其实现:
/**
* @ExcludeFromJavadoc
* @ThreadSafe
*/
public static class Factory {
// String is HystrixCommandKey.name() (we can't use HystrixCommandKey directly as we can't guarantee it implements hashcode/equals correctly)
private static ConcurrentHashMap<String, HystrixCircuitBreaker> circuitBreakersByCommand = new ConcurrentHashMap<String, HystrixCircuitBreaker>();
/**
* Get the {@link HystrixCircuitBreaker} instance for a given {@link HystrixCommandKey}.
* <p>
* This is thread-safe and ensures only 1 {@link HystrixCircuitBreaker} per {@link HystrixCommandKey}.
*
* @param key
* {@link HystrixCommandKey} of {@link HystrixCommand} instance requesting the {@link HystrixCircuitBreaker}
* @param group
* Pass-thru to {@link HystrixCircuitBreaker}
* @param properties
* Pass-thru to {@link HystrixCircuitBreaker}
* @param metrics
* Pass-thru to {@link HystrixCircuitBreaker}
* @return {@link HystrixCircuitBreaker} for {@link HystrixCommandKey}
*/
public static HystrixCircuitBreaker getInstance(HystrixCommandKey key, HystrixCommandGroupKey group, HystrixCommandProperties properties, HystrixCommandMetrics metrics) {
// this should find it for all but the first time
HystrixCircuitBreaker previouslyCached = circuitBreakersByCommand.get(key.name());
if (previouslyCached != null) {
return previouslyCached;
}
// if we get here this is the first time so we need to initialize
// Create and add to the map ... use putIfAbsent to atomically handle the possible race-condition of
// 2 threads hitting this point at the same time and let ConcurrentHashMap provide us our thread-safety
// If 2 threads hit here only one will get added and the other will get a non-null response instead.
HystrixCircuitBreaker cbForCommand = circuitBreakersByCommand.putIfAbsent(key.name(), new HystrixCircuitBreakerImpl(key, group, properties, metrics));
if (cbForCommand == null) {
// this means the putIfAbsent step just created a new one so let's retrieve and return it
return circuitBreakersByCommand.get(key.name());
} else {
// this means a race occurred and while attempting to 'put' another one got there before
// and we instead retrieved it and will now return it
return cbForCommand;
}
}
/**
* Get the {@link HystrixCircuitBreaker} instance for a given {@link HystrixCommandKey} or null if none exists.
*
* @param key
* {@link HystrixCommandKey} of {@link HystrixCommand} instance requesting the {@link HystrixCircuitBreaker}
* @return {@link HystrixCircuitBreaker} for {@link HystrixCommandKey}
*/
public static HystrixCircuitBreaker getInstance(HystrixCommandKey key) {
return circuitBreakersByCommand.get(key.name());
}
/**
* Clears all circuit breakers. If new requests come in instances will be recreated.
*/
/* package */static void reset() {
circuitBreakersByCommand.clear();
}
}
Hystrix在Factory
类中维护了一个ConcurrentHashMap
用于存储与每一个HystrixCommandKey
相对应的HystrixCircuitBreaker
。每当我们通过getInstance
方法从中获取HystrixCircuitBreaker
的时候,Hystrix首先会检查ConcurrentHashMap
中有没有对应的缓存的断路器,如果有的话直接返回。如果没有的话就会新创建一个HystrixCircuitBreaker
实例,将其添加到缓存中并且返回。
HystrixCircuitBreakerImpl
静态类是HystrixCircuitBreaker
接口的实现。我们可以看到HystrixCircuitBreakerImpl
类中有四个成员变量。其中properties
是对应HystrixCommand
的属性类,metrics
是对应HystrixCommand
的度量数据类。由于会工作在并发环境下,我们用一个AtomicBoolean
类型的变量circuitOpen
来代表断路器的状态(默认是false
代表关闭,这里没有特意实现Half-Open这个状态),并用一个AtomicLong
类型的变量circuitOpenedOrLastTestedTime
记录着断路恢复计时器的初始时间,用于Open状态向Close状态的转换。
我们首先来看一下isOpen
方法的实现:
@Override
public boolean isOpen() {
if (circuitOpen.get()) {
// if we're open we immediately return true and don't bother attempting to 'close' ourself as that is left to allowSingleTest and a subsequent successful test to close
return true;
}
// we're closed, so let's see if errors have made us so we should trip the circuit open
HealthCounts health = metrics.getHealthCounts();
// check if we are past the statisticalWindowVolumeThreshold
if (health.getTotalRequests() < properties.circuitBreakerRequestVolumeThreshold().get()) {
// we are not past the minimum volume threshold for the statisticalWindow so we'll return false immediately and not calculate anything
return false;
}
if (health.getErrorPercentage() < properties.circuitBreakerErrorThresholdPercentage().get()) {
return false;
} else {
// our failure rate is too high, trip the circuit
if (circuitOpen.compareAndSet(false, true)) {
// if the previousValue was false then we want to set the currentTime
circuitOpenedOrLastTestedTime.set(System.currentTimeMillis());
return true;
} else {
// How could previousValue be true? If another thread was going through this code at the same time a race-condition could have
// caused another thread to set it to true already even though we were in the process of doing the same
// In this case, we know the circuit is open, so let the other thread set the currentTime and report back that the circuit is open
return true;
}
}
}
首先通过circuitOpen.get()
获取断路器的状态,如果是开启状态(true
)则返回true
。否则,Hystrix会从Metrics数据中获取HealthCounts
对象,然后检查对应的请求总数(totalCount
)是否小于属性中的请求容量阈值(circuitBreakerRequestVolumeThreshold
),如果是的话表示断路器可以保持关闭状态,返回false
。如果不满足请求总数条件,就再检查错误比率(errorPercentage
)是否小于属性中的错误百分比阈值(circuitBreakerErrorThresholdPercentage
,默认 50),如果是的话表示断路器可以保持关闭状态,返回 false
;如果超过阈值,Hystrix会判定服务的某些地方出现了问题,因此通过CAS操作将断路器设为开启状态,并记录此时的系统时间作为定时器初始时间,最后返回 true
。
我们再来看一下判断Open状态下计时器的实现方法allowSingleTest
:
|
首先获取断路恢复计时器记录的初始时间circuitOpenedOrLastTestedTime
,然后判断以下两个条件是否同时满足:
- 断路器的状态为开启状态(
circuitOpen.get() == true
) - 当前时间与计时器初始时间之差大于计时器阈值
circuitBreakerSleepWindowInMilliseconds
(默认为 5 秒)
如果同时满足的话,表示可以从Open
状态向Close
状态转换。Hystrix会通过CAS操作将circuitOpenedOrLastTestedTime
设为当前时间,并返回true
。如果不同时满足,返回false
,代表断路器关闭或者计时器时间未到。
有了这个函数以后,我们再来看一下allowRequest
的实现:
@Override
public boolean allowRequest() {
if (properties.circuitBreakerForceOpen().get()) {
// properties have asked us to force the circuit open so we will allow NO requests
return false;
}
if (properties.circuitBreakerForceClosed().get()) {
// we still want to allow isOpen() to perform it's calculations so we simulate normal behavior
isOpen();
// properties have asked us to ignore errors so we will ignore the results of isOpen and just allow all traffic through
return true;
}
return !isOpen() || allowSingleTest();
}
非常直观。首先先读取属性中的强制设定值(可以强制设定状态),如果没有设定的话,就判断断路器是否关闭或者断路恢复计时器是否到达时间,只要满足其中一个条件就返回true
,即允许执行操作指令。
最后就是markSuccess
方法了,它用于关闭断路器并重置统计数据。代码非常直观,就不多说了:
public void markSuccess() {
if (circuitOpen.get()) {
if (circuitOpen.compareAndSet(true, false)) {
//win the thread race to reset metrics
//Unsubscribe from the current stream to reset the health counts stream. This only affects the health counts view,
//and all other metric consumers are unaffected by the reset
metrics.resetStream();
}
}
}
Hystrix的Circuit Breaker可以用以下的图来总结:
至于Hystrix在底层执行Command时是如何利用HystrixCircuitBreaker
的,可以看AbstractCommand
类中toObservable
方法和getRunObservableDecoratedForMetricsAndErrorHandling
方法的源码。