1 前言
在1.8.0之前的版本中,Sentinel的降级策略可以选择RT,也就是平均响应时间的降级策略。从网上也能了解到基本规则就是:
平均响应时间 (DEGRADE_GRADE_RT):当资源的平均响应时间超过阈值(DegradeRule 中的 count,以 ms 为单位)之后,资源进入准降级状态。接下来如果持续进入 5 个请求,它们的 RT 都持续超过这个阈值,那么在接下的时间窗口(DegradeRule 中的 timeWindow,以 s 为单位)之内,对这个方法的调用都会自动地返回(抛出 DegradeException)。在下一个时间窗口到来时, 会接着再放入5个请求, 再重复上面的判断.
当请求处理时间较长的时候,熔断情况有时候和我的理解不太相符,所以看看到底是什么逻辑。
不过新版中RT已经被慢调用比例替代了,本篇就当满足下好奇心。水平有限,有错误还请指出。
2 测试过程
- 首先就先开一个SpringBoot的服务,配置Sentinel地址,处理3s:
@GetMapping("/testa") public String testA() throws InterruptedException { Thread.sleep(3000); return "testA"; }
- 然后开启Sentinel控制台,在里面给/testa资源添加降级策略:
现在已配置的参数有:时间窗口2s,响应时间3s,响应阈值0.1s。 - 使用JMeter来测试,添加线程组使用Http请求测试,参数如下:
这样就会有100个请求分10s进行发送,也就是每秒会均匀请求10次。然后启动,在Sentinel控制台中记录下了这一波请求的实时监控每秒成功和拒绝的QPS。
3 过程分析
把上面图中的各个数据提取出表格:
第 | 1s | 2s | 3s | 4s | 5s | 6s | 7s | 8s | 9s | 10s | 11s |
---|---|---|---|---|---|---|---|---|---|---|---|
通过 | 4 | 10 | 10 | 10 | 1 | 0 | 4 | 0 | 3 | 7 | 0 |
拒绝 | 0 | 0 | 0 | 0 | 9 | 10 | 6 | 10 | 7 | 3 | 6 |
每一列都是1s的时间段,并且每秒内都有10个请求,共100个请求。
绘制时间线段图进行分析。
绘图注释
- 每秒线段上均匀分布10个请求
- 每个请求在每0.1s的开始发送
- 每一段表示一秒
- 图中表格每一秒和线段每一段位置对应
- 圆圈表示请求,里面的数字只用于标识,不是具体第几个请求
- 图中时间窗口(StatisticNode的rollingCounterInSecond)内的请求表示处理完成后会用于计算所在bucket的avgRt
从第一期请求t1开始,由于业务需要3s来处理,所以在第一个请求结束前的期间(t1 ~ t2),Sentinel统计的平均响应时间一直是0,所以会放行所有请求。也就能看到前4秒所有的请求都通过了:
t2时刻,第一个请求结束,因为时间窗口中的bucket是复用的,响应时间3s加入到了t2时刻时间窗口的一个bucket,请求5进入时计算平均响应时间(3s / 成功请求数(此刻是1))超过阈值,开始统计接下来的请求数passCount。
到请求8时passCount为4,仍然通过,也是第5s唯一通过的请求:
请求9进入后,passCount达到5,被拒绝。这时会熔断一个时间窗口的时长2s,也可以看到在第6s内的所有请求都被拒绝:
熔断结束后,这一时刻的时间窗口内有请求10~请求16,此时avgRt直接是超过阈值的。所以直接开始计算passCount,会通过4个请求(请求17~20),对应表格中的第7s:
在第二次熔断结束的时刻,此时时间窗口内没有完成请求的记录,因为最近的有可能被记录的请求17~20还没有处理完成,所以此时avgRt是0:
对于第二次熔断之后之后的请求21~26,avgRt一直为0所以也不会计算passCount,所以全部通过,第9s内通过请求21~23共三个,请求24~26在第10s。之后在两次熔断之间的请求17处理完成,此时导致avgRt超时
和之前一样,请求27~30会通过,并计算了passCount,之后的请求31(未画出)会被拒绝并开始熔断。所以第10s内通过的有请求24~30共7个:
4 源码简单分析
完整代码可以去idea双击shift搜索对应类自己逐层进入方法,断点调试查看,会更加清晰。
熔断降级从DegradeSlot类的entry方法开始,调用了checkDegrade方法:
for (DegradeRule rule : rules) {
if (!rule.passCheck(context, node, count)) {
throw new DegradeException(rule.getLimitApp(), rule);
}
}
这里面对每个规则检查,进入passCheck方法就是核心逻辑。这个方法返回true就是熔断状态(省略了其他两个熔断策略的代码):
在一些地方加入了输出内容的断点,在LeapArray的values方法也添加了输出断点打印当前时间窗口的两个bucket内容(格式:0: value=…, windowStart=…),不在这里展示了。
再启动一次Jmeter上文设置过的线程组,控制台打印如下,有兴趣的可以结合分析下(successCount > 0的会输出两遍相同的时间窗口内容):
23:49:05.234
0: value=p: 0, b: 0, w: 0, windowStart=23:49:05.000
1: value=p: 0, b: 0, w: 0, windowStart=23:48:31.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:05.336
0: value=p: 1, b: 0, w: 0, windowStart=23:49:05.000
1: value=p: 0, b: 0, w: 0, windowStart=23:48:31.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:05.434
0: value=p: 2, b: 0, w: 0, windowStart=23:49:05.000
1: value=p: 0, b: 0, w: 0, windowStart=23:48:31.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:05.554
0: value=p: 3, b: 0, w: 0, windowStart=23:49:05.000
1: value=p: 0, b: 0, w: 0, windowStart=23:49:05.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:05.636
0: value=p: 3, b: 0, w: 0, windowStart=23:49:05.000
1: value=p: 1, b: 0, w: 0, windowStart=23:49:05.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:05.734
0: value=p: 3, b: 0, w: 0, windowStart=23:49:05.000
1: value=p: 2, b: 0, w: 0, windowStart=23:49:05.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:05.836
0: value=p: 3, b: 0, w: 0, windowStart=23:49:05.000
1: value=p: 3, b: 0, w: 0, windowStart=23:49:05.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:05.936
0: value=p: 3, b: 0, w: 0, windowStart=23:49:05.000
1: value=p: 4, b: 0, w: 0, windowStart=23:49:05.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:06.035
0: value=p: 0, b: 0, w: 0, windowStart=23:49:06.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:05.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:06.137
0: value=p: 1, b: 0, w: 0, windowStart=23:49:06.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:05.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:06.235
0: value=p: 2, b: 0, w: 0, windowStart=23:49:06.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:05.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:06.336
0: value=p: 3, b: 0, w: 0, windowStart=23:49:06.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:05.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:06.435
0: value=p: 4, b: 0, w: 0, windowStart=23:49:06.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:05.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:06.534
0: value=p: 5, b: 0, w: 0, windowStart=23:49:06.000
1: value=p: 0, b: 0, w: 0, windowStart=23:49:06.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:06.634
0: value=p: 5, b: 0, w: 0, windowStart=23:49:06.000
1: value=p: 1, b: 0, w: 0, windowStart=23:49:06.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:06.735
0: value=p: 5, b: 0, w: 0, windowStart=23:49:06.000
1: value=p: 2, b: 0, w: 0, windowStart=23:49:06.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:06.835
0: value=p: 5, b: 0, w: 0, windowStart=23:49:06.000
1: value=p: 3, b: 0, w: 0, windowStart=23:49:06.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:06.935
0: value=p: 5, b: 0, w: 0, windowStart=23:49:06.000
1: value=p: 4, b: 0, w: 0, windowStart=23:49:06.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:07.036
0: value=p: 0, b: 0, w: 0, windowStart=23:49:07.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:06.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:07.135
0: value=p: 1, b: 0, w: 0, windowStart=23:49:07.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:06.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:07.236
0: value=p: 2, b: 0, w: 0, windowStart=23:49:07.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:06.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:07.336
0: value=p: 3, b: 0, w: 0, windowStart=23:49:07.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:06.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:07.434
0: value=p: 4, b: 0, w: 0, windowStart=23:49:07.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:06.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:07.533
0: value=p: 5, b: 0, w: 0, windowStart=23:49:07.000
1: value=p: 0, b: 0, w: 0, windowStart=23:49:07.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:07.634
0: value=p: 5, b: 0, w: 0, windowStart=23:49:07.000
1: value=p: 1, b: 0, w: 0, windowStart=23:49:07.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:07.735
0: value=p: 5, b: 0, w: 0, windowStart=23:49:07.000
1: value=p: 2, b: 0, w: 0, windowStart=23:49:07.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:07.834
0: value=p: 5, b: 0, w: 0, windowStart=23:49:07.000
1: value=p: 3, b: 0, w: 0, windowStart=23:49:07.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:07.935
0: value=p: 5, b: 0, w: 0, windowStart=23:49:07.000
1: value=p: 4, b: 0, w: 0, windowStart=23:49:07.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:08.038
0: value=p: 0, b: 0, w: 0, windowStart=23:49:08.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:07.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:08.136
0: value=p: 1, b: 0, w: 0, windowStart=23:49:08.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:07.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:08.234
0: value=p: 2, b: 0, w: 0, windowStart=23:49:08.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:07.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:08.335
0: value=p: 3, b: 0, w: 0, windowStart=23:49:08.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:07.500
successCount: 1
0: value=p: 3, b: 0, w: 0, windowStart=23:49:08.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:07.500
avgRt = 6073.0
passCount over RT: 1
23:49:08.434
0: value=p: 4, b: 0, w: 0, windowStart=23:49:08.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:07.500
successCount: 2
0: value=p: 4, b: 0, w: 0, windowStart=23:49:08.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:07.500
avgRt = 4551.0
passCount over RT: 2
23:49:08.536
0: value=p: 5, b: 0, w: 0, windowStart=23:49:08.000
1: value=p: 0, b: 0, w: 0, windowStart=23:49:08.500
successCount: 3
0: value=p: 5, b: 0, w: 0, windowStart=23:49:08.000
1: value=p: 0, b: 0, w: 0, windowStart=23:49:08.500
avgRt = 3034.0
passCount over RT: 3
23:49:08.636
0: value=p: 5, b: 0, w: 0, windowStart=23:49:08.000
1: value=p: 1, b: 0, w: 0, windowStart=23:49:08.500
successCount: 4
0: value=p: 5, b: 0, w: 0, windowStart=23:49:08.000
1: value=p: 1, b: 0, w: 0, windowStart=23:49:08.500
avgRt = 3036.25
passCount over RT: 4
23:49:08.733
0: value=p: 5, b: 0, w: 0, windowStart=23:49:08.000
1: value=p: 2, b: 0, w: 0, windowStart=23:49:08.500
successCount: 5
0: value=p: 5, b: 0, w: 0, windowStart=23:49:08.000
1: value=p: 2, b: 0, w: 0, windowStart=23:49:08.500
avgRt = 3036.6
Forbidden and cut!
23:49:08.835
------cut------
23:49:08.934
------cut------
23:49:09.035
------cut------
23:49:09.136
------cut------
23:49:09.235
------cut------
23:49:09.331
------cut------
23:49:09.434
------cut------
23:49:09.534
------cut------
23:49:09.635
------cut------
23:49:09.735
------cut------
23:49:09.836
------cut------
23:49:09.935
------cut------
23:49:10.034
------cut------
23:49:10.133
------cut------
23:49:10.233
------cut------
23:49:10.334
------cut------
23:49:10.435
------cut------
23:49:10.533
------cut------
23:49:10.634
------cut------
23:49:10.734
------cut------
23:49:10.835
0: value=p: 0, b: 5, w: 0, windowStart=23:49:10.000
1: value=p: 0, b: 3, w: 0, windowStart=23:49:10.500
successCount: 8
0: value=p: 0, b: 5, w: 0, windowStart=23:49:10.000
1: value=p: 0, b: 3, w: 0, windowStart=23:49:10.500
avgRt = 3028.125
passCount over RT: 1
23:49:10.936
0: value=p: 0, b: 5, w: 0, windowStart=23:49:10.000
1: value=p: 1, b: 3, w: 0, windowStart=23:49:10.500
successCount: 9
0: value=p: 0, b: 5, w: 0, windowStart=23:49:10.000
1: value=p: 1, b: 3, w: 0, windowStart=23:49:10.500
avgRt = 3365.0
passCount over RT: 2
23:49:11.036
0: value=p: 0, b: 0, w: 0, windowStart=23:49:11.000
1: value=p: 2, b: 3, w: 0, windowStart=23:49:10.500
successCount: 5
0: value=p: 0, b: 0, w: 0, windowStart=23:49:11.000
1: value=p: 2, b: 3, w: 0, windowStart=23:49:10.500
avgRt = 3633.4
passCount over RT: 3
23:49:11.137
0: value=p: 1, b: 0, w: 0, windowStart=23:49:11.000
1: value=p: 2, b: 3, w: 0, windowStart=23:49:10.500
successCount: 6
0: value=p: 1, b: 0, w: 0, windowStart=23:49:11.000
1: value=p: 2, b: 3, w: 0, windowStart=23:49:10.500
avgRt = 3532.6666666666665
passCount over RT: 4
23:49:11.234
0: value=p: 2, b: 0, w: 0, windowStart=23:49:11.000
1: value=p: 2, b: 3, w: 0, windowStart=23:49:10.500
successCount: 7
0: value=p: 2, b: 0, w: 0, windowStart=23:49:11.000
1: value=p: 2, b: 3, w: 0, windowStart=23:49:10.500
avgRt = 3460.285714285714
Forbidden and cut!
23:49:11.336
------cut------
23:49:11.435
------cut------
23:49:11.535
------cut------
23:49:11.634
------cut------
23:49:11.733
------cut------
23:49:11.834
------cut------
23:49:11.935
------cut------
23:49:12.035
------cut------
23:49:12.136
------cut------
23:49:12.234
------cut------
23:49:12.336
------cut------
23:49:12.436
------cut------
23:49:12.535
------cut------
23:49:12.634
------cut------
23:49:12.735
------cut------
23:49:12.835
------cut------
23:49:12.934
------cut------
23:49:13.033
------cut------
23:49:13.134
------cut------
23:49:13.234
------cut------
23:49:13.338
0: value=p: 0, b: 3, w: 0, windowStart=23:49:13.000
1: value=p: 0, b: 5, w: 0, windowStart=23:49:12.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:13.436
0: value=p: 1, b: 3, w: 0, windowStart=23:49:13.000
1: value=p: 0, b: 5, w: 0, windowStart=23:49:12.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:13.533
0: value=p: 2, b: 3, w: 0, windowStart=23:49:13.000
1: value=p: 0, b: 0, w: 0, windowStart=23:49:13.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:13.633
0: value=p: 2, b: 3, w: 0, windowStart=23:49:13.000
1: value=p: 1, b: 0, w: 0, windowStart=23:49:13.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:13.735
0: value=p: 2, b: 3, w: 0, windowStart=23:49:13.000
1: value=p: 2, b: 0, w: 0, windowStart=23:49:13.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:13.834
0: value=p: 2, b: 3, w: 0, windowStart=23:49:13.000
1: value=p: 3, b: 0, w: 0, windowStart=23:49:13.500
successCount: 0
avgRt = 0.0
avgRt < RT
23:49:13.935
0: value=p: 2, b: 3, w: 0, windowStart=23:49:13.000
1: value=p: 4, b: 0, w: 0, windowStart=23:49:13.500
successCount: 1
0: value=p: 2, b: 3, w: 0, windowStart=23:49:13.000
1: value=p: 4, b: 0, w: 0, windowStart=23:49:13.500
avgRt = 3035.0
passCount over RT: 1
23:49:14.034
0: value=p: 0, b: 0, w: 0, windowStart=23:49:14.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:13.500
successCount: 2
0: value=p: 0, b: 0, w: 0, windowStart=23:49:14.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:13.500
avgRt = 3036.0
passCount over RT: 2
23:49:14.134
0: value=p: 1, b: 0, w: 0, windowStart=23:49:14.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:13.500
successCount: 3
0: value=p: 1, b: 0, w: 0, windowStart=23:49:14.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:13.500
avgRt = 3037.0
passCount over RT: 3
23:49:14.234
0: value=p: 2, b: 0, w: 0, windowStart=23:49:14.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:13.500
successCount: 4
0: value=p: 2, b: 0, w: 0, windowStart=23:49:14.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:13.500
avgRt = 3037.25
passCount over RT: 4
23:49:14.334
0: value=p: 3, b: 0, w: 0, windowStart=23:49:14.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:13.500
successCount: 4
0: value=p: 3, b: 0, w: 0, windowStart=23:49:14.000
1: value=p: 5, b: 0, w: 0, windowStart=23:49:13.500
avgRt = 3037.25
Forbidden and cut!
23:49:14.435
------cut------
23:49:14.535
------cut------
23:49:14.634
------cut------
23:49:14.734
------cut------
23:49:14.834
------cut------
23:49:14.933
------cut------
23:49:15.034
------cut------
23:49:15.134
------cut------