阿里sentinel与springboot整合实践——根据request信息限流

阿里sentinel已适配springboot，整合非常简单，只需要添加一些配置就可以了，不需要写任何代码。但是如果有一些特殊需求，需要更细粒度的控制限流，就需要自己写些代码，比如对外的接口需要根据请求者ip，header中的user-agent或者其他信息限流，来防止恶意刷接口，或者爬虫，本人的项目就遇到了有人使用sqlmap工具来寻找sql注入漏洞和爬虫扫描接口，于是研究了一下sentinel，总结一下使用经验。

主要的原理就是，根据request中的信息设置请求来源，然后使用正则表达式来匹配来源限流，比如把来源设置成ip+user-agent，然后在限流规则的来源项中添加想要限流的正则表达式。

sentinel与springboot整合没有设置请求来源，所以需要根据自己需求设置，sentinel与来源的匹配只是使用简单的equals比较，这里使用正则表达式匹配更加灵活。

 

sentinel项目地址：https://github.com/alibaba/Sentinel

sentinel-dashboard控制台下载地址：https://github.com/alibaba/Sentinel/releases

sentinel-dashboard控制台使用：https://github.com/alibaba/Sentinel/tree/master/sentinel-dashboard

先按照文档启动控制台

1. 与springboot整合

在pom.xml中添加依赖：

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-alibaba-dependencies</artifactId>
            <version>0.2.1.RELEASE</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-alibaba-sentinel</artifactId>
    </dependency>
</dependencies>

在application.yml中添加sentinel相关配置：

spring:
  application:
    # 项目名称，sentinel-dashboard中会显示
    name: sentinel-example
  cloud:
    sentinel:
      transport:
        # 项目本地会启动一个服务的端口号，默认8719，用于与sentinel-dashboard通讯
        port: 8719
        # sentinel-dashboard服务地址
        dashboard: localhost:9090
      filter:
        # 需要进行限流监控的接口，多个匹配用逗号隔开
        url-patterns: /sentinel/*
      servlet:
        # 触发限流后重定向的页面
        block-page: /sentinel/block

配置好后，启动项目，这时候控制台还看不到东西，需要请求一下接口就能看到请求接口统计的情况，至此就可以使用sentinel的各种功能。

与springboot的整合实际上是使用了一个过滤器CommonFilter拦截所有请求，但是在获取entry的过程中没有传入参数，所以不能使用热点参数这个功能。

2. 设置请求来源，使用正则表达式匹配来源限流

只需要添加一个RequestOriginParser Bean，会自动设置到WebCellBackManager中。

@Configuration
public class SentinelConfig {

    /**
     * sentinel来源解析器
     * @return
     */
    @Bean
    public RequestOriginParser requestOriginParser() {
        return (request -> {
            String remoteAddr = RequestUtil.getIpAddr(request);
            String userAgent = request.getHeader("user-agent");
            return String.join("|",
                               (remoteAddr == null ? "" : remoteAddr),
                               (userAgent == null ? "" : userAgent));
        });
    }
}

CommonFilter在获取entry之前会使用RequestOriginParser从request中解析来源。

修改代码将来源匹配规则改成正则表达式匹配

需要重写FlowSlot和FlowRuleChecker两个类。虽然这两个类中各种final和私有方法，本意应该是不让改动这两个类的，但是通过研究源码，还是找到了办法，代码如下

新建类RegexOriginFlowSlot继承自FlowSlot

public class RegexOriginFlowSlot extends FlowSlot {

    @Override
    public void entry(Context context, ResourceWrapper resourceWrapper, DefaultNode node, int count,
                      boolean prioritized, Object... args) throws Throwable {
        checkFlow(resourceWrapper, context, node, count, prioritized);
        fireEntry(context, resourceWrapper, node, count, prioritized, args);
    }

    void checkFlow(ResourceWrapper resource, Context context, DefaultNode node, int count, boolean prioritized) throws BlockException {
        // Flow rule map cannot be null.
        Map<String, List<FlowRule>> flowRules = FlowRuleUtil.buildFlowRuleMap(FlowRuleManager.getRules());

        List<FlowRule> rules = flowRules.get(resource.getName());
        if (rules != null) {
            for (FlowRule rule : rules) {
                if (!canPassCheck(rule, context, node, count, prioritized)) {
                    throw new FlowException(rule.getLimitApp());
                }
            }
        }
    }

    boolean canPassCheck(FlowRule rule, Context context, DefaultNode node, int count, boolean prioritized) {
        // 这里使用自己修改过的FlowRuleChecker，自定义来源匹配
        return FlowRuleChecker.passCheck(rule, context, node, count, prioritized);
    }

新建类FlowRuleChecker，或者拷贝原来的类，只修改了selectNodeByRequesterAndStrategy()方法注释下面的代码，改成正则匹配

/**
 * 拷贝原FlowRuleChecker代码，把来源检查改成正则匹配
 */
final class FlowRuleChecker {

    static boolean passCheck(/*@NonNull*/ FlowRule rule, Context context, DefaultNode node, int acquireCount) {
        return passCheck(rule, context, node, acquireCount, false);
    }

    static boolean passCheck(/*@NonNull*/ FlowRule rule, Context context, DefaultNode node, int acquireCount,
                                          boolean prioritized) {
        String limitApp = rule.getLimitApp();
        if (limitApp == null) {
            return true;
        }

        if (rule.isClusterMode()) {
            return passClusterCheck(rule, context, node, acquireCount, prioritized);
        }

        return passLocalCheck(rule, context, node, acquireCount, prioritized);
    }

    private static boolean passLocalCheck(FlowRule rule, Context context, DefaultNode node, int acquireCount,
                                          boolean prioritized) {
        Node selectedNode = selectNodeByRequesterAndStrategy(rule, context, node);
        if (selectedNode == null) {
            return true;
        }

        return generateRater(rule).canPass(selectedNode, acquireCount);
    }

    private static TrafficShapingController generateRater(/*@Valid*/ FlowRule rule) {
        if (rule.getGrade() == RuleConstant.FLOW_GRADE_QPS) {
            switch (rule.getControlBehavior()) {
                case RuleConstant.CONTROL_BEHAVIOR_WARM_UP:
                    return new WarmUpController(rule.getCount(), rule.getWarmUpPeriodSec(), ColdFactorProperty.coldFactor);
                case RuleConstant.CONTROL_BEHAVIOR_RATE_LIMITER:
                    return new RateLimiterController(rule.getMaxQueueingTimeMs(), rule.getCount());
                case RuleConstant.CONTROL_BEHAVIOR_WARM_UP_RATE_LIMITER:
                    return new WarmUpRateLimiterController(rule.getCount(), rule.getWarmUpPeriodSec(),
                                                           rule.getMaxQueueingTimeMs(), ColdFactorProperty.coldFactor);
                case RuleConstant.CONTROL_BEHAVIOR_DEFAULT:
                default:
                    // Default mode or unknown mode: default traffic shaping controller (fast-reject).
            }
        }
        return new DefaultController(rule.getCount(), rule.getGrade());
    }

    static Node selectReferenceNode(FlowRule rule, Context context, DefaultNode node) {
        String refResource = rule.getRefResource();
        int strategy = rule.getStrategy();

        if (StringUtil.isEmpty(refResource)) {
            return null;
        }

        if (strategy == RuleConstant.STRATEGY_RELATE) {
            return ClusterBuilderSlot.getClusterNode(refResource);
        }

        if (strategy == RuleConstant.STRATEGY_CHAIN) {
            if (!refResource.equals(context.getName())) {
                return null;
            }
            return node;
        }
        // No node.
        return null;
    }

    private static boolean filterOrigin(String origin) {
        // Origin cannot be `default` or `other`.
        return !RuleConstant.LIMIT_APP_DEFAULT.equals(origin) && !RuleConstant.LIMIT_APP_OTHER.equals(origin);
    }

    static Node selectNodeByRequesterAndStrategy(/*@NonNull*/ FlowRule rule, Context context, DefaultNode node) {
        // The limit app should not be empty.
        String limitApp = rule.getLimitApp();
        int strategy = rule.getStrategy();
        String origin = context.getOrigin();
        // 把来源改成正则匹配，只改了这里的代码
        if (Pattern.compile(limitApp).matcher(origin).find() && filterOrigin(origin)) {
            if (strategy == RuleConstant.STRATEGY_DIRECT) {
                // Matches limit origin, return origin statistic node.
                return context.getOriginNode();
            }

            return selectReferenceNode(rule, context, node);
        } else if (RuleConstant.LIMIT_APP_DEFAULT.equals(limitApp)) {
            if (strategy == RuleConstant.STRATEGY_DIRECT) {
                // Return the cluster node.
                return node.getClusterNode();
            }

            return selectReferenceNode(rule, context, node);
        } else if (RuleConstant.LIMIT_APP_OTHER.equals(limitApp)
                   && FlowRuleManager.isOtherOrigin(origin, rule.getResource())) {
            if (strategy == RuleConstant.STRATEGY_DIRECT) {
                return context.getOriginNode();
            }

            return selectReferenceNode(rule, context, node);
        }

        return null;
    }

    private static boolean passClusterCheck(FlowRule rule, Context context, DefaultNode node, int acquireCount,
                                            boolean prioritized) {
        try {
            TokenService clusterService = pickClusterService();
            if (clusterService == null) {
                return fallbackToLocalOrPass(rule, context, node, acquireCount, prioritized);
            }
            long flowId = rule.getClusterConfig().getFlowId();
            TokenResult result = clusterService.requestToken(flowId, acquireCount, prioritized);
            return applyTokenResult(result, rule, context, node, acquireCount, prioritized);
            // If client is absent, then fallback to local mode.
        } catch (Throwable ex) {
            RecordLog.warn("[FlowRuleChecker] Request cluster token unexpected failed", ex);
        }
        // Fallback to local flow control when token client or server for this rule is not available.
        // If fallback is not enabled, then directly pass.
        return fallbackToLocalOrPass(rule, context, node, acquireCount, prioritized);
    }

    private static boolean fallbackToLocalOrPass(FlowRule rule, Context context, DefaultNode node, int acquireCount,
                                                 boolean prioritized) {
        if (rule.getClusterConfig().isFallbackToLocalWhenFail()) {
            return passLocalCheck(rule, context, node, acquireCount, prioritized);
        } else {
            // The rule won't be activated, just pass.
            return true;
        }
    }

    private static TokenService pickClusterService() {
        if (ClusterStateManager.isClient()) {
            return TokenClientProvider.getClient();
        }
        if (ClusterStateManager.isServer()) {
            return EmbeddedClusterTokenServerProvider.getServer();
        }
        return null;
    }

    private static boolean applyTokenResult(/*@NonNull*/ TokenResult result, FlowRule rule, Context context, DefaultNode node,
                                                         int acquireCount, boolean prioritized) {
        switch (result.getStatus()) {
            case TokenResultStatus.OK:
                return true;
            case TokenResultStatus.SHOULD_WAIT:
                // Wait for next tick.
                try {
                    Thread.sleep(result.getWaitInMs());
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
                return true;
            case TokenResultStatus.NO_RULE_EXISTS:
            case TokenResultStatus.BAD_REQUEST:
            case TokenResultStatus.FAIL:
                return fallbackToLocalOrPass(rule, context, node, acquireCount, prioritized);
            case TokenResultStatus.BLOCKED:
            default:
                return false;
        }
    }

    private FlowRuleChecker() {}

    static class ColdFactorProperty {
        public static volatile int coldFactor = 3;

        static {
            String strConfig = SentinelConfig.getConfig(SentinelConfig.COLD_FACTOR);
            if (StringUtil.isBlank(strConfig)) {
                coldFactor = 3;
            } else {
                try {
                    coldFactor = Integer.valueOf(strConfig);
                } catch (NumberFormatException e) {
                    RecordLog.info(e.getMessage(), e);
                }

                if (coldFactor <= 1) {
                    coldFactor = 3;
                    RecordLog.info("cold factor should be larger than 3");
                }
            }
        }
    }
}

类FlowSlot对应流量控制规则，调用流程如图

最终在FlowRuleChecker.selectNodeByRequesterAndStrategy()方法里匹配来源。

自定义SlotChainBuilder，把FlowSlot替换为RegexOriginFlowSlot

这样RegexOriginFlowSlot才能起作用

public class CustomSlotChainBuilder implements SlotChainBuilder {
    @Override
    public ProcessorSlotChain build() {
        ProcessorSlotChain chain = new DefaultProcessorSlotChain();
        chain.addLast(new NodeSelectorSlot());
        chain.addLast(new ClusterBuilderSlot());
        chain.addLast(new LogSlot());
        chain.addLast(new StatisticSlot());
        chain.addLast(new ParamFlowSlot());
        chain.addLast(new SystemSlot());
        chain.addLast(new AuthoritySlot());
        // 把原来的FlowSlot替换成RegexOriginFlowSlot()
        chain.addLast(new RegexOriginFlowSlot());
        chain.addLast(new DegradeSlot());
        return chain;
    }
}

注册CustomSlotChainBuilder

在项目的resources目录下新建文件:resources/META-INF/services/com.alibaba.csp.sentinel.slotchain.SlotChainBuilder，文件内容为CustomSlotChainBuilder的全类路径

com.example.springboot.web.sentinel.CustomSlotChainBuilder

sentinel使用的是jdk的ServiceLoader方式加载SlotChainBuilder的，实际只会加载除DefaultSlotChainBuilder之外的第一个SlotChainBuilder，sentinel本身还有一个HotParamSlotChainBuilder实现，所以最终是用到我们自己定义的CustomSlotChainBuilder还是HotParamSlotChainBuilder要看实际情况，最好在CustomSlotChainBuilder初始化时加点日志，看看有没有使用CustomSlotChainBuilder，必要的时候可以在pom.xml中把HotParamSlotChainBuilder排除，HotParamSlotChainBuilder所在的包是sentinel-parameter-flow-control。

这样CustomSlotChainBuilder就会起作用。

com.alibaba.csp.sentinel.slotchain.SlotChainProvider是加载SlotChainBuilder的类，下面是它加载SlotChainBuilder的源码

// jdk的ServiceLoader加载所有的SlotChainBuilder
private static final ServiceLoader<SlotChainBuilder> LOADER = ServiceLoader.load(SlotChainBuilder.class);

private static void resolveSlotChainBuilder() {
        List<SlotChainBuilder> list = new ArrayList<SlotChainBuilder>();
        boolean hasOther = false;
        for (SlotChainBuilder builder : LOADER) {
            if (builder.getClass() != DefaultSlotChainBuilder.class) {
                hasOther = true;
                list.add(builder);
            }
        }
        if (hasOther) {
            // 拿到所有的SlotChainBuilder后只取第一个
            builder = list.get(0);
        } else {
            // No custom builder, using default.
            // 否则使用默认的
            builder = new DefaultSlotChainBuilder();
        }

        RecordLog.info("[SlotChainProvider] Global slot chain builder resolved: "
            + builder.getClass().getCanonicalName());
    }

 

至此再启动项目，就可以在流量控制的来源下使用正则表达式匹配来源，本例中可以过控制特定的ip或者user-agent。

授权规则中的来源也使用正则匹配

同上面的修改原理一样，这里就只贴下代码

RegexOriginAuthoritySlot

public class RegexOriginAuthoritySlot extends AuthoritySlot {

    @Override
    public void entry(Context context, ResourceWrapper resourceWrapper, DefaultNode node, int count, boolean prioritized, Object... args)
            throws Throwable {
        checkBlackWhiteAuthority(resourceWrapper, context);
        fireEntry(context, resourceWrapper, node, count, prioritized, args);
    }
    void checkBlackWhiteAuthority(ResourceWrapper resource, Context context) throws AuthorityException {
        Map<String, Set<AuthorityRule>> authorityRules = loadAuthorityConf(AuthorityRuleManager.getRules());

        if (authorityRules == null) {
            return;
        }

        Set<AuthorityRule> rules = authorityRules.get(resource.getName());
        if (rules == null) {
            return;
        }

        for (AuthorityRule rule : rules) {
            if (!AuthorityRuleChecker.passCheck(rule, context)) {
                throw new AuthorityException(context.getOrigin());
            }
        }
    }

    private Map<String, Set<AuthorityRule>> loadAuthorityConf(List<AuthorityRule> list) {
        Map<String, Set<AuthorityRule>> newRuleMap = new ConcurrentHashMap<>();

        if (list == null || list.isEmpty()) {
            return newRuleMap;
        }

        for (AuthorityRule rule : list) {
            if (!isValidRule(rule)) {
                RecordLog.warn("[AuthorityRuleManager] Ignoring invalid authority rule when loading new rules: " + rule);
                continue;
            }

            if (StringUtil.isBlank(rule.getLimitApp())) {
                rule.setLimitApp(RuleConstant.LIMIT_APP_DEFAULT);
            }

            String identity = rule.getResource();
            Set<AuthorityRule> ruleSet = newRuleMap.get(identity);
            // putIfAbsent
            if (ruleSet == null) {
                ruleSet = new HashSet<>();
                ruleSet.add(rule);
                newRuleMap.put(identity, ruleSet);
            } else {
                // One resource should only have at most one authority rule, so just ignore redundant rules.
                RecordLog.warn("[AuthorityRuleManager] Ignoring redundant rule: " + rule.toString());
            }
        }

        return newRuleMap;
    }

    public static boolean isValidRule(AuthorityRule rule) {
        return rule != null && !StringUtil.isBlank(rule.getResource())
               && rule.getStrategy() >= 0 && StringUtil.isNotBlank(rule.getLimitApp());
    }


}

AuthorityRuleChecker

/**
 * 拷贝原AuthorityRuleChecker代码，把来源检查改成正则匹配
 */
final class AuthorityRuleChecker {

    static boolean passCheck(AuthorityRule rule, Context context) {
        String requester = context.getOrigin();

        // Empty origin or empty limitApp will pass.
        if (StringUtil.isEmpty(requester) || StringUtil.isEmpty(rule.getLimitApp())) {
            return true;
        }

        boolean contain = false;
        String[] patterns = rule.getLimitApp().split(",");
        for (String pattern : patterns) {
            // 把来源改成正则匹配
            if (Pattern.compile(pattern).matcher(requester).find()) {
                contain = true;
                break;
            }
        }

        int strategy = rule.getStrategy();
        if (strategy == RuleConstant.AUTHORITY_BLACK && contain) {
            return false;
        }

        if (strategy == RuleConstant.AUTHORITY_WHITE && !contain) {
            return false;
        }

        return true;
    }

    private AuthorityRuleChecker() {}
}

CustomSlotChainBuilder

public class CustomSlotChainBuilder implements SlotChainBuilder {
    @Override
    public ProcessorSlotChain build() {
        ProcessorSlotChain chain = new DefaultProcessorSlotChain();
        chain.addLast(new NodeSelectorSlot());
        chain.addLast(new ClusterBuilderSlot());
        chain.addLast(new LogSlot());
        chain.addLast(new StatisticSlot());
        chain.addLast(new ParamFlowSlot());
        chain.addLast(new SystemSlot());
        // 把原来的AuthoritySlot替换成RegexOriginAuthoritySlot()
        chain.addLast(new RegexOriginAuthoritySlot());
        // 把原来的FlowSlot替换成RegexOriginFlowSlot()
        chain.addLast(new RegexOriginFlowSlot());
        chain.addLast(new DegradeSlot());
        return chain;
    }
}

3. 处理路径中带参数的问题

路径中带参数，如/sentinel/hello/{param}这样的接口，sentinel会把不同的param值当作不同的接口来处理，这样会有性能问题。

需要使用前面用到的WebCallbackManager设置UrlCleaner，这个接口的参数仅仅是请求路径，要在这个接口中把/sentinel/hello/test，/sentinel/hello/test2，/sentinel/hello/test3这样带参数值的路径转换成/sentinel/hello/{param}。

要解决这个问题需要借助于springmvc，因为spring context中包含有/sentinel/hello/{param}这样的路径信息，在研究了springmvc查找HandlerMapping的代码后，找到了解决办法，代码如下：

@Configuration
public class SentinelConfig {

    /**
     * sentinel来源解析器
     * @return
     */
    @Bean
    public RequestOriginParser requestOriginParser() {
        return (request -> {
            String remoteAddr = RequestUtil.getIpAddr(request);
            String userAgent = request.getHeader("user-agent");
            return String.join("|",
                               (remoteAddr == null ? "" : remoteAddr),
                               (userAgent == null ? "" : userAgent));
        });
    }
    
    /**
     * sentinel路径处理器
     * @param requestMappingHandlerMapping 从RequestMappingHandlerMapping中获取springmvc定义的接口路径
     * @return
     */
    @Bean
    public UrlCleaner urlCleaner(RequestMappingHandlerMapping requestMappingHandlerMapping) {
        // 如果接口路径包含参数，如/sentinel/hello/{param}，则设置路径解析器，处理带参数的url
        // 参照spring匹配路径的源码
        // 获取spring context的所有requestmapping，包含所有请求接口
        // 获取所有接口路径，@GetMapping等注解中的路径
        Map<RequestMappingInfo, HandlerMethod> handlerMethods = requestMappingHandlerMapping.getHandlerMethods();

        // 筛选出包含参数的Mapping，避免与无参数的路径匹配
        List<RequestMappingInfo> filteredHandlerMethods = new ArrayList<>();
        Pattern pattern = Pattern.compile("\\{\\w+\\}");
        handlerMethods.forEach((key, value) -> {
            Set<String> patterns = key.getPatternsCondition().getPatterns();
            for (String path : patterns) {
                if (pattern.matcher(path).find()) {
                    filteredHandlerMethods.add(key);
                    break;
                }
            }
        });

        if (!CollectionUtils.isEmpty(filteredHandlerMethods)) {
            // 只处理包含参数的路径
            return (originUrl -> {

                // 逐个匹配路径
                for (RequestMappingInfo requestMappingInfo : filteredHandlerMethods) {
                    PatternsRequestCondition patternsCondition = requestMappingInfo.getPatternsCondition();
                    // 匹配路径
                    List<String> matchingPatterns = patternsCondition.getMatchingPatterns(originUrl);
                    if (matchingPatterns.size() > 0) {
                        if (matchingPatterns.size() > 1) {
                            matchingPatterns.sort(new AntPathMatcher().getPatternComparator(originUrl));
                        }

                        // 带参数的路径/sentinel/hello/abc，返回如/sentinel/hello/{param}
                        originUrl = matchingPatterns.get(0);
                        break;
                    }
                }
                return originUrl;
            });
        } else {
            // 没有路径带参数的接口则使用默认的
            return new DefaultUrlCleaner();
        }
    }
}