一、引言
在项目中使用博主自己封装的计算限流工具已经很久了,慢慢发展已经比较完善通用,现在开源共享。
github地址:
GitHub - SongTing0711/count-limit
相比较博主之前在java计算限流工具_tingmailang的博客-CSDN博客_限流工具封装的工具,开源的工具除了支持本地锁、本地存储之外还支持spring redis、redisson等进行加锁或者存储,注解配置更为灵活,可以看出工具演变的过程。
二、代码解析
1、注解
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface CountLimit {
String objectName();
String paramName() default "";
int limit() default 10000;
int waitTime() default 2;
//存储与加锁方式,默认本地存储本地锁
String countFactoryEnum() default CountLimitCommonUtil.LOCAL_LOCK_STORE;
}
2、注解配置加锁存储枚举
@Getter
public enum CountFactoryEnum {
LOCAL_LOCK_STORE("local_lock_store", "使用本地锁本地缓存"),
LOCAL_LOCK_REDISSON_STORE("local_lock_redisson_store", "使用本地锁redisson缓存"),
LOCAL_LOCK_SPRING_REDIS_STORE("local_lock_spring_redis_store", "使用本地锁spring redis缓存"),
REDISSON_LOCK_STORE("redisson_lock_store", "使用redisson锁本地缓存"),
REDISSON_LOCK_REDISSON_STORE("redisson_lock_redisson_store", "使用redisson锁redisson缓存"),
SPRING_REDIS_LOCK_STORE("spring_redis_lock_store", "使用spring redis锁本地缓存"),
SPRING_REDIS_LOCK_SPRING_REDIS_STORE("spring_redis_lock_spring_redis_store", "使用spring redis锁spring redis缓存"),
;
private String key;
private String value;
CountFactoryEnum(String key, String value) {
this.key = key;
this.value = value;
}
@JsonCreator
public static CountFactoryEnum of(String key) {
Optional<CountFactoryEnum> systemTypeEnum = Arrays.stream(CountFactoryEnum.values())
.filter(c -> c.getKey().equals(key)).findFirst();
return systemTypeEnum.orElse(null);
}
}
3、工具类
/**
* @description: 计算限流
* @author: tingmailang
*/
@Data
public class CountLimitDTO {
private String Key;
private Integer count;
private Integer limit;
private Boolean isAdd;
}
/**
* 共用类
*/
public class CountLimitCommonUtil {
public static final String COUNT_LIMIT_LOCK = "COUNT_LIMIT_LOCK:";
public static final String COUNT_LIMIT_REDIS_NODE_LOCK = "COUNT_LIMIT_REDIS_NODE_LOCK:";
public static final String COUNT_LIMIT_REDIS_NODE_STORE = "COUNT_LIMIT_REDIS_NODE_STORE:";
public static final String COUNT_LIMIT_STORE = "COUNT_LIMIT_STORE:";
public static final String LOCAL_LOCK_STORE = "local_lock_store";
public static final String LOCAL_LOCK_REDISSON_STORE = "local_lock_redisson_store";
public static final String LOCAL_LOCK_SPRING_REDIS_STORE = "local_lock_spring_redis_store";
public static final String REDISSON_LOCK_STORE = "redisson_lock_store";
public static final String REDISSON_LOCK_REDISSON_STORE = "redisson_lock_redisson_store";
public static final String SPRING_REDIS_LOCK_STORE = "spring_redis_lock_store";
public static final String SPRING_REDIS_LOCK_SPRING_REDIS_STORE = "spring_redis_lock_spring_redis_store";
private static volatile String NODE_ID;
public static volatile int SPILT_SLEEP = 20;
public static final String GET = "get";
public static String getNodeId() {
return NODE_ID;
}
public static synchronized void setNodeId(String nodeId) {
NODE_ID = nodeId;
}
public static synchronized void setSpiltSleep(int spiltSleep) {
SPILT_SLEEP = spiltSleep;
}
public static Object getFieldValueByName(String fieldName, Object o) {
try {
String firstLetter = fieldName.substring(0, 1).toUpperCase();
String getter = GET + firstLetter + fieldName.substring(1);
Method method = o.getClass().getMethod(getter, new Class[]{});
Object value = method.invoke(o, new Object[]{});
return value;
} catch (Exception e) {
throw new CountLimitException("获取属性值失败", e);
}
}
}
4、工厂类
/**
* 计数限流 Facade
*
* @author tingmailang
* @date 2020/3/2
*/
public interface CountLimitFacade<T> extends MatchingBean<CountFactoryEnum> {
boolean process(T t);
}
@Component
public class CountLimitFacadeFactory<T> implements FactoryList<CountLimitFacade<T>, CountFactoryEnum> {
@Resource
private List<CountLimitFacade> countLimitFacades;
@Override
public CountLimitFacade<T> getBean(CountFactoryEnum factory) {
for (CountLimitFacade countLimitFacade : countLimitFacades) {
if (countLimitFacade.matching(factory)) {
return countLimitFacade;
}
}
throw new CountLimitException("找不到计数限流实现类");
}
}
public interface FactoryList<E extends MatchingBean<K>, K> {
E getBean(K factory);
}
public interface MatchingBean<T> {
Boolean matching(T factory);
}
5、工厂实现类
①本地锁本地存储
@Component
public class LocalLockLocalStore extends CountLimitCommonBusiness implements CountLimitFacade<CountLimitDTO> {
private static ReentrantLock lock = new ReentrantLock();
@Override
public Boolean matching(CountFactoryEnum factory) {
return Objects.equals(CountFactoryEnum.LOCAL_LOCK_STORE, factory);
}
@Override
public boolean process(CountLimitDTO countLimitDTO) {
if (countLimitDTO.getIsAdd()) {
return this.checkExceed(countLimitDTO.getKey(), countLimitDTO.getCount(), countLimitDTO.getLimit());
} else {
return this.reduce(countLimitDTO.getKey(), countLimitDTO.getCount());
}
}
/**
* 检查是否超出计算限制
*
* @param key
* @param count
* @param limit
* @return
*/
public boolean checkExceed(String key, int count, int limit) {
try {
if (lock.tryLock()) {
return super.localCheckExceed(key, count, limit);
} else {
return false;
}
} finally {
if (lock.isHeldByCurrentThread()) {
lock.unlock();
}
}
}
/**
* 减少目前在查询中的参数量级
*
* @param key
* @param count
*/
public boolean reduce(String key, int count) {
try {
if (lock.tryLock(10, TimeUnit.SECONDS)) {
return super.localReduce(key, count);
}
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
if (lock.isHeldByCurrentThread()) {
lock.unlock();
}
}
return false;
}
}
②本地锁redisson存储
@Component
public class LocalLockRedissonStore extends CountLimitCommonBusiness implements CountLimitFacade<CountLimitDTO> {
private static ReentrantLock lock = new ReentrantLock();
@Override
public Boolean matching(CountFactoryEnum factory) {
return Objects.equals(CountFactoryEnum.LOCAL_LOCK_REDISSON_STORE, factory);
}
@Override
public boolean process(CountLimitDTO countLimitDTO) {
if (countLimitDTO.getIsAdd()) {
return this.checkExceed(countLimitDTO.getKey(), countLimitDTO.getCount(), countLimitDTO.getLimit());
} else {
return this.reduce(countLimitDTO.getKey(), countLimitDTO.getCount());
}
}
/**
* 检查是否超出计算限制
*
* @param key
* @param count
* @param limit
* @return
*/
public boolean checkExceed(String key, int count, int limit) {
try {
if (lock.tryLock()) {
return super.redissonCheckExceed(key, count, limit);
} else {
return false;
}
} finally {
if (lock.isHeldByCurrentThread()) {
lock.unlock();
}
}
}
/**
* 减少目前在查询中的参数量级
*
* @param key
* @param count
*/
public boolean reduce(String key, int count) {
try {
if (lock.tryLock(10, TimeUnit.SECONDS)) {
return super.redissonReduce(key, count);
}
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
if (lock.isHeldByCurrentThread()) {
lock.unlock();
}
}
return false;
}
}
③本地锁spring redis存储
@Component
public class LocalLockSpringRedisStore extends CountLimitCommonBusiness implements CountLimitFacade<CountLimitDTO> {
private static ReentrantLock lock = new ReentrantLock();
@Override
public Boolean matching(CountFactoryEnum factory) {
return Objects.equals(CountFactoryEnum.LOCAL_LOCK_SPRING_REDIS_STORE, factory);
}
@Override
public boolean process(CountLimitDTO countLimitDTO) {
if (countLimitDTO.getIsAdd()) {
return this.checkExceed(countLimitDTO.getKey(), countLimitDTO.getCount(), countLimitDTO.getLimit());
} else {
return this.reduce(countLimitDTO.getKey(), countLimitDTO.getCount());
}
}
/**
* 检查是否超出计算限制
*
* @param key
* @param count
* @param limit
* @return
*/
public boolean checkExceed(String key, int count, int limit) {
try {
if (lock.tryLock()) {
return super.springRedisCheckExceed(key, count, limit);
} else {
return false;
}
} finally {
if (lock.isHeldByCurrentThread()) {
lock.unlock();
}
}
}
/**
* 减少目前在查询中的参数量级
*
* @param key
* @param count
*/
public boolean reduce(String key, int count) {
try {
if (lock.tryLock(10, TimeUnit.SECONDS)) {
return super.localReduce(key, count);
}
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
if (lock.isHeldByCurrentThread()) {
lock.unlock();
}
}
return false;
}
}
④redisson锁本地存储
这里使用了博主封装的分布式代理锁进行并发控制,后续使用分布式锁的地方都可以看到一个注解搞定加锁开源工具distributed-proxy-lock(分布式代理锁)_tingmailang的博客-CSDN博客
@Component
public class RedissonLockLocalStore extends CountLimitCommonBusiness implements CountLimitFacade<CountLimitDTO> {
@Override
public Boolean matching(CountFactoryEnum factory) {
return Objects.equals(CountFactoryEnum.REDISSON_LOCK_STORE, factory);
}
@Override
@DistributedProxyLock(key = CountLimitCommonUtil.COUNT_LIMIT_LOCK,
suffixKeyTypeEnum = DistributedProxyLockCommonUtil.PARAM,
objectName = "countLimitDTO",
paramName = "LockKey")
public boolean process(CountLimitDTO countLimitDTO) {
if (countLimitDTO.getIsAdd()) {
return super.localCheckExceed(countLimitDTO.getKey(), countLimitDTO.getCount(), countLimitDTO.getLimit());
} else {
return super.localReduce(countLimitDTO.getKey(), countLimitDTO.getCount());
}
}
}
⑤redisson锁redisson存储
@Component
public class RedissonLockRedissonStore extends CountLimitCommonBusiness implements CountLimitFacade<CountLimitDTO> {
@Override
public Boolean matching(CountFactoryEnum factory) {
return Objects.equals(CountFactoryEnum.REDISSON_LOCK_REDISSON_STORE, factory);
}
@Override
@DistributedProxyLock(key = CountLimitCommonUtil.COUNT_LIMIT_LOCK,
suffixKeyTypeEnum = DistributedProxyLockCommonUtil.PARAM,
objectName = "countLimitDTO",
paramName = "LockKey")
public boolean process(CountLimitDTO countLimitDTO) {
if (countLimitDTO.getIsAdd()) {
return super.redissonCheckExceed(countLimitDTO.getKey(), countLimitDTO.getCount(), countLimitDTO.getLimit());
} else {
return super.redissonReduce(countLimitDTO.getKey(), countLimitDTO.getCount());
}
}
}
⑥SpringRedis锁本地存储
@Component
public class SpringRedisLockLocalStore extends CountLimitCommonBusiness implements CountLimitFacade<CountLimitDTO> {
@Override
public Boolean matching(CountFactoryEnum factory) {
return Objects.equals(CountFactoryEnum.SPRING_REDIS_LOCK_STORE, factory);
}
@Override
@DistributedProxyLock(key = CountLimitCommonUtil.COUNT_LIMIT_LOCK,
suffixKeyTypeEnum = DistributedProxyLockCommonUtil.PARAM,
objectName = "countLimitDTO",
paramName = "LockKey",
lockConnectionEnum = DistributedProxyLockCommonUtil.SPRING_REDIS)
public boolean process(CountLimitDTO countLimitDTO) {
if (countLimitDTO.getIsAdd()) {
return super.localCheckExceed(countLimitDTO.getKey(), countLimitDTO.getCount(), countLimitDTO.getLimit());
} else {
return super.localReduce(countLimitDTO.getKey(), countLimitDTO.getCount());
}
}
}
⑦SpringRedis锁SpringRedis存储
@Component
public class SpringRedisLockSpringRedisStore extends CountLimitCommonBusiness implements CountLimitFacade<CountLimitDTO> {
@Override
public Boolean matching(CountFactoryEnum factory) {
return Objects.equals(CountFactoryEnum.SPRING_REDIS_LOCK_SPRING_REDIS_STORE, factory);
}
@Override
@DistributedProxyLock(key = CountLimitCommonUtil.COUNT_LIMIT_LOCK,
suffixKeyTypeEnum = DistributedProxyLockCommonUtil.PARAM,
objectName = "countLimitDTO",
paramName = "LockKey",
lockConnectionEnum = DistributedProxyLockCommonUtil.SPRING_REDIS)
public boolean process(CountLimitDTO countLimitDTO) {
if (countLimitDTO.getIsAdd()) {
return super.springRedisCheckExceed(countLimitDTO.getKey(), countLimitDTO.getCount(), countLimitDTO.getLimit());
} else {
return super.springRedisReduce(countLimitDTO.getKey(), countLimitDTO.getCount());
}
}
}
以上工具都继承了父类CountLimitCommonBusiness,下面来看一下父类中做了什么
@Service
@Slf4j
public class CountLimitCommonBusiness {
@Resource
private RedissonClient redissonClient;
@Resource
private RedisTemplate redisTemplate;
@Resource
private CountLimitNode countLimitNode;
private static volatile ConcurrentHashMap<String, Integer> countMap = new ConcurrentHashMap<>();
/**
* redisson检查是否超出计算限制
*
* @param key
* @param count
* @param limit
* @return
*/
public boolean redissonCheckExceed(String key, int count, int limit) {
this.checkNode(LockConnectionEnum.REDISSON);
RBucket<Integer> bucket = redissonClient.getBucket(CountLimitCommonUtil.COUNT_LIMIT_STORE + CountLimitCommonUtil.getNodeId() + key);
int now = 0;
if (bucket.isExists()) {
now = bucket.get();
}
if (now + count > limit) {
return false;
} else {
log.debug("CountLimitAspect:{}增加计算量:{}", key, now + count);
bucket.set(now + count);
return true;
}
}
/**
* redisson减少目前在查询中的参数量级
*
* @param key
* @param count
*/
public boolean redissonReduce(String key, int count) {
this.checkNode(LockConnectionEnum.REDISSON);
RBucket<Integer> bucket = redissonClient.getBucket(CountLimitCommonUtil.COUNT_LIMIT_STORE + CountLimitCommonUtil.getNodeId() + key);
int now = bucket.get();
log.debug("CountLimitAspect:{}减少计算量:{}", key, now - count);
bucket.set(now - count);
return true;
}
/**
* local检查是否超出计算限制
*
* @param key
* @param count
* @param limit
* @return
*/
public boolean localCheckExceed(String key, int count, int limit) {
int now = countMap.getOrDefault(key, 0);
if (now + count > limit) {
return false;
} else {
log.debug("CountLimitAspect:{}增加计算量:{}", key, now + count);
countMap.put(key, now + count);
return true;
}
}
/**
* local减少目前在查询中的参数量级
*
* @param key
* @param count
*/
public boolean localReduce(String key, int count) {
int now = countMap.get(key);
log.debug("CountLimitAspect:{}减少计算量:{}", key, now - count);
countMap.put(key, now - count);
return true;
}
/**
* springRedis检查是否超出计算限制
*
* @param key
* @param count
* @param limit
* @return
*/
public boolean springRedisCheckExceed(String key, int count, int limit) {
this.checkNode(LockConnectionEnum.SPRING_REDIS);
Integer now = (Integer) redisTemplate.opsForValue().get(CountLimitCommonUtil.COUNT_LIMIT_STORE + CountLimitCommonUtil.getNodeId() + key);
if (now == null) {
now = 0;
}
if (now + count > limit) {
return false;
} else {
log.debug("CountLimitAspect:{}增加计算量:{}", key, now + count);
redisTemplate.opsForValue().set(CountLimitCommonUtil.COUNT_LIMIT_STORE + CountLimitCommonUtil.getNodeId() + key, now + count);
return true;
}
}
/**
* springRedis减少目前在查询中的参数量级
*
* @param key
* @param count
*/
public boolean springRedisReduce(String key, int count) {
this.checkNode(LockConnectionEnum.SPRING_REDIS);
Integer now = (Integer) redisTemplate.opsForValue().get(CountLimitCommonUtil.COUNT_LIMIT_STORE + CountLimitCommonUtil.getNodeId() + key);
if (now == null) {
now = 0;
}
log.debug("CountLimitAspect:{}减少计算量:{}", key, now - count);
redisTemplate.opsForValue().set(CountLimitCommonUtil.COUNT_LIMIT_STORE + CountLimitCommonUtil.getNodeId() + key, now - count);
return true;
}
/**
* 检查节点id,如果为空就设置
*
* @param lockConnectionEnum
*/
public void checkNode(LockConnectionEnum lockConnectionEnum) {
if (CountLimitCommonUtil.getNodeId() != null) {
return;
}
countLimitNode.setNode(lockConnectionEnum);
}
}
可以看出父类封装了共通的操作redis和本地缓存的方法,同时在使用redis的同时考虑了分布式唯一性,在CountLimitCommonUtil中NODE_ID是可以设置的,可以由使用者自己进行赋值操作,volatile保证可见性。
如果使用者不进行赋值操作也没有问题,工具自动在CountLimitNode中实现了分布式节点的唯一id,保证节点存储的独立性。
@Service
public class CountLimitNode {
@Resource
private RedissonClient redissonClient;
@Resource
private RedisTemplate redisTemplate;
@DistributedProxyLock(key = CountLimitCommonUtil.COUNT_LIMIT_REDIS_NODE_LOCK)
public void setNode(LockConnectionEnum lockConnectionEnum) {
if (CountLimitCommonUtil.getNodeId() != null) {
return;
}
switch (lockConnectionEnum) {
case SPRING_REDIS:
Integer temp = (Integer) redisTemplate.opsForValue().get(CountLimitCommonUtil.COUNT_LIMIT_REDIS_NODE_STORE);
if (temp == null) {
temp = 0;
}
temp++;
redisTemplate.opsForValue().set(CountLimitCommonUtil.COUNT_LIMIT_REDIS_NODE_STORE, temp);
CountLimitCommonUtil.setNodeId(temp.toString());
break;
case REDISSON:
RBucket<Integer> bucket = redissonClient.getBucket(CountLimitCommonUtil.COUNT_LIMIT_REDIS_NODE_STORE);
Integer now = 0;
if (bucket.isExists()) {
now = bucket.get() + 1;
}
bucket.set(now);
CountLimitCommonUtil.setNodeId(now.toString());
break;
}
}
}
6、切面
@Aspect
@Component
public class CountLimitAspect<T> {
@Resource
private CountLimitFacadeFactory<CountLimitDTO> countLimitFacadeFactory;
@Pointcut("@annotation(com.annotation.CountLimit)")
public void lockPointCut() {
}
@Around("lockPointCut() && @annotation(countLimit)")
public Object around(ProceedingJoinPoint joinPoint, CountLimit countLimit) throws Throwable {
LocalDateTime start = LocalDateTime.now();
String inter = joinPoint.getTarget().getClass().getName() + joinPoint.getSignature().getName();
String objectName = countLimit.objectName();
Object[] args = joinPoint.getArgs();
String[] paramNames = ((CodeSignature) joinPoint.getSignature()).getParameterNames();
Map<String, Object> param = new HashMap<>();
String par = countLimit.paramName();
for (int i = 0; i < paramNames.length; i++) {
param.put(paramNames[i], args[i]);
}
CountFactoryEnum countFactoryEnum = CountFactoryEnum.of(countLimit.countFactoryEnum());
//获取限制的参数
List<T> queryPar;
String key = inter + objectName;
int count;
CountLimitDTO countLimitDTO = new CountLimitDTO();
try {
if (StringUtil.isBlank(par)) {
//如果没有设置参数,说明在入参对象中
queryPar = (List<T>) param.get(objectName);
} else {
//说明在入参的某个对象中,有一个参数是进行限流
Object o = param.get(objectName);
queryPar = (List<T>) CountLimitCommonUtil.getFieldValueByName(par, o);
key += par;
}
count = queryPar.size();
countLimitDTO.setKey(key);
countLimitDTO.setCount(count);
countLimitDTO.setLimit(countLimit.limit());
countLimitDTO.setIsAdd(Boolean.TRUE);
while (!countLimitFacadeFactory.getBean(countFactoryEnum).process(countLimitDTO)) {
//是否超出等待时间
if (start.plusSeconds(countLimit.waitTime()).isBefore(LocalDateTime.now())) {
throw new CountLimitException("超出等待时间" + key);
}
//将等待时长划分为等份时长
Thread.sleep(countLimit.waitTime() * 1000 / CountLimitCommonUtil.SPILT_SLEEP);
}
} catch (Exception e) {
throw new CountLimitException("计算限流异常", e);
}
try {
return joinPoint.proceed();
} finally {
countLimitDTO.setIsAdd(Boolean.FALSE);
boolean success = countLimitFacadeFactory.getBean(countFactoryEnum).process(countLimitDTO);
if (!success) {
throw new CountLimitException("计算量减少失败" + countLimitDTO.toString());
}
}
}
}
三、使用
1、计算量的存储与并发控制主要分为redis和本地两种,目前支持链接redis的工具主要是redisson、spring redis
2、因此有以下七种使用方式,默认使用ReentrantLock加锁,本地map缓存
3、如果使用工具的地方很多,存储比较适合使用redis
4、如果限流的方法qps很高,使用redis进行加锁处理可能是比ReentrantLock更好的选择
1、ReentrantLock加锁,本地map缓存
@CountLimit(objectName = "request",
paramName = "shopIdList",
limit = 20000)
public void audit(ShopOnlineDTO request) {
//业务处理
}
2、ReentrantLock加锁,redisson缓存
@CountLimit(objectName = "request",
paramName = "shopIdList",
limit = 20000,
countFactoryEnum = CountLimitCommonUtil.LOCAL_LOCK_REDISSON_STORE)
public void audit(ShopOnlineDTO request) {
//业务处理
}
3、ReentrantLock加锁,spring redis缓存
@CountLimit(objectName = "request",
paramName = "shopIdList",
limit = 20000,
countFactoryEnum = CountLimitCommonUtil.LOCAL_LOCK_SPRING_REDIS_STORE)
public void audit(ShopOnlineDTO request) {
//业务处理
}
4、redisson加锁,本地map缓存
@CountLimit(objectName = "request",
paramName = "shopIdList",
limit = 20000,
countFactoryEnum = CountLimitCommonUtil.REDISSON_LOCK_STORE)
public void audit(ShopOnlineDTO request) {
//业务处理
}
5、redisson加锁,redisson缓存
@CountLimit(objectName = "request",
paramName = "shopIdList",
limit = 20000,
countFactoryEnum = CountLimitCommonUtil.REDISSON_LOCK_REDISSON_STORE)
public void audit(ShopOnlineDTO request) {
//业务处理
}
6、spring redis加锁,本地map缓存
@CountLimit(objectName = "request",
paramName = "shopIdList",
limit = 20000,
countFactoryEnum = CountLimitCommonUtil.SPRING_REDIS_LOCK_STORE)
public void audit(ShopOnlineDTO request) {
//业务处理
}
7、spring redis加锁,spring redis缓存
@CountLimit(objectName = "request",
paramName = "shopIdList",
limit = 20000,
countFactoryEnum = CountLimitCommonUtil.SPRING_REDIS_LOCK_SPRING_REDIS_STORE)
public void audit(ShopOnlineDTO request) {
//业务处理
}
四、总结
新生的工具总是可以做到更多事情的,希望有兴趣的同学发挥想象力提出issue或者加入到工具开源的扩展中。