概要
- 本方案是由于工作中出现的系统数据库某些表数据量大(几千万)、关联表多等业务情况,导致接口响应速度过长,同时业务接口数量也过多,故而需要个方案对部分旧接口进行改造,所以自己便在工作之余设想的一个较为通用的接口缓存方案
- 本方案主要采用了注解的方式,利用Spring的Aop特性,通过把接口放回数据更新进Redis中,实现接口数据的缓存
- 同时采用数据库(案例是mysql),对接口的类名、方法、入参、更新cron等进行存储,采用反序列化的方式来实现接口的调用,并通过SpringTask对不同的接口进行不同频率的定时更新
- 采用前端Headers添加isActive入参的方式(试过其他的方式但需要适配),判断接口是否是接口请求方式调用,当是接口请求方式调用时会更新表字段last_active_time,根据该时间可以对长期没被主动调用过的接口进行停止缓存更新,一定程度上防止系统的内存占用越来越大
导入依赖:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
<version>3.0.4</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
<dependency>
<groupId>com.baomidou</groupId>
<artifactId>mybatis-plus-boot-starter</artifactId>
<version>3.5.3.1</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>31.1-jre</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>2.0.32</version>
</dependency>
类介绍
- RedisCache: 自定义接口缓存注解
@Retention(RetentionPolicy.RUNTIME) @Target(ElementType.METHOD) public @interface RedisCache { /** * 超时时间 * */ int timeout() default 61*3; /** * 超时时间单位 * */ TimeUnit timeUnit() default TimeUnit.SECONDS; /** * 自定义cron * */ String cron() default "*/3 * * * * *"; }
- RedisCacheAspect: 自定义接口缓存注解RedisCache的切面
/** * @author yuqingxin * 自定义接口缓存注解RedisCache的切面 * */ @Aspect @Component @Log4j2 public class RedisCacheAspect { final RedisCacheService redisCacheService; final Executor threadPool; final ApiActiveService apiActiveService; final ApiScheduleMapper apiScheduleMapper; public RedisCacheAspect(RedisCacheService redisCacheService, Executor threadPool, ApiActiveService apiActiveService, ApiScheduleMapper apiScheduleMapper){ this.redisCacheService = redisCacheService; this.threadPool = threadPool; this.apiActiveService = apiActiveService; this.apiScheduleMapper = apiScheduleMapper; } private final ObjectMapper objectMapper = new ObjectMapper(); @Around("@annotation(redisCache)") public Object getControllerData(ProceedingJoinPoint joinPoint, RedisCache redisCache) throws Throwable { //采用前端Header入参方式判断调用该方法的方式 ServletRequestAttributes attributes = (ServletRequestAttributes) RequestContextHolder.getRequestAttributes(); String isActive = null; if (attributes != null) { HttpServletRequest request = attributes.getRequest(); isActive = request.getHeader("isActive"); } //获取方法的相关信息 String className = joinPoint.getTarget().getClass().getName(); String arguments = objectMapper.writeValueAsString(joinPoint.getArgs()); Method method = ((MethodSignature) joinPoint.getSignature()).getMethod(); String methodName = method.getName(); String argumentsType = Arrays.toString(method.getParameterTypes()); // 构建缓存键:类名 + 方法 + 参数值 String key = className + ":" + methodName + ":" + arguments; String result = redisCacheService.get(key); log.info("RedisCache:{}",result); if (result == null) { //调用接口逻辑并获取对应的返回数据并转换为字符串 result = objectMapper.writeValueAsString(joinPoint.proceed()); log.info("set RedisCache:{} :: {}",key,result); redisCacheService.set(key, result, redisCache.timeout(),redisCache.timeUnit()); } //定时调用->更新redis数据 if (null == isActive){ threadPool.execute(()-> { try { String updateResult = objectMapper.writeValueAsString(joinPoint.proceed()); log.info("Set RedisCache:{} :: {}", key, updateResult); redisCacheService.set(key, updateResult, redisCache.timeout(), redisCache.timeUnit()); } catch (Throwable e) { log.error("Set RedisCache Error:{}", key); e.printStackTrace(); } }); } //把记录更新到api缓存表 String finalIsActive = isActive; threadPool.execute(()->{ LocalDateTime now = LocalDateTime.now(); LambdaQueryWrapper<ApiScheduleEntity> asL = new LambdaQueryWrapper<>(); asL.eq(ApiScheduleEntity::getClassName,className) .eq(ApiScheduleEntity::getMethodName,method.getName()) .eq(ApiScheduleEntity::getArgumentsType,argumentsType) .eq(ApiScheduleEntity::getArguments,arguments); ApiScheduleEntity apiSchedule = apiScheduleMapper.selectOne(asL); if (apiSchedule != null){ apiSchedule.setUpdateAt(now); //主动调用 则 更新 主动调用时间 if ("1".equals(finalIsActive)){ apiSchedule.setLastActiveTime(now); }if (1 != apiSchedule.getStatus()){ apiSchedule.setStatus(1); //添加对应的定时任务 log.info("addApiScheduledTask:{}"+key); apiActiveService.addApiScheduledTask(apiSchedule); } apiScheduleMapper.updateById(apiSchedule); }else { ApiScheduleEntity apiScheduleEntity = new ApiScheduleEntity(); apiScheduleEntity.setClassName(className); apiScheduleEntity.setMethodName(method.getName()); apiScheduleEntity.setArgumentsType(argumentsType); apiScheduleEntity.setArguments(arguments); apiScheduleEntity.setCron(redisCache.cron()); apiScheduleEntity.setLastActiveTime(now); apiScheduleEntity.setUpdateAt(now); apiScheduleMapper.insert(apiScheduleEntity); //添加对应的定时任务 log.info("addApiScheduledTask:{}"+key); apiActiveService.addApiScheduledTask(apiScheduleEntity); } }); //把获取到的字符串转化为接口的返回类型 return objectMapper.readValue(result,method.getReturnType()); } }
- RedisCacheService: RedisTemplate<String,String>的部分简单封装
@Service public class RedisCacheService { private final RedisTemplate<String,String> redisTemplate; public RedisCacheService(RedisTemplate<String,String> redisTemplate){ this.redisTemplate = redisTemplate; } public String get(String key){ return redisTemplate.opsForValue().get(key); } public void set(String key, String value){ redisTemplate.opsForValue().set(key,value,60, TimeUnit.SECONDS); } public void set(String key, String value, long seconds){ redisTemplate.opsForValue().set(key,value,seconds, TimeUnit.SECONDS); } public void set(String key, String value, long seconds, TimeUnit timeUnit){ redisTemplate.opsForValue().set(key,value,seconds, timeUnit); } }
- RedisScheduleConfig:自定义定时任务配置,主要作用为在服务启动时添加添加需要缓存的接口的定时任务
/** * @author yuqingxin * */ @Log4j2 @Configuration @EnableScheduling public class RedisScheduleConfig implements SchedulingConfigurer { final Executor threadPool; final ApplicationContext applicationContext; final ApiActiveService apiActiveService; final ThreadPoolTaskScheduler threadPoolTaskScheduler; final Map<String,ScheduledFuture<?>> scheduledFutureMap; public RedisScheduleConfig(Executor threadPool, ApplicationContext applicationContext, ApiActiveService apiActiveService, ThreadPoolTaskScheduler threadPoolTaskScheduler, Map<String,ScheduledFuture<?>> scheduledFutureMap){ this.threadPool = threadPool; this.applicationContext = applicationContext; this.apiActiveService = apiActiveService; this.threadPoolTaskScheduler = threadPoolTaskScheduler; this.scheduledFutureMap = scheduledFutureMap; } /** * 程序启动时添加定时任务 * 如果是分布式服务,一般只用一台服务器更新接口缓存就行 * 可以简单通过设置服务器/容器的环境变量,使用@Value获取变量值来判断该服务器/容器是否是启用更新接口缓存 * */ @Override public void configureTasks(@Nonnull ScheduledTaskRegistrar scheduledTaskRegistrar) { //查询所有需要更新缓存的接口的定时任务 log.info("初始化需缓存接口的定时任务"); List<ApiScheduleEntity> apiScheduleList = apiActiveService.getApiScheduleList(); for (ApiScheduleEntity apiSchedule : apiScheduleList) { try { apiActiveService.addApiScheduledTask(apiSchedule); }catch (Exception e){ log.error("Init Active Api Error:{}",apiSchedule); } } } }
- ApplicationClosedListenerConfig:自定义监听服务关闭配置,主要目的为在服务关闭时先关闭正在运行的定时任务
@Configuration public class ApplicationClosedListenerConfig implements ApplicationListener<ContextClosedEvent> { final ThreadPoolTaskScheduler threadPoolTaskScheduler; public ApplicationClosedListenerConfig(ThreadPoolTaskScheduler threadPoolTaskScheduler){ this.threadPoolTaskScheduler = threadPoolTaskScheduler; } @Override public void onApplicationEvent(@Nonnull ContextClosedEvent contextClosedEvent) { threadPoolTaskScheduler.shutdown(); } }
- ThreadPoolConfig: 线程池相关配置,包括线程池、线程池任务调度器、定时任务类缓存
@Log4j2 @Configuration public class ThreadPoolConfig { @Value("${thread-pool.core-pool-size}") private int corePoolSize; @Value("${thread-pool.max-pool-size}") private int maxPoolSize; @Value("${thread-pool.keep-alive-time}") private long keepAliveTime; @Bean public Executor threadPool() { log.info("创建自定义线程池"); return new ThreadPoolExecutor( corePoolSize, maxPoolSize, keepAliveTime, TimeUnit.SECONDS, new LinkedBlockingQueue<>(1000), new ThreadFactoryBuilder().setNameFormat("Thread-pool-%d").build(), new ThreadPoolExecutor.AbortPolicy() ); } @Bean public ThreadPoolTaskScheduler threadPoolTaskScheduler(){ log.info("创建自定义线程池任务调度器"); ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler(); threadPoolTaskScheduler.setPoolSize(20); threadPoolTaskScheduler.setThreadGroupName("TaskScheduler-"); threadPoolTaskScheduler.setAwaitTerminationSeconds(60); threadPoolTaskScheduler.setWaitForTasksToCompleteOnShutdown(true); return threadPoolTaskScheduler; } @Bean public Map<String, ScheduledFuture<?>> scheduledFutureMap(){ log.info("创建定时任务缓存集合"); return new HashMap<>(); } }
- ArgumentsUtil:实现参数更对应类的转换方法类
/** * @author yuqingxin * */ @Service public class ArgumentsUtil { final ObjectMapper objectMapper; public ArgumentsUtil(ObjectMapper objectMapper){ this.objectMapper = objectMapper; } /** * 参数类型转换为对应对象 * */ public Class<?>[] getClassArray(String typesString){ String[] classNames = typesString.substring(1, typesString.length() - 1).split("class "); Class<?>[] classes = new Class<?>[classNames.length-1]; for (int i = 1; i < classNames.length; i++) { try { Class<?> clazz = Class.forName(classNames[i].replace(", ","")); classes[i-1] = clazz; } catch (ClassNotFoundException e) { e.printStackTrace(); } } return classes; } /** * 参数值转换为对象 * */ public Object[] getObjectArray(String argumentsString,Class<?>[] parameterTypes) { Object[] objects = JSON.parseObject(argumentsString,Object[].class); for (int i = 0; i < objects.length; i++) { try { objects[i] = objectMapper.readValue(objectMapper.writeValueAsString(objects[i]),parameterTypes[i]); } catch (JsonProcessingException e) { throw new RuntimeException(e); } } return objects; } }
- ApiScheduleEntity: 接口缓存调用表对应的实体类
/** * @author yuqingxin * */ @Data @AllArgsConstructor @NoArgsConstructor @TableName("tbl_api_schedule") public class ApiScheduleEntity { private Long id; /** * 类名 * */ private String className; /** * 方法名 * */ private String methodName; /** * 参数类型 * */ private String argumentsType; /** * 参数值 * */ private String arguments; /** * 更新频率cron * */ private String cron; /** * 最后调用时间 * */ private LocalDateTime updateAt; /** * 最后通过接口调用方式的调用时间 * */ private LocalDateTime lastActiveTime; /** * 是否启用:1是0否 * */ private Integer status; }
- ApiActiveService: 调用缓存接口相关逻辑类,主要功能为:
- 获取需要调用/长时间为调用的缓存接口
- 通过反序列的方式对单个/多个需缓存的接口进行调用
- 对长期未使用的缓存接口取消对应的定时任务
/** * @author yuqingxin * */ @Log4j2 @Service public class ApiActiveService { //超时时间 private final Integer timeout = 60*60*24*3; final ApiScheduleMapper apiScheduleMapper; final ApplicationContext applicationContext; final ThreadPoolTaskScheduler threadPoolTaskScheduler; final Executor threadPool; final Map<String, ScheduledFuture<?>> scheduledFutureMap; final ArgumentsUtil argumentsUtil; public ApiActiveService(ApiScheduleMapper apiScheduleMapper, ApplicationContext applicationContext, ThreadPoolTaskScheduler threadPoolTaskScheduler, Executor threadPool, Map<String, ScheduledFuture<?>> scheduledFutureMap, ArgumentsUtil argumentsUtil){ this.apiScheduleMapper = apiScheduleMapper; this.applicationContext = applicationContext; this.threadPoolTaskScheduler = threadPoolTaskScheduler; this.threadPool = threadPool; this.scheduledFutureMap = scheduledFutureMap; this.argumentsUtil = argumentsUtil; } /** * 获取需要更新的Api * */ public List<ApiScheduleEntity> getApiScheduleList(){ LambdaQueryWrapper<ApiScheduleEntity> aL = new LambdaQueryWrapper<>(); aL.eq(ApiScheduleEntity::getStatus,1); return apiScheduleMapper.selectList(aL); } /** * 获取长时间未使用的Api * @param timeout 多长时间未使用 秒 * */ public List<ApiScheduleEntity> getLongTimeUnUseApiList(Integer timeout){ LambdaQueryWrapper<ApiScheduleEntity> aL = new LambdaQueryWrapper<>(); aL.eq(ApiScheduleEntity::getStatus,1) .le(ApiScheduleEntity::getLastActiveTime, LocalDateTime.now().minusSeconds(timeout)); return apiScheduleMapper.selectList(aL); } /** * 调用单个Api方法 * */ public void runApiOne(ApiScheduleEntity apiSchedule){ // 类名 String className = apiSchedule.getClassName(); // 方法名 String methodName = apiSchedule.getMethodName(); // 参数值 String argumentsString = apiSchedule.getArguments(); // 参数类型 String argumentsType = apiSchedule.getArgumentsType(); Class<?>[] parameterTypes = argumentsUtil.getClassArray(argumentsType); Object[] arguments = argumentsUtil.getObjectArray(argumentsString,parameterTypes); try { // 获取类对象 Class<?> apiClass = Class.forName(className); // 获取方法对象 Method method = apiClass.getMethod(methodName, parameterTypes); // 获取该类的Bean对象 Object instance = applicationContext.getBean(apiClass); // 调用方法 Object result = method.invoke(instance, arguments); // 处理方法返回值 log.info("Result: " + result); } catch (Exception e) { log.error("RunApiMethod Error: {}:{}::{}",className,methodName,argumentsString); e.printStackTrace(); } } /** * 批量调用需要更新缓存的Api * */ public Boolean runApiMethod(){ List<ApiScheduleEntity> list = getApiScheduleList(); for (ApiScheduleEntity api : list) { runApiOne(api); } return true; } /** * 添加单个Api定时任务 * */ public void addApiScheduledTask(ApiScheduleEntity apiSchedule){ TimerTask timerTask = new TimerTask() { @Override public void run() { runApiOne(apiSchedule); } }; ScheduledFuture<?> scheduledFuture = threadPoolTaskScheduler.schedule(timerTask,new CronTrigger(apiSchedule.getCron())); //将该实例存入到缓存map中 if (null != scheduledFuture){ threadPool.execute(()->{ String key = apiSchedule.getClassName() + ":" + apiSchedule.getMethodName()+ ":" +apiSchedule.getArguments(); scheduledFutureMap.put(key,scheduledFuture); }); } } /** * 批量取消长时间未被主动调用的Api定时更新任务 * */ public Boolean cancelApiScheduleTask(){ List<ApiScheduleEntity> list = getLongTimeUnUseApiList(timeout); for (ApiScheduleEntity apiSchedule : list) { threadPool.execute(()->{ //去除对应定时认为 String key = apiSchedule.getClassName() + ":" +apiSchedule.getMethodName() + ":" + apiSchedule.getArguments(); ScheduledFuture<?> scheduledFuture = scheduledFutureMap.get(key); if (null != scheduledFuture){ scheduledFuture.cancel(true); scheduledFutureMap.remove(key); } //更新数据库状态 apiSchedule.setStatus(0); apiScheduleMapper.updateById(apiSchedule); }); } return true; } }
调用示例
完整代码获取
注:如果在使用中出现不能适配的bug,麻烦方便的话请告知我-> 1073260490@qq.com