多级缓存基础架构组件设计

背景

项目开发中对于缓存使用的不规范，如redis工具类使用五花八门，配置五花八门，生成key不规范，存储数据，序列化等不规范，对于缓存使用场景无法精确把控缓存使用粒度，对于缓存规模无法预估，缓存击穿，穿透，雪崩，缓存数据不一致，使用缓存不当造成的进程内外缓存不稳定等问题，通过基础缓存组件来解决一系列问题，简化和规范缓存使用，并提供统一的问题解决方案（技术层面）

目的

• 支持spring cache注解无缝集成多级缓存
• 支持单独配置使用不同的缓存类型(本地缓存或redis缓存或多级缓存)
• 支持原生redisTemplate的所有操作，移除cluster及部分不常用或危险操作，使用简单
• 注释和示例丰富，方便使用
• redis添加逻辑过期，请求进入自动刷新，大并发下不阻塞连接池，防止连接池被打爆
• 最终数据一致性，适用于对数据一致性要求不过于严格（强一致的场景)，一级缓存过期时间短，二级缓存主动拉取无限存储，可达到最终一致性，解决缓存数据不一致问题
• 缓存击穿与缓存雪崩由上述特性（redis永不过期+定时拉取+保证最终数据一致性）来解决，缓存穿透通过缓存null值解决
• 大key治理和内存优化：protostuff+lz4压缩，数据空间占比优化百分之35以上，提供预估数量级的分片功能，可自行选择分片数据，分片会自行按照预估数量级，尽量满足redis底层数据结构优化的条件。达到大key切分为小key，内存进一步压缩的可能。
• 提供标准的redis 发布订阅、key过期监听等使用方式，避免各种其他自定义方式创建redis pub/sub
• 一级缓存二级缓存可配置相关属性，提供类似于spring cache的配置项的丰富配置

缓存架构

服务层多级缓存核心流程设计

上述缓存设计架构图已经展示较为明确，本次主要分享服务端多级缓存的架构设计

1、缓存介质

caffine作为本地缓存，即一级缓存L1cache (进程内)
redis作为分布式缓存即L2cache (进程外)

缓存设计采取由近及远，由快到慢的思维进行设计

2、如何实现spring注解方式多级缓存

核心逻辑就是继承AbstractValueAdaptingCache并重写其相关方法，实现CacheManager接口中的相关接口并将该实现类交给spring托管

如何实现缓存的永久存储又可以过期刷新，而避免缓存失效带来的一系列缓存雪崩、击穿等问题

可以实现一个带有逻辑过期时间的缓存包装类，所有缓存值都被这个类所装饰，当逻辑过期时间过期时，访问的请求中只有一个缓存进行缓存的刷新，而其余进入请求的线程则可以直接返回缓存的旧值，这样就避免了@Cacheable(sync=true)时同步获取缓存带来的高并发下大量请求线程引刷新线程带来的请求hang住的情况

实现缓存包装类

/**
 * @classDesc:
 * @author: cyjer
 * @date: 2023/6/21 9:44
 */
@Data
@Accessors(chain = true)
public class CacheValueWrapper<V> {
    private V data;
    private LocalDateTime expireDate;

    public CacheValueWrapper() {
        this.expireDate = LocalDateTime.now().plusSeconds(8);
    }
}

可以看到其实很简单,使用一个逻辑的expireDate,在new 实例的过程中,这个逻辑过期时间就被赋予了8秒后过期，因此在请求进入命中缓存时可以先判断这个缓存值是否已经过期，如果已经过期则进行抢锁并刷新缓存，抢锁失败的线程则直接返回这个旧值

多级缓存处理器实现

/**
 * @classDesc: 功能描述:(二级缓存处理器)
 * @author: cyjer
 * @date: 2022/11/21 15:30
 */
@Slf4j
@Getter
public class RedisCaffeineCache extends AbstractValueAdaptingCache {

    private final String name;
    private final Cache<Object, Object> caffeineCache;
    private final StringCache stringCache;
    private final String cachePrefix;
    private final Duration defaultExpiration;
    private final Duration defaultNullValuesExpiration;
    private final Map<String, Duration> expires;
    private final String topic;
    private RedisMessagePublisher redisMessagePublisher;
    private final Map<String, LockKey> keyLockMap = new ConcurrentHashMap<>();

    public RedisCaffeineCache(String name, StringCache stringCache,
                              Cache<Object, Object> caffeineCache,
                              CacheConfigProperties cacheConfigProperties,
                              RedisMessagePublisher redisMessagePublisher) {
        super(cacheConfigProperties.isCacheNullValues());
        this.name = name;
        this.stringCache = stringCache;
        this.caffeineCache = caffeineCache;
        this.cachePrefix = cacheConfigProperties.getCachePrefix();
        this.defaultExpiration = cacheConfigProperties.getRedis().getDefaultExpiration();
        this.defaultNullValuesExpiration = cacheConfigProperties.getRedis().getDefaultNullValuesExpiration();
        this.expires = cacheConfigProperties.getRedis().getExpires();
        this.topic = PubTopic.topic;
        this.redisMessagePublisher = redisMessagePublisher;
    }

    private static class LockKey {
        private static final LockKey instance = new LockKey();
    }

    public static LockKey getLockKey() {
        return LockKey.instance;
    }


    @Override
    public Object getNativeCache() {
        return this;
    }

	//@Cacheable(sync=false)的时候会走这个方法
	//由于这个方法的参数没有方法的回调,因此缓存未命中时我们没法去执行业务方法去刷新缓存，只能设置为null
	//因此还需要一个机制去强制使用者必须设置@Cacheable(sync=true)
    @Override
    public ValueWrapper get(Object key) {
        Object value = this.lookup(key);
        if (value instanceof CacheValueWrapper) {
            if (LocalDateTime.now().isAfter(((CacheValueWrapper<?>) value).getExpireDate())) {
                return null;
            } else {
                return toValueWrapper(((CacheValueWrapper<?>) value).getData());
            }
        }
        return toValueWrapper(value);
    }

	//同步获取缓存
    @Override
    public <T> T get(Object key, Callable<T> valueLoader) {
        Object value = lookup(key);
        if (value != null) {
            //from l2 cache
            if (value instanceof CacheValueWrapper) {
                CacheValueWrapper<T> cacheValueWrapper = (CacheValueWrapper<T>) value;
                //判断是否已经逻辑过期
                if (LocalDateTime.now().isAfter(cacheValueWrapper.getExpireDate())) {
                    //l2 cache过期,获取缓存内容
                    //轻量锁，内部cas实现乐观锁，减小锁粒度，解决持有多把锁带来的内存飙升，以及重量级锁带来的性能损耗
                    LockKey lockKey = keyLockMap.putIfAbsent(key.toString(), RedisCaffeineCache.getLockKey());
                    if (Objects.isNull(lockKey)) {
                        //执行业务逻辑
                        try {
                            log.warn("refresh l2 and l1 cache from biz data.. key is {}", key);
                            value = valueLoader.call();
                            Object storeValue = toStoreValue(value);
                            put(key, storeValue);
                        } catch (Exception e) {
                            log.error("执行业务逻辑出错:", e);
                        } finally {
                            keyLockMap.remove(key.toString());
                        }
                    } else {
                        value = ((CacheValueWrapper<?>) value).getData();
                    }
                } else {
                    value = cacheValueWrapper.getData();
                }
            }
        } else {
            //应对缓存初始化或all keys 相关淘汰策略，当内存不足，永久key被淘汰后走这里，极端情况，少见
            //只有一个线程进入执行业务，其余线程阻塞。
            synchronized (key) {
                value = lookup(key);
                if (value != null) {
                    if (value instanceof CacheValueWrapper) {
                        CacheValueWrapper<T> cacheValueWrapper = (CacheValueWrapper<T>) value;
                        value = cacheValueWrapper.getData();
                    }
                } else {
                    try {
                        log.warn("refresh l2 and l1 cache from biz data.. key is {}", key);
                        value = valueLoader.call();
                        Object storeValue = toStoreValue(value);
                        put(key, storeValue);
                    } catch (Exception e) {
                        throw new ValueRetrievalException(key, valueLoader, e.getCause());
                    }
                }

            }
        }
        return (T) value;
    }

    @Override
    public void put(Object key, Object value) {
        if (!super.isAllowNullValues() && value == null) {
            this.evict(key);
            return;
        }
        doPut(key, value);
    }

    @Override
    public ValueWrapper putIfAbsent(Object key, Object value) {
        Object prevValue;
        prevValue = getRedisValue(key);
        if (prevValue == null) {
            doPut(key, value);
        }
        return toValueWrapper(prevValue);
    }

    private void doPut(Object key, Object value) {
        value = toStoreValue(value);
        Duration expire = getExpire(value);
        //存储包装类对象
        setRedisValue(key, value, expire);
        //推送消息通知本地缓存删除
        push(new CacheMessage(this.name, key));
        //存储为不包装对象
        setCaffeineValue(key, value);
    }

    @Override
    public void evict(Object key) {
        //删除当前key
        stringCache.getOperations().delete(getKey(key).get());
        //推送消息通知本地缓存删除
        push(new CacheMessage(this.name, key));
        //使无效化
        caffeineCache.invalidate(key);
    }

    @Override
    public void clear() {
        //清空该cacheName下的所有缓存对象
        Set<Object> keys = stringCache.getOperations().keys(this.name.concat(":*"));

        if (!CollectionUtils.isEmpty(keys)) {
            stringCache.getOperations().delete(keys);
        }
        //通知清除
        push(new CacheMessage(this.name, null));
        //使其无效
        caffeineCache.invalidateAll();
    }


    /**
     * 查找缓存
     *
     * @param key 缓存key
     * @return 缓存对象
     */
    @Override
    protected Object lookup(Object key) {
        //生成key
        Object cacheKey = getKey(key);
        //form l1 cache 缓存对象应为不包装对象
        Object value = getCaffeineValue(key);
        if (value != null) {
            log.info("get cache from l1 cache the key is : {}", cacheKey);
            return value;
        }
        //from l2 cache 对象为包装对象
        value = getRedisValue(key);

        if (value != null) {
            log.info("get cache from l2 cache and put in l1 cache, the key is : {}", cacheKey);
            //设置时应使用不包装对象保存
            if (value instanceof CacheValueWrapper) {
                CacheValueWrapper<Object> cacheValueWrapper = (CacheValueWrapper<Object>) value;
                setCaffeineValue(key, cacheValueWrapper.getData());
            }
        }
        //返回包装对象
        return value;
    }

    protected CacheKey getKey(Object key) {
        CacheKeyGenerator.CacheKeyGeneratorBuilder keyGeneratorBuilder = CacheKeyGenerator.builder().group(this.name);
        if (StringUtils.hasLength(cachePrefix)) {
            keyGeneratorBuilder.key(this.cachePrefix.concat(":").concat(key.toString()));
        } else {
            keyGeneratorBuilder.key(key.toString());
        }
        return keyGeneratorBuilder.build();
    }

    protected Duration getExpire(Object value) {
        Duration cacheNameExpire = expires.get(this.name);
        if (cacheNameExpire == null) {
            cacheNameExpire = defaultExpiration;
        }
        if ((value == null || value == NullValue.INSTANCE) && this.defaultNullValuesExpiration != null) {
            cacheNameExpire = this.defaultNullValuesExpiration;
        }
        return cacheNameExpire;
    }

    protected void push(CacheMessage message) {
        redisMessagePublisher.publish(topic, message);
    }

    public void clearLocal(Object key) {
        log.debug("clear local cache, the key is : {}", key);
        if (key == null) {
            caffeineCache.invalidateAll();
        } else {
            caffeineCache.invalidate(key);
        }
    }

    protected void setRedisValue(Object key, Object value, Duration expire) {
        if (!expire.isNegative() && !expire.isZero()) {
            stringCache.set(getKey(key), value, expire);
        } else {
            stringCache.set(getKey(key), value);
        }
    }

    protected Object getRedisValue(Object key) {
        return stringCache.getWrapper(getKey(key));
    }

    protected void setCaffeineValue(Object key, Object value) {
        caffeineCache.put(key, value);
    }

    protected Object getCaffeineValue(Object key) {
        return caffeineCache.getIfPresent(key);
    }

}

上述代码主要重写了get(Object key)，get(Object key, Callable valueLoader)，put(Object key, Object value)，putIfAbsent(Object key, Object value)，evict(Object key)，lookup(Object key)以及clear()方法，以下将解释上述部分方法有什么作用

1、lookup(Object key)

此方法为缓存查找方法,我们首先通过getKey方法，生成自定义规范的redis key,然后从l1即caffeine中查询缓存，如果缓存不为空，直接返回该缓存值，如果缓存为空，则继续查找redis缓存，如果缓存不为空，写入一级缓存，然后返回

2、get(Object key)

此方法为当使用@Cacheable()注解时，sync属性为false及异步获取缓存时，将会从这个方法获取,我们首先调用lookup方法查找缓存，通过LocalDateTime.now().isAfter(((CacheValueWrapper<?>) value).getExpireDate())判断缓存是否过期。

3、重头戏get(Object key, Callable valueLoader)

首先也去查找缓存,判断缓存是否存在,如果存在判断缓存是否已经过期，如果缓存已经过期，通过ConcurrentHashMap内部的乐观cas锁实现抢锁逻辑，减小锁粒度，解决持有多把锁带来的内存飙升，以及重量级锁带来的性能损耗，通过Objects.isNull(lockKey)判断是否抢锁成功，抢锁成功执行业务逻辑，并缓存该结果

如果缓存未命中，则代表缓存初始化或all keys 相关淘汰策略，当内存不足，永久key被淘汰，极端情，因此此时抢锁成功的线程，将进行缓存的刷新写入

缓存结果细节:

缓存结果主要是通过doPut方法进行 ，通过redis的发布订阅通知本地缓存更新

如何监听redis发布订阅事件?

1、定义消息生产者

/**
 * @classDesc:
 * @author: cyjer
 * @date: 2023/6/21 20:14
 */
@Component
public class RedisMessagePublisher {
    @Autowired
    @Qualifier("redisJsonClient")
    private RedisTemplate<String, Object> redisJsonClient;

    public void publish(String channel, Object message) {
        redisJsonClient.convertAndSend(channel, message);
    }
}

2、定义消息消费者接口

/**
 * @author cyjer
 */
public interface MessageConsumer {
    /**
     * 要监听的主题
     */
    String topic();

    /**
     * 收到消息处理业务，需要自行实现
     *
     * @param message 接收到的消息
     */
    void receiveMessage(Object message);
}

3、自动配置

/**
 * @classDesc:
 * @author: cyjer
 * @date: 2023/6/21 16:58
 */
@Configuration
@AutoConfigureAfter(L2CacheAutoConfiguration.class)
public class PubSubDiscoverer implements ApplicationContextAware, InitializingBean {
    private ApplicationContext applicationContext;

    @Override
    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
        this.applicationContext = applicationContext;
    }

    @Override
    public void afterPropertiesSet() throws Exception {
        Map<String, MessageConsumer> beansOfType = applicationContext.getBeansOfType(MessageConsumer.class);
        RedisMessageListenerContainer redisMessageListenerContainer = applicationContext.getBean("cacheMessageListenerContainerByCY", RedisMessageListenerContainer.class);
        for (Map.Entry<String, MessageConsumer> listener : beansOfType.entrySet()) {
            MessageListenerAdapter messageListenerAdapter = new MessageListenerAdapter(listener.getValue(), "receiveMessage");
            messageListenerAdapter.afterPropertiesSet();
            redisMessageListenerContainer.addMessageListener(messageListenerAdapter, new PatternTopic(listener.getValue().topic()));
        }
    }
}

4、实现缓存变动监听

/**
 * @classDesc: 功能描述:(key变动监听)
 * @author: cyjer
 * @date: 2022/11/21 15:30
 */
@Slf4j
@RequiredArgsConstructor
@Data
public class CacheMessageListener implements MessageConsumer {

    private RedisCaffeineCacheManager redisCaffeineCacheManager;

    @Override
    public String topic() {
        return PubTopic.topic;
    }

    @Override
    public void receiveMessage(Object message) {
        CacheMessage cacheMessage = JSONObject.parseObject(message.toString(), CacheMessage.class);
        redisCaffeineCacheManager.clearLocal(cacheMessage.getCacheName(), cacheMessage.getKey());
    }

}

缓存管理器实现

/**
 * @classDesc: 功能描述:(二级缓存管理器)
 * @author: cyjer
 * @date: 2022/11/21 15:30
 */
@Slf4j
@Getter
@Setter
public class RedisCaffeineCacheManager implements CacheManager {

    private ConcurrentMap<String, Cache> cacheMap = new ConcurrentHashMap<>();
    private CacheConfigProperties cacheConfigProperties;
    private StringCache stringCache;
    private boolean dynamic;
    private Set<String> cacheNames;
    private RedisMessagePublisher redisMessagePublisher;
    private CaffeineCacheClient caffeineCacheClient;

    public RedisCaffeineCacheManager(CacheConfigProperties cacheConfigProperties,
                                     StringCache stringCache,
                                     RedisMessagePublisher redisMessagePublisher,
                                     CaffeineCacheClient caffeineCacheClient) {
        super();
        this.caffeineCacheClient = caffeineCacheClient;
        this.cacheConfigProperties = cacheConfigProperties;
        this.stringCache = stringCache;
        this.dynamic = cacheConfigProperties.isDynamic();
        this.cacheNames = cacheConfigProperties.getCacheNames();
        this.redisMessagePublisher = redisMessagePublisher;
    }

    @Override
    public Cache getCache(String name) {
        Cache cache = cacheMap.get(name);
        if (cache != null) {
            return cache;
        }
        if (!dynamic && !cacheNames.contains(name)) {
            return null;
        }

        cache = createCache(name);
        Cache oldCache = cacheMap.putIfAbsent(name, cache);
        return oldCache == null ? cache : oldCache;
    }

    public RedisCaffeineCache createCache(String name) {
        return new RedisCaffeineCache(name, stringCache, caffeineCacheClient.getClient(), cacheConfigProperties, redisMessagePublisher);
    }


    @Override
    public Collection<String> getCacheNames() {
        return this.cacheNames;
    }

    public void clearLocal(String cacheName, Object key) {
        Cache cache = cacheMap.get(cacheName);
        if (cache == null) {
            return;
        }

        RedisCaffeineCache redisCaffeineCache = (RedisCaffeineCache) cache;
        redisCaffeineCache.clearLocal(key);
    }

}

如何规范缓存注解的使用

上述多级缓存管理器的重写带来了一个问题，那就是要求开发者们在使用@Cacheable注解时必须配置sync=true，如何限制开发者们的开发规范呢？可以使用BeanPostProcessor这个spring扩展点来做

BeanPostProcessor后置处理器：bean初始化前后进行处理工作

实现:

/**
 * @classDesc: 功能描述:()
 * @author: cyjer
 * @date: 2023/6/24 3:18
 */
@Configuration
@ConditionalOnProperty(name = "cache.useCacheValid", havingValue = "true", matchIfMissing = true)
public class UseCacheValid implements BeanPostProcessor {
    @Override
    public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException {
        Class<?> aClass = bean.getClass();
        Method[] methods = aClass.getMethods();
        Arrays.stream(methods).parallel().forEach(e -> {
            if (e.isAnnotationPresent(Cacheable.class)) {
                Cacheable annotation = e.getAnnotation(Cacheable.class);
                boolean sync = annotation.sync();
                if (!sync) {
                    throw new BeanCreationException("使用@Cacheable注解时必须指定sync属性为true");
                }
            }
        });
        return bean;
    }
}

大家自行体会~~~~

如何规范开发者们的key生成以及客户端在缓存时自动使用包装的缓存类进行缓存的呢?

• 以下仅拿StringCache举一个例子解释即可

1、定义CacheKey接口

/**
 * @classDesc: 功能描述:(key顶层接口)
 * @author: cyjer
 * @date: 2022/11/21 15:30
 */
public interface CacheKey {

    /**
     * 获取key
     *
     * @return
     */
    String get();

    String getCacheGroup();

    String getCacheKey();

}

2、默认实现类

/**
 * @classDesc: 功能描述:(默认key构造器,可自定义扩展key生成方式)
 * @author: cyjer
 * @date: 2022/11/21 15:30
 */
@Builder
@Data
@Slf4j
public class CacheKeyGenerator implements CacheKey {

    private static final String SPLIT = ":";

    private String group;

    private String key;

    @Override
    public String get() {
        return StringUtils.isNotBlank(key) ? group + SPLIT + key : group + SPLIT;
    }

    @Override
    public String getCacheGroup() {
        return group + SPLIT;
    }

    @Override
    public String getCacheKey() {
        return key;
    }
}

上述方式即通过冒号对group和key进行分割,在具体使用的时候通过形如：
CacheKeyGenerator.builder().group(“spu”).key(“spuId”).build()
即可构造出spu:spuId的缓存key来

3、定义标准redis客户端

定义顶层接口

/**
 * @classDesc: 功能描述:()
 * @author: cyjer
 * @date: 2022/11/21 15:30
 */
@Example("StringCache<MessageGenerator> stringCache = StringCache.cacheGenerate();")
public interface StringCache<V> {

    @Comment("缓存生成器")
    static <V> StringCache<V> cacheGenerate(){
        return StringCacheStandardClient.generate();
    }

    @Comment("设置一个值到redis")
    @Example("stringCache.set(...)")
    void set(CacheKey key, V value);

    @Comment("设置一个值到redis,给定过期时间")
    @Example("stringCache.set(...)")
    void set(CacheKey key, V value, long timeout, TimeUnit unit);

    @Comment("设置一个值到redis,给定过期时间")
    @Example("stringCache.set(...)")
    default void set(CacheKey key, V value, Duration timeout) {

        Assert.notNull(timeout, "Timeout must not be null!");

        if (TimeoutUtils.hasMillis(timeout)) {
            set(key, value, timeout.toMillis(), TimeUnit.MILLISECONDS);
        } else {
            set(key, value, timeout.getSeconds(), TimeUnit.SECONDS);
        }
    }

    @Comment("如果值在redis不存在,设置一个值到redis")
    @Example("stringCache.setIfAbsent(...)")
    @Nullable
    Boolean setIfAbsent(CacheKey key, V value);

    @Comment("如果值在redis不存在,设置一个值到redis并给定过期时间")
    @Example("stringCache.setIfAbsent(...)")
    @Nullable
    Boolean setIfAbsent(CacheKey key, V value, long timeout, TimeUnit unit);

    @Comment("如果值在redis不存在,设置一个值到redis并给定过期时间")
    @Example("stringCache.setIfAbsent(...)")
    @Nullable
    default Boolean setIfAbsent(CacheKey key, V value, Duration timeout) {

        Assert.notNull(timeout, "Timeout must not be null!");

        if (TimeoutUtils.hasMillis(timeout)) {
            return setIfAbsent(key, value, timeout.toMillis(), TimeUnit.MILLISECONDS);
        }

        return setIfAbsent(key, value, timeout.getSeconds(), TimeUnit.SECONDS);
    }

    @Comment("如果值在redis存在,更新值")
    @Example("stringCache.setIfPresent(...)")
    @Nullable
    Boolean setIfPresent(CacheKey key, V value);

    @Comment("如果值在redis存在,更新值并给定过期时间")
    @Example("stringCache.setIfPresent(...)")
    @Nullable
    Boolean setIfPresent(CacheKey key, V value, long timeout, TimeUnit unit);

    @Comment("如果值在redis存在,更新值并给定过期时间")
    @Example("stringCache.setIfPresent(...)")
    @Nullable
    default Boolean setIfPresent(CacheKey key, V value, Duration timeout) {

        Assert.notNull(timeout, "Timeout must not be null!");

        if (TimeoutUtils.hasMillis(timeout)) {
            return setIfPresent(key, value, timeout.toMillis(), TimeUnit.MILLISECONDS);
        }

        return setIfPresent(key, value, timeout.getSeconds(), TimeUnit.SECONDS);
    }

    @Comment("设置多个key,value到redis")
    @Example("stringCache.multiSet(...)")
    void multiSet(Map<? extends CacheKey, ? extends V> map);

    @Comment("设置多个key,value到redis，如果key不存在")
    @Example("stringCache.multiSetIfAbsent(...)")
    @Nullable
    Boolean multiSetIfAbsent(Map<? extends CacheKey, ? extends V> map);

    @Comment("根据key获取值")
    @Example("stringCache.get(...)")
    @Nullable
    V get(CacheKey key);

    @Comment("根据key获取包装值")
    @Example("stringCache.get(...)")
    @Nullable
    CacheValueWrapper<V> getWrapper(CacheKey key);

    @Comment("根据key获取值并删除值")
    @Example("stringCache.getAndDelete(...)")
    @Nullable
    V getAndDelete(CacheKey key);

    @Comment("根据key获取值并设置过期时间")
    @Example("stringCache.getAndExpire(...)")
    @Nullable
    V getAndExpire(CacheKey key, long timeout, TimeUnit unit);

    @Comment("根据key获取值并设置过期时间")
    @Example("stringCache.getAndExpire(...)")
    @Nullable
    V getAndExpire(CacheKey key, Duration timeout);

    @Comment("根据key获取值并移除过期时间")
    @Example("stringCache.getAndPersist(...)")
    @Nullable
    V getAndPersist(CacheKey key);

    @Comment("根据key获取并更新值")
    @Example("stringCache.getAndSet(...)")
    @Nullable
    V getAndSet(CacheKey key, V value);

    @Comment("批量获取值")
    @Example("stringCache.multiGet(...)")
    @Nullable
    List<V> multiGet(Collection<CacheKey> keys);

    @Comment("将key值自增1")
    @Example("stringCache.increment(...)")
    @Nullable
    Long increment(CacheKey key);

    @Comment("将key值延迟delta后自增1")
    @Example("stringCache.increment(...)")
    @Nullable
    Long increment(CacheKey key, long delta);

    @Comment("将key值延迟delta后自增1")
    @Example("stringCache.increment(...)")
    @Nullable
    Double increment(CacheKey key, double delta);

    @Comment("将key值自减1")
    @Example("stringCache.decrement(...)")
    @Nullable
    Long decrement(CacheKey key);

    @Comment("将key值延迟delta后自减1")
    @Example("stringCache.decrement(...)")
    @Nullable
    Long decrement(CacheKey key, long delta);

    @Comment("在字符串值后增加value")
    @Example("stringCache.append(...)")
    @Nullable
    Integer append(CacheKey key, String value);

    @Comment("截取key所对应的字符串")
    @Example("stringCache.get(...)")
    @Nullable
    String get(CacheKey key, long start, long end);

    @Comment("从offset开始替换，索引下表0开始，offset超过长度会报错")
    @Example("stringCache.set(...)")
    void set(CacheKey key, V value, long offset);

    @Comment("该值长度")
    @Example("stringCache.size(...)")
    @Nullable
    Long size(CacheKey key);

    @Comment("新增单个bitmap键值对")
    @Example("stringCache.setBit(...)")
    @Nullable
    Boolean setBit(CacheKey key, long offset, boolean value);

    @Comment("判断bitmap下的offset位置是否true")
    @Example("stringCache.getBit(...)")
    @Nullable
    Boolean getBit(CacheKey key, long offset);

    @Nullable
    List<Long> bitField(CacheKey key, BitFieldSubCommands subCommands);

    Boolean delete(CacheKey key);

    RedisOperations<CacheKey, V> getOperations();
}

标准实现类

/**
 * @classDesc: 功能描述:(String操作)
 * @author: cyjer
 * @date: 2022/11/21 15:30
 */
@Slf4j
public class StringCacheStandardClient<V> implements StringCache<V> {
    private ValueOperations<String, CacheValueWrapper<V>> operations;
    private RedisTemplate redisTemplate;

    public StringCacheStandardClient() {
        this.redisTemplate = SpringBeanUtil.getBean("redisStandardClient");
        this.operations = redisTemplate.opsForValue();
    }

    public StringCacheStandardClient(RedisTemplate redisTemplate) {
        this.redisTemplate = redisTemplate;
        this.operations = redisTemplate.opsForValue();
    }

    private static class CacheGenerator {
        private static final StringCache cache = new StringCacheStandardClient<>();
    }

    public static <V> StringCache<V> generate() {
        return CacheGenerator.cache;
    }

    @Override
    public void set(CacheKey key, V value) {
        operations.set(key.get(), new CacheValueWrapper<V>().setData(value));
    }

    @Override
    public void set(CacheKey key, V value, long timeout, TimeUnit unit) {
        operations.set(key.get(), new CacheValueWrapper<V>().setData(value), timeout, unit);
    }

    @Override
    public Boolean setIfAbsent(CacheKey key, V value) {
        return operations.setIfAbsent(key.get(), new CacheValueWrapper<V>().setData(value));
    }

    @Override
    public Boolean setIfAbsent(CacheKey key, V value, long timeout, TimeUnit unit) {
        return operations.setIfAbsent(key.get(), new CacheValueWrapper<V>().setData(value), timeout, unit);
    }

    @Override
    public Boolean setIfPresent(CacheKey key, V value) {
        return operations.setIfPresent(key.get(), new CacheValueWrapper<V>().setData(value));
    }

    @Override
    public Boolean setIfPresent(CacheKey key, V value, long timeout, TimeUnit unit) {
        return operations.setIfAbsent(key.get(), new CacheValueWrapper<V>().setData(value), timeout, unit);
    }

    @Override
    public void multiSet(Map<? extends CacheKey, ? extends V> map) {
        Map<String, CacheValueWrapper<V>> values = new HashMap<>();
        for (Map.Entry<? extends CacheKey, ? extends V> v : map.entrySet()) {
            values.put(v.getKey().get(), new CacheValueWrapper<V>().setData(v.getValue()));
        }
        operations.multiSet(values);
    }

    @Override
    public Boolean multiSetIfAbsent(Map<? extends CacheKey, ? extends V> map) {
        Map<String, CacheValueWrapper<V>> values = new HashMap<>();
        for (Map.Entry<? extends CacheKey, ? extends V> v : map.entrySet()) {
            values.put(v.getKey().get(), new CacheValueWrapper<V>().setData(v.getValue()));
        }
        return operations.multiSetIfAbsent(values);
    }

    @Override
    public V get(CacheKey key) {
        CacheValueWrapper<V> vCacheValueWrapper = operations.get(key.get());
        if (Objects.nonNull(vCacheValueWrapper)){
            return vCacheValueWrapper.getData();
        }
        return null;
    }

    @Override
    public CacheValueWrapper<V> getWrapper(CacheKey key) {
        return operations.get(key.get());
    }

    @Override
    public V getAndDelete(CacheKey key) {
        CacheValueWrapper<V> vCacheValueWrapper = operations.getAndDelete(key.get());
        if (Objects.nonNull(vCacheValueWrapper)){
            return vCacheValueWrapper.getData();
        }
        return null;
    }

    @Override
    public V getAndExpire(CacheKey key, long timeout, TimeUnit unit) {
        CacheValueWrapper<V> vCacheValueWrapper = operations.getAndExpire(key.get(), timeout, unit);
        if (Objects.nonNull(vCacheValueWrapper)){
            return vCacheValueWrapper.getData();
        }
        return null;
    }

    @Override
    public V getAndExpire(CacheKey key, Duration timeout) {
        CacheValueWrapper<V> vCacheValueWrapper = operations.getAndExpire(key.get(), timeout);
        if (Objects.nonNull(vCacheValueWrapper)){
            return vCacheValueWrapper.getData();
        }
        return null;
    }

    @Override
    public V getAndPersist(CacheKey key) {
        CacheValueWrapper<V> vCacheValueWrapper = operations.getAndPersist(key.get());
        if (Objects.nonNull(vCacheValueWrapper)){
            return vCacheValueWrapper.getData();
        }
        return null;
    }

    @Override
    public V getAndSet(CacheKey key, V value) {
        CacheValueWrapper<V> vCacheValueWrapper = operations.getAndSet(key.get(), new CacheValueWrapper<V>().setData(value));
        if (Objects.nonNull(vCacheValueWrapper)){
            return vCacheValueWrapper.getData();
        }
        return null;
    }

    @Override
    public List<V> multiGet(Collection<CacheKey> keys) {
        List<CacheValueWrapper<V>> cacheValueWrappers = operations.multiGet(keys.stream().map(CacheKey::get).collect(Collectors.toList()));
        if (cacheValueWrappers != null) {
            return cacheValueWrappers.stream().map(CacheValueWrapper::getData).collect(Collectors.toList());
        }
        return null;
    }

    @Override
    public Long increment(CacheKey key) {
        return operations.increment(key.get());
    }

    @Override
    public Long increment(CacheKey key, long delta) {
        return operations.increment(key.get(), delta);
    }

    @Override
    public Double increment(CacheKey key, double delta) {
        return operations.increment(key.get(), delta);
    }

    @Override
    public Long decrement(CacheKey key) {
        return operations.decrement(key.get());
    }

    @Override
    public Long decrement(CacheKey key, long delta) {
        return operations.decrement(key.get(), delta);
    }

    @Override
    public Integer append(CacheKey key, String value) {
        return operations.append(key.get(), value);
    }

    @Override
    public String get(CacheKey key, long start, long end) {
        return operations.get(key.get(), start, end);
    }

    @Override
    public void set(CacheKey key, V value, long offset) {
        operations.set(key.get(), new CacheValueWrapper<V>().setData(value), offset);
    }

    @Override
    public Long size(CacheKey key) {
        return operations.size(key.get());
    }

    @Override
    public Boolean setBit(CacheKey key, long offset, boolean value) {
        return operations.setBit(key.get(), offset, value);
    }

    @Override
    public Boolean getBit(CacheKey key, long offset) {
        return operations.getBit(key.get(), offset);
    }

    @Override
    public List<Long> bitField(CacheKey key, BitFieldSubCommands subCommands) {
        return operations.bitField(key.get(), subCommands);
    }

    @Override
    public Boolean delete(CacheKey key) {
        return redisTemplate.delete(key.get());
    }

    @Override
    public RedisOperations<CacheKey, V> getOperations() {
        return redisTemplate;
    }
}

上述代码中set操作的缓存值都通过new CacheValueWrapper().setData(value)进行了缓存类的包装,因此开发者使用时可以无感知的将缓存进行存取

关于SpringBeanUtil.getBean(“redisStandardClient”)

关于这个类是怎么来的呢？且看下文

/**
 * @classDesc:
 * @author: cyjer
 * @date: 2023/6/21 18:51
 */
@Configuration
public class MRedisTemplateConfig {
    @Bean("redisStandardClient")
    @Primary
    public RedisTemplate<String, Object> redisTemplate(LettuceConnectionFactory factory) {
        RedisTemplate<String, Object> template = new RedisTemplate<>();
        template.setConnectionFactory(factory);
        RedisSerializer<Object> serializer = new ProtostuffSerializer<>(true);
        template.setKeySerializer(new ProtostuffSerializer());
        template.setValueSerializer(serializer);
        template.setDefaultSerializer(serializer);
        template.setHashKeySerializer(new ProtostuffSerializer());
        template.setHashValueSerializer(serializer);
        ProtostuffSerializer<String> stringProtostuffSerializer = new ProtostuffSerializer<>(true);
        template.setStringSerializer(stringProtostuffSerializer);
        return template;
    }
}

通过自定义RedisTemplate并通过@Primary将该bean设置为主要bean，当出现重复类型的RedisTemplate时将优先是由此bean

ProtostuffSerializer序列化,内存和性能优化的关键

上述代码大家注意到了ProtostuffSerializer这个类，这是为什么呢？除redis发布订阅不能使用这个序列化器之外，其余我们的所有操作其实都是使用的ProtostuffSerializer，这是大key治理和内存优化的关键protostuff+lz4压缩，数据空间占比可以优化百分之35以上

@Data
@AllArgsConstructor
@NoArgsConstructor
public class ProtostuffSerializer<T> implements RedisSerializer<T> {

    private static final Schema<ObjectWrapper> SCHEMA = RuntimeSchema.getSchema(ObjectWrapper.class);
    private boolean compress;

    @Override
    public byte[] serialize(Object t) throws SerializationException {
        LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE);
        byte[] bytes;
        try {
            bytes = ProtostuffIOUtil.toByteArray(new ObjectWrapper(t), SCHEMA, buffer);
        } finally {
            buffer.clear();
        }
        if (compress) {
            bytes = Lz4Util.compress(bytes);
        }
        return bytes;
    }

    @Override
    public T deserialize(byte[] bytes) throws SerializationException {
        if (bytes == null || bytes.length == 0) {
            return null;
        }
        try {
            if (compress) {
                bytes = Lz4Util.unCompress(bytes);
            }
            ObjectWrapper<T> objectWrapper = new ObjectWrapper<>();
            ProtostuffIOUtil.mergeFrom(bytes, objectWrapper, SCHEMA);
            return objectWrapper.getObject();
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

    public static class ObjectWrapper<T> {
        private T object;

        ObjectWrapper() {
        }

        ObjectWrapper(T object) {
            this.object = object;
        }

        public T getObject() {
            return object;
        }

        public void setObject(T object) {
            this.object = object;
        }
    }
}

lz4压缩，压缩比高且效率比gzip快的压缩算法

上述代码可以看到，在此序列化器当中，还使用到了一个Lz4Util，此压缩工具类在缓存数据时对数据进行压缩，取数据时对缓存值进行解压缩，实测对性能影响不大，但内存优化如160k的数据可以压缩到50k左右。

@Slf4j
public class Lz4Util {

    private static final int ARRAY_SIZE = 4096;
    private static LZ4Factory factory = LZ4Factory.fastestInstance();
    private static LZ4Compressor compressor = factory.fastCompressor();
    private static LZ4FastDecompressor decompressor = factory.fastDecompressor();


    public static byte[] compress(byte bytes[]) {
        if (bytes == null || bytes.length == 0) {
            return null;
        }
        ByteArrayOutputStream outputStream = null;
        LZ4BlockOutputStream lz4BlockOutputStream = null;
        try {
            outputStream = new ByteArrayOutputStream();
            lz4BlockOutputStream = new LZ4BlockOutputStream(outputStream, ARRAY_SIZE, compressor);
            lz4BlockOutputStream.write(bytes);
            lz4BlockOutputStream.finish();
            return outputStream.toByteArray();
        } catch (Exception e) {
            log.error("Lz4Util compress error", e);
            return null;
        } finally {
            closeStream(lz4BlockOutputStream);
            closeStream(outputStream);
        }
    }

    public static byte[] unCompress(byte[] bytes) {
        if (bytes == null || bytes.length == 0) {
            return null;
        }
        ByteArrayOutputStream outputStream = null;

        ByteArrayInputStream inputStream = null;
        LZ4BlockInputStream decompressedInputStream = null;
        try {
            outputStream = new ByteArrayOutputStream(ARRAY_SIZE);

            inputStream = new ByteArrayInputStream(bytes);
            decompressedInputStream = new LZ4BlockInputStream(inputStream, decompressor);
            int count;
            byte[] buffer = new byte[ARRAY_SIZE];
            while ((count = decompressedInputStream.read(buffer)) != -1) {
                outputStream.write(buffer, 0, count);
            }

            return outputStream.toByteArray();
        } catch (Exception e) {
            e.printStackTrace();
            log.error("Lz4Util uncompress error", e);
            return null;
        } finally {
            closeStream(decompressedInputStream);
            closeStream(inputStream);
            closeStream(outputStream);
        }
    }

    private static void closeStream(Closeable oStream) {
        if (null != oStream) {
            try {
                oStream.close();
            } catch (IOException e) {
                oStream = null;
                e.printStackTrace();
            }
        }
    }
}

将缓存管理器交给spring托管

/**
 * @classDesc: 功能描述:(config)
 * @author: 曹越
 * @date: 2022/11/21 15:30
 */
@Configuration
@AutoConfigureAfter({RedisAutoConfiguration.class, CaffeineCacheClient.class})
@EnableConfigurationProperties(CacheConfigProperties.class)
public class L2CacheAutoConfiguration {

    @Resource
    private RedisMessagePublisher redisMessagePublisher;
    @Autowired
    @Qualifier("redisJsonClient")
    private RedisTemplate<String, Object> redisJsonClient;
    @Resource
    private CaffeineCacheClient caffeineCacheClient;

    @Bean
    public CacheManager cacheManager(CacheConfigProperties cacheConfigProperties, @Qualifier("redisStandardClient") RedisTemplate<String, Object> redisStandardClient) {
        if (cacheConfigProperties.getCacheType().equals(CacheConfigProperties.CacheType.redis)) {
            RedisConfigProp redis = cacheConfigProperties.getRedis();
            RedisCacheConfiguration redisCacheConfiguration = RedisCacheConfiguration.defaultCacheConfig()
                    .entryTtl(redis.getDefaultExpiration());
            redisCacheConfiguration.serializeValuesWith(RedisSerializationContext.SerializationPair.fromSerializer(new ProtostuffSerializer<>(true)));
            redisCacheConfiguration.serializeKeysWith(RedisSerializationContext.SerializationPair.fromSerializer(new ProtostuffSerializer<>()));
            if (StringUtils.isNotBlank(cacheConfigProperties.getCachePrefix())) {
                redisCacheConfiguration.prefixCacheNameWith(cacheConfigProperties.getCachePrefix());
            }
            RedisCacheWriter cacheWriter = RedisCacheWriter.nonLockingRedisCacheWriter(redisStandardClient.getConnectionFactory());
            return RedisCacheManager
                    .builder(cacheWriter)
                    .initialCacheNames(cacheConfigProperties.getCacheNames())
                    .cacheDefaults(redisCacheConfiguration).build();

        } else if (cacheConfigProperties.getCacheType().equals(CacheConfigProperties.CacheType.l2cache)) {

            return new RedisCaffeineCacheManager(cacheConfigProperties, new StringCacheStandardClient(redisStandardClient), redisMessagePublisher, caffeineCacheClient);

        } else if (cacheConfigProperties.getCacheType().equals(CacheConfigProperties.CacheType.caffeine)) {
            SimpleCacheManager simpleCacheManager = new SimpleCacheManager();
            List<Cache> caches = new ArrayList<>();
            Set<String> cacheNames = cacheConfigProperties.getCacheNames();
            if (cacheNames == null || cacheNames.isEmpty()) {
                throw new BeanCreationException("cacheNames.cacheNames不能为空");
            }
            for (String cacheName : cacheConfigProperties.getCacheNames()) {
                caches.add(new CaffeineCache(cacheName, caffeineCacheClient.getClient()));
            }
            simpleCacheManager.setCaches(caches);
            return simpleCacheManager;
        } else {
            throw new BeanCreationException("不支持当前的缓存类型,缓存支持类型为:l2cache、redis、caffeine");
        }
    }

    @Bean
    @ConditionalOnProperty(value = "cache.onlyRedis", havingValue = "false")
    public CacheMessageListener cacheMessageListener(CacheManager cacheManager) {
        CacheMessageListener cacheMessageListener = new CacheMessageListener();
        cacheMessageListener.setRedisCaffeineCacheManager((RedisCaffeineCacheManager) cacheManager);
        return cacheMessageListener;
    }

    @Bean("cacheMessageListenerContainerByCY")
    public RedisMessageListenerContainer cacheMessageListenerContainer() {
        RedisMessageListenerContainer redisMessageListenerContainer = new RedisMessageListenerContainer();
        redisMessageListenerContainer.setConnectionFactory(redisJsonClient.getConnectionFactory());
        return redisMessageListenerContainer;
    }


}

看着有点懵逼？没关系，继续往下走

定义CacheMessageListener的bean同时通过@ConditionalOnProperty(value = “cache.onlyRedis”, havingValue = “false”)判断当前的缓存类型是否是仅使用redis缓存，如果是仅使用redis缓存则不需要进行本地缓存的刷新，也就不需要这个bean了

RedisMessageListenerContainer是redis发布订阅的容器bean定义，还记的上面的PubSubDiscoverer吗，其中的afterPropertiesSet便是通过InitializingBean扩展点，对MessageConsumer的接口实现bean进行注册添加到RedisMessageListenerContainer容器中。

关于CacheManager的bean定义，还涉及到相关的配置文件，且看下文

配置文件定义

配置文件主要分为以下几个：

• CacheConfigProperties
• CaffeineConfigProp
• RedisConfigProp

1、CacheConfigProperties

/**
 * @classDesc:
 * @author: cyjer
 * @date: 2023/1/31 15:14
 */
@Data
@ConfigurationProperties(prefix = "cache")
public class CacheConfigProperties {

    private Set<String> cacheNames = new HashSet<>();

    /**
     * 是否存储空值，默认true，防止缓存穿透
     */
    private boolean cacheNullValues = true;

    /**
     * 是否动态根据cacheName创建Cache的实现，默认true
     */
    private boolean dynamic = true;

    /**
     * 缓存类型
     */
    private CacheType cacheType = CacheType.l2cache;

    /**
     * 缓存key的前缀
     */
    private String cachePrefix;

    @NestedConfigurationProperty
    private RedisConfigProp redis = new RedisConfigProp();

    @NestedConfigurationProperty
    private CaffeineConfigProp caffeine = new CaffeineConfigProp();

    /**
     * 缓存类型
     */
    public enum CacheType {
        /**
         * 多级缓存
         */
        l2cache,
        /**
         * 仅redis缓存
         */
        redis,
        /**
         * 仅caffeine本地缓存
         */
        caffeine;

        CacheType() {
        }
    }

}

2、CaffeineConfigProp

/**
 * @classDesc:
 * @author: cyjer
 * @date: 2023/1/31 15:14
 */
@Data
public class CaffeineConfigProp {

    /**
     * 访问后过期时间
     */
    private Duration expireAfterAccess;

    /**
     * 写入后过期时间
     */
    private Duration expireAfterWrite;

    /**
     * 初始化大小
     */
    private int initialCapacity;

    /**
     * 最大缓存对象个数，超过此数量时之前放入的缓存将失效
     */
    private long maximumSize;
    /**
     * value 强度
     */
    private CaffeineStrength valueStrength;

}

3、RedisConfigProp

/**
 * @classDesc:
 * @author: cyjer
 * @date: 2023/1/31 15:14
 */
@Data
public class RedisConfigProp {

	/**
	 * 全局过期时间，默认不过期
	 */
	private Duration defaultExpiration = Duration.ZERO;

	/**
	 * 全局空值过期时间，默认和有值的过期时间一致，一般设置空值过期时间较短
	 */
	private Duration defaultNullValuesExpiration = Duration.ofMinutes(5L);

	/**
	 * 每个cacheName的过期时间，优先级比defaultExpiration高
	 */
	private Map<String, Duration> expires = new HashMap<>();

}

具体配置示例:

cache:
  cacheType: l2cache #缓存类型: l2cache(多级缓存),redis,caffeine(本地缓存)
  cacheNames: 'test1,test2,test3' #cacheName,尽量不要使用冒号等特殊字符隔开
  cacheNullValues: true #是否缓存空值
  cachePrefix: cache #缓存前缀
  redis:
    defaultExpiration: 30s #默认永不过期
    defaultNullValuesExpiration: 5m #默认5分钟
    expires: #针对cacheName的过期时间设置，比defaultExpiration优先级高
      #冒号会被识别为yml语法会导致cacheName对应不起来
      'test1': 1h
      'test2': 1m
      'test3': 10s
  caffeine:
    expireAfterAccess: 10s #访问后多久过期，失效后同步加载缓存,不会配置使用默认
    expireAfterWrite: 10s #写入后多久过期，失效后同步加载缓存，默认五秒,不会配置使用默认
    initialCapacity: 1000 #初始化大小,默认1000,不会配置使用默认
    maximumSize: 1000 #最大缓存对象个数，超过此数量时之前放入的缓存将失效,默认10000,不会配置使用默认
    valueStrength: SOFT #默认SOFT软引用,可选:SOFT(软引用)，WEAK(弱引用),不会配置使用默认

如何在redis、caffeine、多级缓存之间自由切换

继续上述CacheManager的bean定义，主要流程为：

1、判断开发者配置的cacheType是否是redis，如果是redis类型缓存则初始化RedisCacheManager
2、如果是多级缓存则使用我们自定义的RedisCaffeineCacheManager
3、如果是caffeine缓存则使用SimpleCacheManager对caffeine进行管理

3.如何实现预估数量级的分片功能

为什么要这么做？

了解redis的人就会知道，像hash类型，set类型等，高级数据类型，都会有底层数据结构的优化，当我们的hash类型的一个键中存在几千几万甚至几十万的数据时，这个key将会变成一个名副其实的大key，由于redis单线程的特性，对于大key的存取，将很有可能会导致线程阻塞等性能问题，因此对于这些大key，我们可以采取一些措施进行优化

大key优化治理

在上文中我们提到我们使用了protobuff和lz4对数据进行了压缩，这带来了一部分性能提升，同时redis本身对数据结构的优化我们其实也可以针对性的进行一些调教，例如redis的hash类型，默认当hash键内的数据每条数据都在128字节内，同时整个key的hash值数量不超过512的时候，redis便会自行将hash结构优化为ziplist压缩链表，这会大大提升redis的存取性能，因此在上述基础上，我们还为了满足redis的这些要求，提供了一些预估数量分片的工具类

预估数量级枚举

/**
 * @classDesc: 未来可能的数据量级别
 * @author: cyjer
 * @date: 2023/6/21 9:13
 */
@AllArgsConstructor
@Getter
public enum ShardingLevel {
    /**
     * 100w级别数据
     */
    shard100W(2500),
    shard200W(5000),
    shard300W(7500),
    shard400W(9500),
    shard500W(12000),
    shard600W(15000),
    shard700W(18000),
    shard800W(21000),
    shard900W(24000),
    /**
     * 1000w级别数据
     */
    shard1000W(25000),
    shard3000W(75000),
    shard5000W(125000),
    shard7000W(170000),
    shard1E(230000),
    ;

    final Integer bucketCount;
}

以上的bucketCount是我经过计算后，得出的在某数量级下大概需要的分片数量。

分片工具类

/**
 * @classDesc:
 * @author: cyjer
 * @date: 2023/3/6 12:50
 */
@Slf4j
public class ShardingUtil {

    public static void main(String[] args) {
        //桶，数量
        HashMap<String, Long> hashMap = new HashMap<>();
        for (int i = 0; i < 7000000; i++) {
            String id = String.valueOf(i);
            String bucket = sharding(id,ShardingLevel.shard900W);
            //存放在哪个桶下
            Long orDefault = hashMap.getOrDefault(bucket, 0L);
            hashMap.put(bucket, orDefault + 1);
        }
        //查看结果
        int count = 0;
        for (Map.Entry<String, Long> key : hashMap.entrySet()) {
            if (key.getValue() > 512) {
                count++;
            }
            log.info("命中桶:" + key.getKey() + ",数量:" + key.getValue());
        }
        log.info("大于512的桶:{}个", count);

    }

    static CRC32 crc32 = new CRC32();

    public static String sharding(String shardingKey, ShardingLevel shardingLevel) {
        crc32.update(shardingKey.getBytes());
        String bucket = String.valueOf(Math.abs(crc32.getValue() % shardingLevel.getBucketCount()));
        crc32.reset();
        return bucket;
    }
}

上述的工具类其实比较简单，就是通过hash散列比较剧烈的CRC32，对分片key hash后对分片数量进行取余操作再取绝对值，结果理想

存取key 的时候通过sharding方法获取桶之后将key拼接为:key+bucket进行存取即可

ps~~~头一次写这么长的文章。。。真累