redis工具包开发——redis布隆过滤器实现

最新推荐文章于 2024-07-31 11:17:00 发布

LL小蜗牛

最新推荐文章于 2024-07-31 11:17:00 发布

阅读量1.4k

点赞数 2

分类专栏： redis

本文链接：https://blog.csdn.net/weixin_37703281/article/details/103936753

版权

redis 专栏收录该内容

10 篇文章 0 订阅

订阅专栏

布隆过滤器实现其实核心也是模仿guava的布隆过滤器实现，只是操作字节数组的载体改为redis的bitmap结构，在该工具包的布隆过滤器开发里面涉及到比较核心的点有

1.guava里的布隆过滤器理解

2.如何接入spring

3.redis 布隆过滤器实现

先看第一部分

布隆过滤器他能够判断某个对象肯定不存在或者可能存在，核心的思想是，对一个对象进行不同的hash函数处理所得到的值标志到相对应的字节数组中，比如key1 通过3个hash函数解析后，对应到下标为1，3，5，此时把该字节数组的相应的标志置为1，当判断一个新的key是否存在时，通过同样的计算去相应的下标查看是否都为1，如果是的话，就可能存在，为什么可能存在呢？当存入的key越来越多，字节数组里面的位为1也越来越多，布隆过滤器是有一定的误判率的。

了解好基本的概念后，这里贴一下布隆过滤器的一些公式

其中m为布隆过滤器的bitsize，n为预计插入的长度，p为误差率，k为hash函数的个数，guava里面是如何实现的呢？

主要有两个类

BloomFilter

该类的四个属性

其中bits是自定义的一个类似于bitset的类，操作通过cas完成，然后numHashFunctions是hash函数的个数，funnel类则是用于把对象转换成java基本的数据类型，stragtegy则是hash策略

先看创建的过程，首先是前置校验，然后到计算字节长度，hash函数个数

static <T> BloomFilter<T> create(
      Funnel<? super T> funnel, long expectedInsertions, double fpp, Strategy strategy) {
    checkNotNull(funnel);
    checkArgument(
        expectedInsertions >= 0, "Expected insertions (%s) must be >= 0", expectedInsertions);
    checkArgument(fpp > 0.0, "False positive probability (%s) must be > 0.0", fpp);
    checkArgument(fpp < 1.0, "False positive probability (%s) must be < 1.0", fpp);
    checkNotNull(strategy);

    if (expectedInsertions == 0) {
      expectedInsertions = 1;
    }
    
    long numBits = optimalNumOfBits(expectedInsertions, fpp);
    int numHashFunctions = optimalNumOfHashFunctions(expectedInsertions, numBits);
    try {
      return new BloomFilter<T>(new LockFreeBitArray(numBits), numHashFunctions, funnel, strategy);
    } catch (IllegalArgumentException e) {
      throw new IllegalArgumentException("Could not create BloomFilter of " + numBits + " bits", e);
    }
  }

计算字节数组长度，其实就是上面贴的公式的第一个条实现，入参为预计插入长度和错误率

static long optimalNumOfBits(long n, double p) {
    if (p == 0) {
      p = Double.MIN_VALUE;
    }
    return (long) (-n * Math.log(p) / (Math.log(2) * Math.log(2)));
  }

计算hash函数个数

 static int optimalNumOfHashFunctions(long n, long m) {
    // (m / n) * log(2), but avoid truncation due to division!
    return Math.max(1, (int) Math.round((double) m / n * Math.log(2)));
  }

然后到创建自己的一个字节数组底层是AtomicLongArray,通过cas操作，LongAddable用来统计数量，默认实现为LongAddr，出错后采用继承AtomicLong的自定义类来实现，funnel则是起到一些转换的作用，strategy是put和get的一些方法实现，对于64位和32位有一些细微的差别，这里以64位来演示

put方法通过通过funnel把对象转为对应的字节数组，然后取低八字节作为hash值1，高八字节作为hash值2，通过两者复合的hash做一定的与运算后对字节数组长度取模并设置相应的位

mightContain方法也是同样的方法取出相应的位数并判断


  MURMUR128_MITZ_64() {
    @Override
    public <T> boolean put(
        T object, Funnel<? super T> funnel, int numHashFunctions, LockFreeBitArray bits) {
      long bitSize = bits.bitSize();
      byte[] bytes = Hashing.murmur3_128().hashObject(object, funnel).getBytesInternal();
      long hash1 = lowerEight(bytes);
      long hash2 = upperEight(bytes);

      boolean bitsChanged = false;
      long combinedHash = hash1;
      for (int i = 0; i < numHashFunctions; i++) {
        // Make the combined hash positive and indexable
        bitsChanged |= bits.set((combinedHash & Long.MAX_VALUE) % bitSize);
        combinedHash += hash2;
      }
      return bitsChanged;
    }

    @Override
    public <T> boolean mightContain(
        T object, Funnel<? super T> funnel, int numHashFunctions, LockFreeBitArray bits) {
      long bitSize = bits.bitSize();
      byte[] bytes = Hashing.murmur3_128().hashObject(object, funnel).getBytesInternal();
      long hash1 = lowerEight(bytes);
      long hash2 = upperEight(bytes);

      long combinedHash = hash1;
      for (int i = 0; i < numHashFunctions; i++) {
        // Make the combined hash positive and indexable
        if (!bits.get((combinedHash & Long.MAX_VALUE) % bitSize)) {
          return false;
        }
        combinedHash += hash2;
      }
      return true;
    }

    private /* static */ long lowerEight(byte[] bytes) {
      return Longs.fromBytes(
          bytes[7], bytes[6], bytes[5], bytes[4], bytes[3], bytes[2], bytes[1], bytes[0]);
    }

    private /* static */ long upperEight(byte[] bytes) {
      return Longs.fromBytes(
          bytes[15], bytes[14], bytes[13], bytes[12], bytes[11], bytes[10], bytes[9], bytes[8]);
    }
  };

而这里主要把bitArray这个换成操作redis的bitArray，主要的命令有bitset，bitget,bitcount，具体的实现将在文章下面统一介绍讲解

如何接入spring？主要是用到了@Import注解

@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Import({RedisBloomFilterRegistar.class, RedisLimiterRegistar.class})
public @interface EnableRedisAux {
    String[] bloomFilterPath() default "";
    boolean enableLimit() default false;
    boolean transaction() default false;

}

spring会解析并加载@Import里的类，然后看一下布隆过滤器的注册类

public class RedisBloomFilterRegistar implements ImportBeanDefinitionRegistrar {
    public static Map<String, Map<String, BloomFilterProperty>> bloomFilterFieldMap;
    public static boolean transaction;

    @Override
    public void registerBeanDefinitions(AnnotationMetadata importingClassMetadata, BeanDefinitionRegistry registry) {
        Map<String, Object> attributes = importingClassMetadata
                .getAnnotationAttributes(EnableRedisAux.class.getCanonicalName());
        transaction = (boolean) attributes.get("transaction");
        String[] scanPaths = (String[]) attributes.get(BloomFilterConsts.SCAPATH);
        if (!scanPaths[0].trim().equals("")) {
            bloomFilterFieldMap = new HashMap<>();
            ClassPathScanningCandidateComponentProvider scanner = new ClassPathScanningCandidateComponentProvider(
                    false);
            scanner.addIncludeFilter(new AnnotationTypeFilter(BloomFilterPrefix.class));
            for (String basePath : scanPaths) {
                scanner.findCandidateComponents(basePath).forEach(e -> {
                    Class<?> clazz = null;
                    try {
                        clazz = Class.forName(e.getBeanClassName());
                    } catch (ClassNotFoundException e1) {
                        e1.printStackTrace();
                    }
                    if (clazz.isAnnotationPresent(BloomFilterPrefix.class)) {
                        Map<String, BloomFilterProperty> map = new HashMap();
                        String prefix = clazz.getAnnotation(BloomFilterPrefix.class).prefix();
                        ReflectionUtils.doWithFields(clazz, field -> {
                            field.setAccessible(Boolean.TRUE);
                            if (field.isAnnotationPresent(BloomFilterProperty.class)) {
                                String key = field.getName();
                                String keyName = CommonUtil.getKeyName(prefix, key);
                                map.put(keyName, field.getAnnotation(BloomFilterProperty.class));
                            }
                        });
                        bloomFilterFieldMap.put(prefix, map);
                    }
                });
            }
            bloomFilterFieldMap = Collections.unmodifiableMap(bloomFilterFieldMap);
        } else {
            System.err.println("=============redisbloomfilter未设置扫描路径，不支持lambda调用=============");
        }

        //指定扫描自己写的符合默认扫描注解的组件
        ClassPathBeanDefinitionScanner scanConfigure =
                new ClassPathBeanDefinitionScanner(registry, true);
        scanConfigure.scan(BloomFilterConsts.PATH);
    }

}

这个注册类第一个部分逻辑是存储一下支持lambda调用的类的属性信息

扫描配置的路径下有BloomFilterPrefix注解的类，然后把有BloomFilterProperty注解的属性信息存入一个map里面，该配置类的信息为<键前缀,<属性名,BloomfilterProperty信息>

解析lambda时，GetBloomFilterField这个类负责从lambda表达式里面获取对应的字段名称，再从上述的map里面获取到对应的信息封装成一个对象用于接下来的调用

public class GetBloomFilterField {

    private static Map<Class, SerializedLambda> map = new ConcurrentHashMap();
    private static Map<String, BloomFilterInfo> bloomFilterInfoMap = new ConcurrentHashMap();

    public static <T> BloomFilterInfo resolveFieldName(SFunction<T> sFunction) {
        SerializedLambda lambda = map.get(sFunction.getClass());
        if (lambda == null) {
            try {
                Method writeReplace = sFunction.getClass().getDeclaredMethod(BloomFilterConstants.LAMBDAMETHODNAME);
                writeReplace.setAccessible(true);
                lambda = (SerializedLambda) writeReplace.invoke(sFunction);
            } catch (NoSuchMethodException e) {
                e.printStackTrace();
            } catch (IllegalAccessException e) {
                e.printStackTrace();
            } catch (InvocationTargetException e) {
                e.printStackTrace();
            }
            map.put(sFunction.getClass(), lambda);
        }
        String fieldName = getFieldName(lambda);
        String capturingClass = lambda.getImplClass();
        String infoKey = CommonUtil.getKeyName(fieldName, capturingClass);
        BloomFilterInfo res = bloomFilterInfoMap.get(infoKey);
        if (res == null) {
            capturingClass = capturingClass.replace("/", ".");
            Class<?> aClass = null;
            try {
                aClass = Class.forName(capturingClass);
            } catch (ClassNotFoundException e) {
                e.printStackTrace();
            }
            if (aClass.isAnnotationPresent(BloomFilterPrefix.class)) {
                BloomFilterPrefix annotation = aClass.getAnnotation(BloomFilterPrefix.class);
                if (RedisBloomFilterRegistar.bloomFilterFieldMap != null) {
                    Map<String, BloomFilterProperty> map = RedisBloomFilterRegistar.bloomFilterFieldMap.get(annotation.prefix());
                    BloomFilterProperty field = map.get(CommonUtil.getKeyName(annotation.prefix(), fieldName));
                    res = new BloomFilterInfo(annotation.prefix().trim().equals("") ? aClass.getCanonicalName() : annotation.prefix(),
                            field.key().trim().equals("") ? fieldName : field.key(),
                            field.exceptionInsert(),
                            field.fpp(),
                            field.timeout(),
                            field.timeUnit(),

                    field.local()
                    );
                }
            }
            if (res != null) {
                bloomFilterInfoMap.put(infoKey, res);
            }

        }
        return res;

    }

    public static class BloomFilterInfo {
        private final String keyPrefix;
        private final String keyName;
        private final long exceptionInsert;
        private final double fpp;
        private final long timeout;
        private final TimeUnit timeUnit;
        private final boolean local;

        public BloomFilterInfo(String keyPrefix, String keyName, Long exceptionInsert, double fpp, Long timeout, TimeUnit timeUnit, boolean local) {
            this.keyPrefix = keyPrefix;
            this.keyName = keyName;
            this.exceptionInsert = exceptionInsert;
            this.fpp = fpp;
            this.timeout = timeout;
            this.timeUnit = timeUnit;
            this.local=local;

        }

        public String getKeyPrefix() {
            return keyPrefix;
        }

        public String getKeyName() {
            return keyName;
        }

        public long getExceptionInsert() {
            return exceptionInsert;
        }

        public double getFpp() {
            return fpp;
        }

        public long getTimeout() {
            return timeout;
        }

        public TimeUnit getTimeUnit() {
            return timeUnit;
        }

        public boolean isLocal(){return local;}
    }


    private static String getFieldName(SerializedLambda lambda) {
        String getMethodName = lambda.getImplMethodName();
        if (getMethodName.startsWith(BloomFilterConstants.GET)) {
            getMethodName = getMethodName.substring(3);
        } else if (getMethodName.startsWith(BloomFilterConstants.IS)) {
            getMethodName = getMethodName.substring(2);
        } else {
            throw new RedisAuxException("没有相应的属性方法");
        }
        // 小写第一个字母
        return Character.isLowerCase(getMethodName.charAt(0)) ? getMethodName : Character.toLowerCase(getMethodName.charAt(0)) + getMethodName.substring(1);
    }
}

第二部分，扫描本应用的某些路径，这样就把对应的资源加载

主要加载有两个类

第一个是内部的redisTemplate，这里就不多说

第二个是布隆过滤器的配置类，@ConditionalOnClass,@AutoConfigureAfter都是一些前置工作，规定该配置类在什么样的情况下才加载，然后就加载相应的类

@Configuration
@ConditionalOnClass(RedisBloomFilter.class)
@AutoConfigureAfter(RedisAutoConfiguration.class)
@SuppressWarnings("unchecked")
class RedisBloomFilterAutoConfiguration {

    @Autowired
    @Qualifier(BloomFilterConstants.INNERTEMPLATE)
    private RedisTemplate redisTemplate;



    @Bean
    @ConditionalOnMissingBean(RedisBloomFilter.class)
    public RedisBloomFilter redisBloomFilter() {
        Properties properties = System.getProperties();
        String property = properties.getProperty("sun.arch.data.model");
        Strategy strategy = RedisBloomFilterStrategies.getStrategy(property);
        if (strategy == null) {
            strategy = RedisBloomFilterStrategies.MURMUR128_MITZ_32.getStrategy();
        }
        Map<Class, RedisBloomFilterItem> map = new HashMap(FunnelEnum.values().length);
        for (FunnelEnum funnelEnum : FunnelEnum.values()) {
            RedisBloomFilterItem item = RedisBloomFilterItem.create(funnelEnum.getFunnel(), strategy, redisBitArrayFactory());
            checkTask().addListener(item);
            map.put(funnelEnum.getCode(), item);
        }
        return new RedisBloomFilter(map);
    }

    @Bean(name = "resetBitScript")
    public DefaultRedisScript resetBitScript() {
        DefaultRedisScript<Void> script = new DefaultRedisScript<Void>();
        script.setScriptText(resetBitScriptStr());
        return script;
    }

    @Bean(name = "setBitScript")
    public DefaultRedisScript setBitScript() {
        DefaultRedisScript script = new DefaultRedisScript();
        script.setScriptText(setBitScripStr());
        return script;
    }

    @Bean(name = "getBitScript")
    public DefaultRedisScript getBitScript() {
        DefaultRedisScript<List> script = new DefaultRedisScript();
        script.setScriptText(getBitScriptStr());
        script.setResultType(List.class);
        return script;
    }


    @Bean
    public BitArrayOperator redisBitArrayFactory() {

        return new BitArrayOperator(
                setBitScript(),
                getBitScript(),
                resetBitScript(),
                redisTemplate,
                checkTask()
        );
    }

    @Bean
    public CheckTask checkTask() {
        return new CheckTask();
    }


    private String setBitScripStr() {
        StringBuilder builder = new StringBuilder();
        builder.append("local kL = table.getn(KEYS)\n")
                .append("local aL = table.getn(ARGV)\n")
                .append("for i = 1, kL\n")
                .append("do\n")
                .append("    for k = 1, aL\n")
                .append("    do redis.call('setbit', KEYS[i], tonumber(ARGV[k]), 1)\n")
                .append("    end\nend");
        return builder.toString();
    }

    private String getBitScriptStr() {
        StringBuilder builder = new StringBuilder();
        builder.append("local array = {}\n").append("local aL = table.getn(ARGV) - 1\n")
                .append("local kL = table.getn(KEYS)\n").append("local bitL = ARGV[1]\n")
                .append("local elementSize = aL / bitL\n").append("for index = 1, elementSize\n")
                .append("do local notAdd = true\n").append("    for i = (index - 1) * bitL + 2, index * bitL + 1\n")
                .append("    do\n").append("        local k = 1\n").append("        while (notAdd and k <= kL)\n")
                .append("        do\n").append("            if redis.call('getbit', KEYS[k], ARGV[i]) == 0 then\n")
                .append("                array[index] = 0\n").append("                notAdd = false\n")
                .append("            else\n").append("                k = k + 1\n")
                .append("            end\n").append("        end\n").append("    end\n")
                .append("    if notAdd then array[index] = 1 end\n").append("end\nreturn array");
        return builder.toString();
    }

    private String resetBitScriptStr() {
        StringBuilder builder = new StringBuilder();
        builder.append("local key = KEYS[1]\n")
                .append("local start, last = 0, tonumber(ARGV[1])\n")
                .append("local b = '\\0'\n")
                .append("local mstart, mlast = start % 8, (last + 1) % 8\n")
                .append("if mlast > 0 then\n")
                .append("    local t = math.max(last - mlast + 1, start)\n")
                .append("    for i = t, last do\n")
                .append("        redis.call('SETBIT', key, i, b)\n    end\n    last = t\nend\n")
                .append("if mstart > 0 then\n")
                .append("    local t = math.min(start - mstart + 7, last)\n")
                .append("    for i = start, t do\n")
                .append("        redis.call('SETBIT', key, i, b)\n    end\n    start = t + 1\nend\n")
                .append("local rs, re = start / 8, (last + 1) / 8\nlocal rl = re - rs\nif rl > 0 then\n")
                .append("    redis.call('SETRANGE', key, rs, string.rep(b, rl))\nend\n");
        return builder.toString();
    }


}

这就是和spring接入的方式，主要是通过@Import和自定义扫描路径加载对应的配置类把相关的类给spring托管

3.redis 布隆过滤器实现

目前写的功能有

1.添加、批量添加

2.查询、批量查询

3.重置

4.键过期

这里的设计是这样的

对外提供一个RedisBloomFilter,但里面有一个map，该map存放class和对应的操作类，操作类通过strategy类来确定对应的hash值，然后通过这些hash值操作字节数组。

RedisBloomFilter类

public class RedisBloomFilter {
    private final Map<Class, RedisBloomFilterItem> bloomFilterMap;


    public RedisBloomFilter(Map<Class, RedisBloomFilterItem> bloomFilterMap) {
        this.bloomFilterMap = bloomFilterMap;
    }

    /**
     * 通过解析lambda获取信息
     *
     * @param sFunction
     * @param member
     * @param <T>
     */
    public <T, R> void add(SFunction<T> sFunction, R member) {
        GetBloomFilterField.BloomFilterInfo bloomFilterInfo = check(sFunction);
        add(bloomFilterInfo.getKeyPrefix(),
                bloomFilterInfo.getKeyName(),
                bloomFilterInfo.getExceptionInsert(),
                bloomFilterInfo.getFpp(),
                bloomFilterInfo.getTimeout(),
                bloomFilterInfo.getTimeUnit(),
                bloomFilterInfo.isLocal(),
                member);
    }

    public <R> void add(AddCondition addCondition, R member) {
        InnerInfo condition = addCondition.build();
        addCondition.clear();
        add(condition.getKeyPrefix(),
                condition.getKeyName(),
                condition.getExceptionInsert(),
                condition.getFpp(),
                condition.getTimeout(),
                condition.getTimeUnit(),
                condition.isLocal(),
                member
        );
    }

    private <R> void add(String keyPrefix, String key, long exceptedInsertions, double fpp, long timeout, TimeUnit timeUnit, boolean local,R member) {
        Class clzz = member.getClass();
        Object res = member;
        RedisBloomFilterItem filter = bloomFilterMap.get(clzz);
        String keyName = checkKey(keyPrefix, key);
        if (filter == null) {
            filter = bloomFilterMap.get(Byte.class);
        }
        filter.put(keyName, res, exceptedInsertions, fpp, timeout, timeUnit,local);
    }

    public <T, R> void addAll(SFunction<T> sFunction, List<R> members) {
        GetBloomFilterField.BloomFilterInfo bloomFilterInfo = check(sFunction);
        addAll(bloomFilterInfo.getKeyPrefix(),
                bloomFilterInfo.getKeyName(),
                bloomFilterInfo.getExceptionInsert(),
                bloomFilterInfo.getFpp(),
                bloomFilterInfo.getTimeout(),
                bloomFilterInfo.getTimeUnit(),
                bloomFilterInfo.isLocal(),
                members);
    }

    public <R> void addAll(AddCondition addCondition, List<R> members) {
        InnerInfo innerInfo = addCondition.build();
        addCondition.clear();
        this.addAll(
                innerInfo.getKeyPrefix(),
                innerInfo.getKeyName(),
                innerInfo.getExceptionInsert(),
                innerInfo.getFpp(),
                innerInfo.getTimeout(),
                innerInfo.getTimeUnit(),
                innerInfo.isLocal(),
                members
        );
    }

    private <R> void addAll(String keyPrefix, String key, Long exceptedInsertions, Double fpp, long timeout, TimeUnit timeUnit, boolean local, List<R> members) {
        if (members.isEmpty()) {
            throw new RedisAuxException("参数有误!");
        }
        String keyName = checkKey(keyPrefix, key);
        Class clzz = members.get(0).getClass();
        RedisBloomFilterItem filter = bloomFilterMap.get(clzz);
        List<Object> resList = new ArrayList(members.size());
        for (R member : members) {
            resList.add(member);
        }
        if (filter == null) {
            filter = bloomFilterMap.get(Byte.class);
        }
        filter.putAll(keyName, exceptedInsertions, fpp, resList, timeout, timeUnit,local);
    }

    public <R> boolean mightContain(BaseCondition queryCondition, R member) {
        InnerInfo build = queryCondition.build();
        return mightContain(build.getKeyPrefix(), build.getKeyName(), member);
    }

    public <T, R> boolean mightContain(SFunction<T> sFunction, R member) {
        GetBloomFilterField.BloomFilterInfo bloomFilterInfo = check(sFunction);
        return mightContain(bloomFilterInfo.getKeyPrefix(), bloomFilterInfo.getKeyName(), member);
    }

    private <R> boolean mightContain(String keyPrefix, String key, R member) {
        String keyName = checkKey(keyPrefix, key);
        Class clzz = member.getClass();
        RedisBloomFilterItem filter = bloomFilterMap.get(clzz);
        if (filter == null) {
            filter = bloomFilterMap.get(Byte.class);
        }
        return filter.mightContain(keyName, member);
    }


    public <T, R> List<Boolean> mightContains(SFunction<T> sFunction, List<R> members) {
        GetBloomFilterField.BloomFilterInfo bloomFilterInfo = check(sFunction);
        return mightContains(bloomFilterInfo.getKeyPrefix(), bloomFilterInfo.getKeyName(), members);
    }

    public <R> List<Boolean> mightContains(BaseCondition queryCondition, List<R> members) {
        InnerInfo build = queryCondition.build();
        return mightContains(build.getKeyPrefix(), build.getKeyName(), members);
    }

    private <R> List<Boolean> mightContains(String keyPrefix, String key, List<R> members) {
        if (members.isEmpty()) {
            return null;
        }
        String keyName = checkKey(keyPrefix, key);
        Class clzz = members.get(0).getClass();
        RedisBloomFilterItem filter = bloomFilterMap.get(clzz);
        List<Object> resList = new ArrayList(members.size());
        for (R member : members) {
            resList.add(member);
        }
        if (filter == null) {
            filter = bloomFilterMap.get(Byte.class);
        }
        return filter.mightContains(keyName, resList);
    }


    public <T> void remove(SFunction<T> sFunction) {
        GetBloomFilterField.BloomFilterInfo bloomFilterInfo = check(sFunction);
        remove(bloomFilterInfo.getKeyPrefix(), bloomFilterInfo.getKeyName());

    }

    public void remove(BaseCondition deleteCondition) {
        InnerInfo build = deleteCondition.build();
        remove(build.getKeyPrefix(), build.getKeyName());
    }


    private void remove(String keyPrefix, String key) {
        String keyname = checkKey(keyPrefix, key);
        for (RedisBloomFilterItem filter : bloomFilterMap.values()) {
            filter.remove(keyname);
        }
    }

    /**
     * 这里因为不同类对应不同的item存放，所以
     *
     * @param keys
     * @param keyPrefix
     */
    public void removeAll(Collection<String> keys, String keyPrefix) {

        List<String> keyList = new LinkedList();
        for (String key : keys) {
            keyList.add(checkKey(keyPrefix, key));
        }
        for (RedisBloomFilterItem filter : bloomFilterMap.values()) {
            filter.removeAll(keyList);
        }
    }

    public void removeAll(Collection<String> keys) {
        removeAll(keys, null);
    }


    public <T> void reset(SFunction<T> sFunction) {
        GetBloomFilterField.BloomFilterInfo bloomFilterInfo = GetBloomFilterField.resolveFieldName(sFunction);
        reset(bloomFilterInfo.getKeyPrefix(), bloomFilterInfo.getKeyName());
    }

    public void reset(BaseCondition resetCondition) {
        InnerInfo build = resetCondition.build();
        reset(build.getKeyPrefix(), build.getKeyName());
    }


    private void reset(String keyPrefix, String keyName) {
        keyName = checkKey(keyPrefix, keyName);
        for (RedisBloomFilterItem filter : bloomFilterMap.values()) {
            filter.reset(keyName);
        }
    }

    public void expire(ExpireCondition expireCondition) {
        InnerInfo condition = expireCondition.build();
        expireCondition.clear();
        expire(condition.getKeyPrefix(), condition.getKeyName(), condition.getTimeout(), condition.getTimeUnit(),condition.isLocal());
    }

    private void expire(String keyPrefix, String keyName, long timeout, TimeUnit timeUnit,boolean local) {
        keyName = checkKey(keyPrefix, keyName);
        for (RedisBloomFilterItem filter : bloomFilterMap.values()) {
            filter.expire(keyName, timeout, timeUnit,local);
        }
    }

    private GetBloomFilterField.BloomFilterInfo check(SFunction sFunction) {
        GetBloomFilterField.BloomFilterInfo bloomFilterInfo = GetBloomFilterField.resolveFieldName(sFunction);
        if (bloomFilterInfo == null) {
            throw new RedisAuxException("请检查注解配置是否正确!");
        }
        return bloomFilterInfo;
    }

    private String checkKey(String prefix, String key) {
        return StringUtils.isEmpty(prefix) ? key : CommonUtil.getKeyName(prefix, key);
    }

    @PreDestroy
    protected void destory() {
        for (RedisBloomFilterItem value : this.bloomFilterMap.values()) {
            value.clear();
        }
        this.bloomFilterMap.clear();
    }

    private boolean containKey(String keyPrefix,String keyName){
        String key = checkKey(keyPrefix, keyName);
        for (RedisBloomFilterItem filter : bloomFilterMap.values()) {
           if(filter.containKey(key)){
               return true;
           };
        }
        return false;
    }
    public boolean containKey(BaseCondition condition){
        return this.containKey(condition.keyPrefix,condition.keyName);
    }
    public <T> boolean containKey(SFunction<T> sFunction){
        GetBloomFilterField.BloomFilterInfo bloomFilterInfo = GetBloomFilterField.resolveFieldName(sFunction);
        return this.containKey(bloomFilterInfo.getKeyPrefix(),bloomFilterInfo.getKeyName());
    }
}

具体项，每个class对应一个操作类，这层缓存了hash个数，字节操作类，然后stragey类计算对应的hash并操作这个字节操作类

 @Override
    public boolean set(long[] index) {
        Object[] value = Arrays.stream(index).boxed().toArray();
        redisTemplate.execute(setBitScript, keyList, value);
        return Boolean.TRUE;
    }

这里把keyList直接作为参数，value列表则是对应的set的位置下标，然后执行lua脚本，通过该脚本执行

local kL=table.getn(KEYS)
local aL=table.getn(ARGV)
for i = 1, kL
do
    for k = 1,aL
    do redis.call('setbit', KEYS[i], tonumber(ARGV[k]), 1)
    end
end

设置单个的大致思路就是如此，然后批量设置其实也差不多，只要把对应的位数算出来设置就好了

然后到判断：

单个判断,这里只要判断取出来位置，是否都是1即可

 @Override
    public boolean get(long[] index) {
        List<Long> res = getBitScriptExecute(index, index.length);
        boolean exists = res.get(0).equals(BloomFilterConstants.TRUE);
        return exists;
    }

这里的size是单个元素被hash后的长度，放在value数组里面里面，然后可以复用keyList对象

private List<Long> getBitScriptExecute(long[] index,int size) {
        Object[] value = new Long[index.length+1];
        value[0]=Long.valueOf(size);
        for (int i = 1; i < value.length; i++) {
            value[i] = Long.valueOf(index[i-1]);
        }
        List res = (List) redisTemplate.execute(getBitScript, keyList, value);
        return res;
    }

取的脚本，这个比较核心，这里的做法是先整理成一个大一维数组，然后先取出每个元素的下标位置，再判断这些范围里面，本身和副本上面的位置是否有0，有就代表该元素不存在

--ARGV[1]代表单个待判断元素对应的字节长度，
-- 索引 1-length 代表键，0为不存在，1为存在
local array = {}
local aL = table.getn(ARGV)-1
local kL = table.getn(KEYS)
local bitL = ARGV[1]
--把单个元素所对应的下标取出来
local elementSize = aL / bitL
--第一个元素2...ARGV[1]+1,第二个ARGV[1]+2...2ARGV[1]+1
for index = 1, elementSize
    --判断单个元素的情况
do local notAdd = true
    for i = (index - 1) * bitL + 2, index * bitL+1
        --判断对应的键下标是否都为true
    do
        local k = 1
        while (notAdd and k <= kL)
        do
            if redis.call('getbit', KEYS[k], ARGV[i]) == 0 then
                array[index] = 0
                notAdd = false
            else
                k = k + 1
            end
        end
    end
    if notAdd then array[index] = 1 end
end
return array

批量取

   public List<Boolean> getBatch(List index) {
        //index.size*keyList.size
        readWriteLock.readLock().lock();
        long[] array = getArrayFromList(index);
        List<Long> list = getBitScriptExecute(array,((long[]) index.get(0)).length);
        List<Boolean> res = new ArrayList(index.size());
        for (Long temp : list) {
            res.add(Boolean.valueOf(temp.equals(BloomFilterConsts.TRUE)));
        }
        readWriteLock.readLock().unlock();
        return res;
    }

然后到重置功能，重置主要是lua脚本逻辑实现，脚本如下


local key = KEYS[1]
local start, last = 0, tonumber(ARGV[1])

local b = '\0'
-- 把多余的头和尾用setbit处理
local mstart, mlast = start % 8, (last + 1) % 8
--如果尾部有多出来的字节,采用单个字符处理并且把尾指针改变
if mlast > 0 then
    --获取尾指针的地址
    local t = math.max(last - mlast + 1, start)
    for i = t,last do
        redis.call('SETBIT', key, i, b)
    end
    last = t
end

-- 处理开头多余出来的部分
if mstart > 0 then
    local t = math.min(start - mstart + 7, last)
    for i = start, t do
        redis.call('SETBIT', key, i, b)
    end
    start = t + 1
end

-- 设置剩余范围
local rs, re = start / 8, (last + 1) / 8
local rl = re - rs
if rl > 0 then
    --string.rep拼接功能
    redis.call('SETRANGE', key, rs, string.rep(b, rl))
end

最后到一个键过期的功能，主要是判断是否设置了过期时间，如果不为-1，则在添加键后配置过期时间，然后用一个线程去定时处理过期键，这个定时在没有任务处理时，会每5s检查一次是否需要处理，如果有的话，就把当前的处理好后，睡眠直到下次键过期（通过最小堆来把快要过期的键放在顶端，然后睡眠相应的时间），这里的做处理是指由于上面的一些缓存的存在，当键过期时，对应的缓存也要删除，所以这里用了监听者模式来通知对应的类做对应的清理工作

public class CheckTask extends Thread implements KeyExpirePublisher, InitializingBean {
    private List<KeyExpireListener> listeners = new ArrayList<>();
    private PriorityBlockingQueue<WatiForDeleteKey> priorityQueue;
    private volatile Boolean run = true;
    private ThreadPoolExecutor executors;

    public CheckTask() {
        super("checkTask");
        this.priorityQueue = new PriorityBlockingQueue<>();
        executors = new ThreadPoolExecutor(4, 4, 0, TimeUnit.SECONDS, new ArrayBlockingQueue<>(1024), new ThreadPoolExecutor.CallerRunsPolicy());
    }

    @Override
    public void run() {
        while (run) {
            //当队列为空时，每5秒检测一次
            if (priorityQueue.isEmpty()) {
                try {
                    TimeUnit.SECONDS.sleep(CHECK_TASK_PER_SECOND);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            } else {
                //查看存活时间最小的那个节点
                WatiForDeleteKey peek = priorityQueue.peek();
                long dispearTime = 0;
                long now = 0;
                //针对键的存活时间较为密切的情况，一直弹出直到存活时间大于当前时间
                while (peek != null && (dispearTime = (peek.getStartTime() + peek.getExistTime())) <= (now = System.currentTimeMillis())) {
                    WatiForDeleteKey poll = priorityQueue.poll();
                    notifyListener(poll.getKey());
                    peek = priorityQueue.peek();
                }
                //等待下一次过期的时间
                try {
                    if (dispearTime > now) {
                        TimeUnit.MILLISECONDS.sleep(dispearTime - now);
                    }
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }

            }
        }
    }

    public  void addExpireKey(WatiForDeleteKey watiForDeleteKey) {
        priorityQueue.offer(watiForDeleteKey);
    }

    @Override
    public void addListener(KeyExpireListener listener) {
        listeners.add(listener);
    }

    @Override
    public void removeListener(KeyExpireListener listener) {
        listeners.remove(listener);
    }

    @Override
    public void notifyListener(String key) {
//通过线程池提交删除任务
        for (KeyExpireListener listener : listeners) {
            executors.submit(() -> listener.removeKey(key));
        }

    }

    @Override
    public void afterPropertiesSet() throws Exception {
        this.start();
    }

    @PreDestroy
    public void stopRun() {
        this.run = Boolean.FALSE;
        executors.shutdown();
    }
}

过期键信息

public class WatiForDeleteKey implements Comparable {
    private String key;
    private long existTime;
    private long startTime;

    public String getKey() {
        return key;
    }

    public long getExistTime() {
        return existTime;
    }

    public long getStartTime() {
        return startTime;
    }

    public WatiForDeleteKey(String key, long existTime, long startTime) {
        this.key = key;
        this.existTime = existTime;
        this.startTime = startTime;
    }

    @Override
    public int compareTo(Object o) {
        if (o instanceof WatiForDeleteKey) {
            long now = System.currentTimeMillis();
            //剩下的存活时间
            WatiForDeleteKey ot = (WatiForDeleteKey) o;
            long other = ot.existTime - (now - ot.startTime);

            long self = existTime - (now - startTime);
            //小顶堆，在上面是最小的，存活时间最少在上面
            return self<other?-1:(self==other)?0:1;
        }
        throw new RedisAuxException("WaitForDeleteKey无法比对");
    }

    @Override
    public String toString() {
        return key+":"+((startTime+existTime)-System.currentTimeMillis());
    }
}

上面的bloomfilterItem类就是监听者类，通过一开始的配置初始化来为checkTask添加监听者。

另外附上本地字节数组

public class LocalBitArray implements BitArray {

    private static final int LONG_ADDRESSABLE_BITS = 6;
    private AtomicLongArray data;
    private LongAdder bitCount;
    private final long bitSize;
    private final String key;

    public LocalBitArray(String key,Long bitSize) {
        long[] longs = new long[Ints.checkedCast(LongMath.divide(bitSize, 64, RoundingMode.CEILING))];
        this.data = new AtomicLongArray(longs);
        this.bitCount = new LongAdder();
        long bitCount = 0;
        this.bitCount.add(bitCount);
        this.bitSize = bitSize;
        this.key=key;
    }




    @Override
    public boolean set(long[] indexs) {
        for (long bitIndex : indexs) {
            if (!getBitIndex(bitIndex, data) && setBitIndex(bitIndex, data)) {
                bitCount.increment();
            }
        }
        return true;
    }


    @Override
    public boolean get(long[] indexs) {

        for (long index : indexs) {
            if (!getBitIndex(index, data)) {
                return false;
            }
        }
        return true;
    }

    private boolean getBitIndex(long bitIndex, AtomicLongArray data) {
        return (data.get((int) (bitIndex >>> LONG_ADDRESSABLE_BITS)) & (1L << bitIndex)) != 0;
    }

    private boolean setBitIndex(long bitIndex, AtomicLongArray data) {
        int longIndex = (int) (bitIndex >>> LONG_ADDRESSABLE_BITS);
        // only cares about low 6 bits of bitIndex
        long mask = 1L << bitIndex;
        long oldValue;
        long newValue;
        do {
            oldValue = data.get(longIndex);
            newValue = oldValue | mask;
            if (oldValue == newValue) {
                return false;
            }
        } while (!data.compareAndSet(longIndex, oldValue, newValue));
        return true;
    }

    @Override
    public long bitSize() {
        return (long) data.length() * Long.SIZE;
    }

    @Override
    public void reset() {
        long[] longs = new long[Ints.checkedCast(LongMath.divide(bitSize, 64, RoundingMode.CEILING))];
        this.bitCount = new LongAdder();
        this.data = new AtomicLongArray(longs);
    }

    @Override
    public void clear() {
        this.data = null;
        this.bitCount = null;
    }

    @Override
    public String getKey() {
        return this.key;
    }



    @Override
    public List<Boolean> getBatch(List indexs) {
        List<Boolean> list = new ArrayList(indexs.size());
        for (Object o : indexs) {
            long[] temp = (long[]) o;
            boolean b = get(temp);
            list.add(b);
        }
        return list;
    }

    @Override
    public boolean setBatch(List indexs) {
        for (Object o : indexs) {
            long[] temp = (long[]) o;
            set(temp);
        }
        return true;
    }



}

大概的实现思路就是这样，关于简单的实现，可以查看我的另一篇文章，但功能相对较少