Redis高级类型(统计全站访问量，日活跃用户)

• HyperLogLog(超级日志)(统计访客)

• • 采用一种基数算法，用于完成独立总数的统计
  • 占据空间小，无论统计多少个数据，只占12K的内存空间
  • 不精确的统计算法，标准误差为0.81%

• Bitmap(位图)

• • 不是一种独立的数据结构，实际上就是字符串
  • 支持按位存取数据，可以将其看成是byte数组
  • 适合存储大量的连续的数据的布尔值

两种数据结构都适合用来统计网站运营数据，节约内存。

HyperLogLog

测试使用HyperLogLog来统计200000次访问去重

/**
     * 统计20万重复数据的独立总数
     */
    @Test
    public void  testHyperLogLog(){
        String redisKey = "test:h11:01";
        for (int i = 1; i <= 100000; i++) {
            redisTemplate.opsForHyperLogLog().add(redisKey,i);
        }
        for (int i = 1; i <= 100000; i++) {
            redisTemplate.opsForHyperLogLog().add(redisKey,(int)Math.random()*100000+1);
        }

        System.out.println(redisTemplate.opsForHyperLogLog().size(redisKey));
    }

结果：

虽然有误差但是完全可以接受

测试使用HyperLogLog来统计三组数据合并去重

   //将三组数据合并，在合并后的重复数据中统计独立总数
    @Test
    public void testHyperLogLogUnion(){
        String redisKey1 = "test:h11:02";

        for (int i = 1; i <= 10000 ; i++) {
            redisTemplate.opsForHyperLogLog().add(redisKey1,i);
        }

        String redisKey2 = "test:h11:02";
        for (int i = 5001; i <= 15000 ; i++)
            redisTemplate.opsForHyperLogLog().add(redisKey2,i);
        String redisKey3 = "test:h11:02";
        for (int i = 10001; i <= 20000 ; i++)
            redisTemplate.opsForHyperLogLog().add(redisKey3,i);

        String redisKey = "test:h11:union";
        redisTemplate.opsForHyperLogLog().union(redisKey,redisKey1,redisKey2,redisKey3);

       System.out.println( redisTemplate.opsForHyperLogLog().size(redisKey) );
    }

将数据合并并且进行查重，结果

**
**

BitMap操作

可以用来统计签到数据

测试使用BitMap来统计数据

  @Test
    public void testBitMap(){
        String redisKey = "test:bm:01";
        //记录 签到
        redisTemplate.opsForValue().setBit(redisKey,1,true);
        redisTemplate.opsForValue().setBit(redisKey,4,true);
        redisTemplate.opsForValue().setBit(redisKey,7,true);

        //查询
        System.out.println(redisTemplate.opsForValue().getBit(redisKey,0));
        System.out.println(redisTemplate.opsForValue().getBit(redisKey,1));
        System.out.println(redisTemplate.opsForValue().getBit(redisKey,2));

        //统计
        Object obj = redisTemplate.execute( (RedisCallback) connection ->{
            return connection.bitCount(redisKey.getBytes());
        });
        System.out.println(obj);
    }

需要使用redisCallback方法，传入key值的byte数组

测试使用BitMap进行OR运算

 //统计3组数据的布尔值，并对这3组数据做OR运算
    @Test
    public void testBitMapOperation(){
        String rediskey = "test:bm:02";
        redisTemplate.opsForValue().setBit(rediskey,0,true);
        redisTemplate.opsForValue().setBit(rediskey,1,true);
        redisTemplate.opsForValue().setBit(rediskey,2,true);

        String rediskey2 = "test:bm:03";
        redisTemplate.opsForValue().setBit(rediskey2,2,true);
        redisTemplate.opsForValue().setBit(rediskey2,3,true);
        redisTemplate.opsForValue().setBit(rediskey2,4,true);

        String rediskey3 = "test:bm:04";
        redisTemplate.opsForValue().setBit(rediskey3,4,true);
        redisTemplate.opsForValue().setBit(rediskey3,5,true);
        redisTemplate.opsForValue().setBit(rediskey3,6,true);

        String rediskey4 = "test:or";
        Object obj = redisTemplate.execute( (RedisCallback) connection ->{
             connection.bitOp(RedisStringCommands.BitOperation.OR,
                    rediskey4.getBytes(),rediskey.getBytes(),rediskey2.getBytes(),rediskey3.getBytes());
             return connection.bitCount(rediskey4.getBytes());
        });
        System.out.println(obj);
        System.out.println(redisTemplate.opsForValue().getBit(rediskey4,0));
        System.out.println(redisTemplate.opsForValue().getBit(rediskey4,1));
        System.out.println(redisTemplate.opsForValue().getBit(rediskey4,2));
        System.out.println(redisTemplate.opsForValue().getBit(rediskey4,3));
        System.out.println(redisTemplate.opsForValue().getBit(rediskey4,4));
        System.out.println(redisTemplate.opsForValue().getBit(rediskey4,5));
        System.out.println(redisTemplate.opsForValue().getBit(rediskey4,6));
    }

结果

Redis高级数据类型实战

在项目中用作网站数据统计

• UV(Unique Visitor)

• • 独立访客，需通过用户IP排重统计数据
  • 每次访问都要进行统计
  • HyperLogLog，性能好，且存储空间小

• DAU(Daily Active User)

• • 日活跃用户，通过用户ID排重统计数据。
  • 访问过一次，认为其活跃
  • Bitmap，性能好、且可以统计精确的结果

UV实战
redis已经将基本操作封装完毕，所以我们直接编写service即可

/**
 * 提供统计网站访客功能
 * @author ：LY
 * @date ：Created in 2021/3/11 22:20
 * @modified By：
 */
@Service
public class DataService {

    @Autowired
    private RedisTemplate redisTemplate;

    private SimpleDateFormat df = new SimpleDateFormat("yyyyMMdd");

    /**
     * 将指定IP计入UV
     * @param ip
     */
    public void  recordUV(String ip){
        String redisKey = RedisKeyUtil.getUVKey(df.format(new Date()));
        redisTemplate.opsForHyperLogLog().add(redisKey,ip);

    }

    /**
     * 统计置顶范围内的id数量
     * @param start
     * @param end
     * @return
     */
    public long calculateUV(Date start,Date end){
        if (start == null || end == null){
            throw  new IllegalArgumentException("参数不能为空");
        }
        //整理该日期范围的Key
        List<String> keyList = new ArrayList<>();
        Calendar calendar = Calendar.getInstance();
        calendar.setTime(start);
        while (!calendar.getTime().after(end)){
           String key = RedisKeyUtil.getUVKey(df.format(calendar.getTime()));
           keyList.add(key);
           calendar.add(Calendar.DATE,1);

        }
        //合并数据
        String redisKey = RedisKeyUtil.getUVKey(df.format(start),df.format(end));
        redisTemplate.opsForHyperLogLog().union(redisKey,keyList.toArray());
        //返回统计结果
        return redisTemplate.opsForHyperLogLog().size(redisKey);
    }


    /**
     * 统计日活跃用户
     * @param userId
     */
    public void recordDAU(int userId){
        String redisKey = RedisKeyUtil.getDAUKey(df.format(new Date()));
        redisTemplate.opsForValue().setBit(redisKey,userId,true);

    }

    /**
     * 统计指定日期范围的DAU数量
     * @param start
     * @param end
     * @return
     */
    public long calculateDAU(Date start,Date end){
        if (start == null || end == null){
            throw  new IllegalArgumentException("参数不能为空");
        }
        //整理该日期范围的Key
        List<byte[]> keyList = new ArrayList<>();
        Calendar calendar = Calendar.getInstance();
        calendar.setTime(start);
        while (!calendar.getTime().after(end)){
            String key =  RedisKeyUtil.getDAUKey(df.format(calendar.getTime()));
            keyList.add(key.getBytes());
            calendar.add(Calendar.DATE,1);
        }
        //进行OR运算
        return (long) redisTemplate.execute((RedisCallback)  connection -> {
            String redisKey = RedisKeyUtil.getDAUKey(df.format(start),df.format(end));
            connection.bitOp(RedisStringCommands.BitOperation.OR,
                    redisKey.getBytes(),keyList.toArray(new byte[0][0]));//转成byte[0][0]格式
            return connection.bitCount(redisKey.getBytes());
        });
    }
}

service中实现了UV和DAU的基本操作，以uv为例，编写了单个添加，与时间范围查询，单个添加用于在拦截器中统计访客，范围查询用来根据时间区间统计访问量数据，DAU基本同理。

接下来编写拦截器，每次用用户访问服务器获取服务时记录。

/**
 * @author ：LY
 * @date ：Created in 2021/3/11 22:40
 * @modified By：
 */
@Component
public class DataInterceptor  implements HandlerInterceptor {


    @Autowired
    private DataService dataService;
    @Autowired
    private HostHolder hostHolder;

    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
        //统计UV
        String ip = request.getRemoteHost();
        dataService.recordUV(ip);

        //统计DAU
        User user = hostHolder.getUser();
        if (user!=null){
            dataService.recordDAU(user.getId());
        }
        return true;
    }
}

每个访客通过Ip来记录，dau通过获取用户id存储，如果未登录就不算活跃用户。
拦截器编写完毕后需要在WebMvcConfig来进行配置放行路径，静态资源不做拦截。

     registry.addInterceptor(dataInterceptor)
                .excludePathPatterns("/**/*.css","/**/*.js","/**/*.png","/**/*.jpg","/**/*.jpeg");

最后一步编写controller提供数据获取接口

/**
 * @author ：LY
 * @date ：Created in 2021/3/11 22:43
 * @modified By：
 */
@Controller
public class DataController {

    @Autowired
    private DataService dataService;

    @RequestMapping(path = "/data",method = {RequestMethod.GET,RequestMethod.POST})
    public String getDataPage(){
        return "/site/admin/data";
    }

    @PostMapping("/data/uv")
    public String getUV(@DateTimeFormat(pattern = "yyyy-MM-dd") Date start,
                        @DateTimeFormat(pattern = "yyyy-MM-dd")Date end,
                        Model model){
        long uv = dataService.calculateUV(start,end);
        model.addAttribute("uvResult",uv);
        model.addAttribute("uvStartDate",start);
        model.addAttribute("uvEndDate",end);

        return "forward:/data";
    }

    @PostMapping("/data/dau")
    public String getDAU(@DateTimeFormat(pattern = "yyyy-MM-dd") Date start,
                        @DateTimeFormat(pattern = "yyyy-MM-dd")Date end,
                        Model model){
        long dau = dataService.calculateDAU(start,end);
        model.addAttribute("dauResult",dau);
        model.addAttribute("dauStartDate",start);
        model.addAttribute("dauEndDate",end);
        return "forward:/data";
    }
}

最后来看看实际效果