对于bigkey来说,我们不能直接使用del删除,避免删除时间过长,导致阻塞其他客户端命令,正确的姿势应该是采用hsan、sscan、zscan渐进式删除,同时我们还要防止bigkey过期时间自动删除的问题(如一个200万的zset设置1小时过期,会触发del操作,造成阻塞,且该操作不会出现在慢查询中)
基于Jedis客户端实现渐进式删除
1、 Hash删除: hscan + hdel
public static void delBigHash(String host , int port , String bigHashKey){
Jedis jedis = new Jedis(host, port);
ScanParams scanParams = new ScanParams().count(100);
String cursor = "0";
do {
ScanResult<Map.Entry<String, String>> scanResult = jedis.hscan(bigHashKey, cursor, scanParams);
List<Map.Entry<String, String>> entryList = scanResult.getResult();
for (Map.Entry<String, String> entry : entryList) {
jedis.hdel(bigHashKey, entry.getKey());
}
cursor = scanResult.getCursor();
}while (!"0".equals(cursor));
}
2、List删除: ltrim
public static void delBigList(String host , int port , String bigListKey){
Jedis jedis = new Jedis(host, port);
Long llen = jedis.llen(bigListKey); //队列中的总数量
int counter = 0; //记录已经删除的数量
int size = 100; //每次截掉(删除)的数量
while (counter < llen){
//每次从左侧删除100个数
jedis.ltrim(bigListKey, size, llen);
counter += size;
}
//最终删除key
jedis.del(bigListKey);
}
3、Set删除: sscan + srem
public static void delBigSet(String host , int port , String bigSetKey){
Jedis jedis = new Jedis(host, port);
ScanParams scanParams = new ScanParams().count(100);
String cursor = "0";
do {
ScanResult<String> scanResult = jedis.sscan(bigSetKey, cursor,scanParams);
List<String> memberList = scanResult.getResult();
for (String member : memberList) {
jedis.srem(bigSetKey, member);
}
cursor = scanResult.getCursor();
}while (!"0".equals(cursor));
//删除bigkey
jedis.del(bigSetKey);
}
4、SortedSet删除: zscan + zrem
public static void delBigZSet(String host , int port , String bigZSetKey){
Jedis jedis = new Jedis(host, port);
ScanParams scanParams = new ScanParams().count(100);
String cursor = "0";
do {
ScanResult<Tuple> zscanResult = jedis.zscan(bigZSetKey, cursor,scanParams);
List<Tuple> tupleList = zscanResult.getResult();
for (Tuple tuple : tupleList) {
jedis.zrem(bigZSetKey, tuple.getElement());
}
cursor = zscanResult.getCursor();
}while (!"0".equals(cursor));
//删除bigkey
jedis.del(bigZSetKey);
}
在实际开发中,我们可能整合了Spring Boot,使用StringRedisTemplate
来操作redis,接下来就来看看StringRedisTemplate如何实现渐进式删除。
基于StringRedisTemplate实现渐进式删除
1、hash渐进式删除
public void delBigHash() throws IOException {
String bigHashKey = "bigHashKey";
ScanOptions scanOptions = ScanOptions.scanOptions().count(100).build();
Cursor<Map.Entry<Object, Object>> cursor = redisTemplate.opsForHash().scan(bigHashKey, scanOptions);
while (cursor.hasNext()){
Map.Entry<Object, Object> next = cursor.next();
redisTemplate.opsForHash().delete(bigHashKey, next.getKey());
}
cursor.close();
}
等价于:
public void delBigHash() throws IOException {
String bigHashKey = "bigHashKey";
ScanOptions scanOptions = ScanOptions.scanOptions().count(100).build();
Cursor<Map.Entry<Object, Object>> cursor = redisTemplate.opsForHash().scan(bigHashKey, scanOptions);
//java8新特性
cursor.forEachRemaining(member -> {
redisTemplate.opsForHash().delete(bigHashKey, member.getKey());
});
cursor.close();
}
- cursor.close(); 游标一定要关闭,不然连接会一直增长**;可以使用
client lists
、info clients
、info stats
命令查看客户端连接状态,会发现scan操作一直存在。 - 代码虽然只是调用一次scan方法,但是spring-data-redis已经对scan做了封装,这个scan结合cursor.hasNext会多次redis scan,最终拿到所有match的结果。
2、list渐进式删除
public void delList(){
Long size = redisTemplate.opsForList().size("bigListKey");
Long counter = 0L;
int trimNum = 100; //每次删除的个数
while (counter < size){
redisTemplate.opsForList().trim("bigListKey", counter, trimNum);
counter += trimNum;
}
}
3、set渐进式删除
public void delSet() throws IOException {
String bigSetKey = "bigSetKey";
ScanOptions scanOptions = ScanOptions.scanOptions().count(100).build();
Cursor<String> cursor = redisTemplate.opsForSet().scan(bigSetKey, scanOptions);
while (cursor.hasNext()){
redisTemplate.opsForSet().remove(bigSetKey, cursor.next());
}
cursor.close();
}
等价于:
public void delSet() throws IOException {
String bigSetKey = "bigSetKey";
ScanOptions scanOptions = ScanOptions.scanOptions().count(100).build();
Cursor<String> cursor = redisTemplate.opsForSet().scan(bigSetKey, scanOptions);
//java8新特性
cursor.forEachRemaining(member -> {
redisTemplate.opsForSet().remove(bigSetKey, member);
});
cursor.close();
}
4、ZSet渐进式删除
public void delZSet() throws IOException {
String bigZSetKey = "bigZSetKey";
ScanOptions scanOptions = ScanOptions.scanOptions().count(100).build();
Cursor<ZSetOperations.TypedTuple<String>> cursor = redisTemplate.opsForZSet().scan(bigZSetKey, scanOptions);
while (cursor.hasNext()){
redisTemplate.opsForZSet().remove(bigZSetKey, cursor.next().getValue());
}
cursor.close();
}
等价于:
public void delZSet() throws IOException {
String bigZSetKey = "bigZSetKey";
ScanOptions scanOptions = ScanOptions.scanOptions().count(100).build();
Cursor<ZSetOperations.TypedTuple<String>> cursor = redisTemplate.opsForZSet().scan(bigZSetKey, scanOptions);
cursor.forEachRemaining(member -> {
redisTemplate.opsForZSet().remove(bigZSetKey, member.getValue());
});
cursor.close();
}
参考资料:
https://www.jianshu.com/p/4c842c41ba41