缓存使用
为了系统性能的提升,我们一般都会将部分数据放入缓存中,加速访问。而db承担数据落盘工作。
哪些数据适合放入缓存?
- 即时性、数据一致性要求不高的
- 访问量大且更新频率不高的数据(读多,写少)
举例:电商类应用,商品分类,商品列表等适合缓存并加一个失效时间(根据数据更新频率来定),后台如果发布一个商品,买家需要5分钟才能看到新的商品一般还是可以接受的。
本地缓存可以使用map来做缓存,分布式下则不行,则需要加入缓存中间件来统一缓存
这里我们使用redis,Java使用redis有很多方法。
SpringBoot使用redis异常
springBoot可以直接使用redis,但是可能会有异常
@Override
public Map<String, List<Catalog2Vo>> getCatalogJson() {
String catalogJSON = stringRedisTemplate.opsForValue().get("catalogJSON");
if(StringUtils.isBlank("catalogJSON") || catalogJSON == null){
Map<String, List<Catalog2Vo>> catalogJsonFromDb = getCatalogJsonFromDb();
String s = JSON.toJSONString(catalogJsonFromDb);
stringRedisTemplate.opsForValue().set("catalogJSON",s,1,TimeUnit.SECONDS);
return catalogJsonFromDb;
}
Map<String, List<Catalog2Vo>> map = JSON.parseObject(catalogJSON, new TypeReference<Map<String, List<Catalog2Vo>>>() {
});
return map;
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
堆外内存溢出:
原因:
springboot在2.0后默认使用lettuce操作redis客户端,它使用netty进行网络通信,lettuce的bug导致堆外内存溢出,底层实现是只要有操作,就会统计内存使用量,操作完会decrement减内存,可能是lettuce客户端在减内存的过程出错
解决:
对这个问题调节-Xmx大小不起作用,而且如果netty(netty如没有指定,默认是-Xmx300m)没指定堆外内存,默认使用-Xmx,所以一旦出现问题,就算调大-Xmx也总会到堆外内存溢出的地步。所以我们不能用io.netty.maxDirectMemory只修改堆外内存大小。
还有2种方案:
1、升级lettuce客户端
2、排除lettuce依赖,使用jedis
Jedis
jedis是什么
Jedis是Redis的Java实现的客户端,其API提供了比较全面的Redis命令的支持;
使用jedis
Jedis实例不是线程安全的,所以不可以多个线程共用一个Jedis实例,但是创建太多的实现也不好因为这意味着会建立很多sokcet连接。
JedisPool是一个线程安全的网络连接池。可以用JedisPool创建一些可靠Jedis实例,可以从池中获取Jedis实例,使用完后再把Jedis实例还回JedisPool。这种方式可以避免创建大量socket连接并且会实现高效的性能.
依赖
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.9.3</version>
</dependency>
- 1
- 2
- 3
- 4
- 5
RedisUtil
public class RedisUtil {
<span class="token keyword">private</span> JedisPool jedisPool<span class="token punctuation">;</span> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">initPool</span><span class="token punctuation">(</span>String host<span class="token punctuation">,</span><span class="token keyword">int</span> port <span class="token punctuation">,</span><span class="token keyword">int</span> database<span class="token punctuation">)</span><span class="token punctuation">{</span> JedisPoolConfig poolConfig <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">JedisPoolConfig</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> poolConfig<span class="token punctuation">.</span><span class="token function">setMaxTotal</span><span class="token punctuation">(</span><span class="token number">200</span><span class="token punctuation">)</span><span class="token punctuation">;</span> poolConfig<span class="token punctuation">.</span><span class="token function">setMaxIdle</span><span class="token punctuation">(</span><span class="token number">30</span><span class="token punctuation">)</span><span class="token punctuation">;</span> poolConfig<span class="token punctuation">.</span><span class="token function">setBlockWhenExhausted</span><span class="token punctuation">(</span><span class="token boolean">true</span><span class="token punctuation">)</span><span class="token punctuation">;</span> poolConfig<span class="token punctuation">.</span><span class="token function">setMaxWaitMillis</span><span class="token punctuation">(</span><span class="token number">10</span><span class="token operator">*</span><span class="token number">1000</span><span class="token punctuation">)</span><span class="token punctuation">;</span> poolConfig<span class="token punctuation">.</span><span class="token function">setTestOnBorrow</span><span class="token punctuation">(</span><span class="token boolean">true</span><span class="token punctuation">)</span><span class="token punctuation">;</span> jedisPool<span class="token operator">=</span><span class="token keyword">new</span> <span class="token class-name">JedisPool</span><span class="token punctuation">(</span>poolConfig<span class="token punctuation">,</span>host<span class="token punctuation">,</span>port<span class="token punctuation">,</span><span class="token number">20</span><span class="token operator">*</span><span class="token number">1000</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">public</span> Jedis <span class="token function">getJedis</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">{</span> Jedis jedis <span class="token operator">=</span> jedisPool<span class="token punctuation">.</span><span class="token function">getResource</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> jedis<span class="token punctuation">;</span> <span class="token punctuation">}</span>
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
RedisConfig
@Configuration public class RedisConfig {
<span class="token comment">//读取配置文件中的redis的ip地址</span> <span class="token annotation punctuation">@Value</span><span class="token punctuation">(</span><span class="token string">"${spring.redis.host:disabled}"</span><span class="token punctuation">)</span> <span class="token keyword">private</span> String host<span class="token punctuation">;</span> <span class="token annotation punctuation">@Value</span><span class="token punctuation">(</span><span class="token string">"${spring.redis.port:0}"</span><span class="token punctuation">)</span> <span class="token keyword">private</span> <span class="token keyword">int</span> port<span class="token punctuation">;</span> <span class="token annotation punctuation">@Value</span><span class="token punctuation">(</span><span class="token string">"${spring.redis.database:0}"</span><span class="token punctuation">)</span> <span class="token keyword">private</span> <span class="token keyword">int</span> database<span class="token punctuation">;</span> <span class="token annotation punctuation">@Bean</span> <span class="token keyword">public</span> RedisUtil <span class="token function">getRedisUtil</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">{</span> <span class="token keyword">if</span><span class="token punctuation">(</span>host<span class="token punctuation">.</span><span class="token function">equals</span><span class="token punctuation">(</span><span class="token string">"disabled"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">{</span> <span class="token keyword">return</span> null<span class="token punctuation">;</span> <span class="token punctuation">}</span> RedisUtil redisUtil<span class="token operator">=</span><span class="token keyword">new</span> <span class="token class-name">RedisUtil</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> redisUtil<span class="token punctuation">.</span><span class="token function">initPool</span><span class="token punctuation">(</span>host<span class="token punctuation">,</span>port<span class="token punctuation">,</span>database<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> redisUtil<span class="token punctuation">;</span> <span class="token punctuation">}</span>
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
application.properties
spring.redis.host=redis服务地址
spring.redis.port=6379
spring.redis.database=0
- 1
- 2
- 3
使用测试
@Autowired RedisUtil redisUtil;
<span class="token annotation punctuation">@Test</span> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">testRedis</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> Jedis jedis <span class="token operator">=</span> redisUtil<span class="token punctuation">.</span><span class="token function">getJedis</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//jedis.set("test","test");</span> String s <span class="token operator">=</span> jedis<span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token string">"test"</span><span class="token punctuation">)</span><span class="token punctuation">;</span> System<span class="token punctuation">.</span>out<span class="token punctuation">.</span><span class="token function">println</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
业务使用
/** * 查出所有分类 返回首页json */ @Override public Map<String, List<Catalog2Vo>> getCatalogJson() { //加入缓存逻辑 Jedis jedis = redisUtil.getJedis(); String catalogJson = jedis.get("catalogJson");
Map<span class="token operator"><</span>String<span class="token punctuation">,</span> List<span class="token generics function"><span class="token punctuation"><</span>Catalog2Vo<span class="token punctuation">></span></span><span class="token operator">></span> json <span class="token operator">=</span> null<span class="token punctuation">;</span> <span class="token comment">//缓存存在 转换返回</span> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span>StringUtils<span class="token punctuation">.</span><span class="token function">isEmpty</span><span class="token punctuation">(</span>catalogJson<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> json <span class="token operator">=</span> JSONObject<span class="token punctuation">.</span><span class="token function">parseObject</span><span class="token punctuation">(</span>catalogJson<span class="token punctuation">,</span> <span class="token keyword">new</span> <span class="token class-name">TypeReference</span><span class="token operator"><</span>Map<span class="token operator"><</span>String<span class="token punctuation">,</span> List<span class="token generics function"><span class="token punctuation"><</span>Catalog2Vo<span class="token punctuation">></span></span><span class="token operator">>></span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> json<span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token comment">//缓存没有从数据查询</span> json <span class="token operator">=</span> <span class="token function">getCatalogJsonFromDB</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//转成str 加入缓存</span> String jsonString <span class="token operator">=</span> JSON<span class="token punctuation">.</span><span class="token function">toJSONString</span><span class="token punctuation">(</span>json<span class="token punctuation">)</span><span class="token punctuation">;</span> jedis<span class="token punctuation">.</span><span class="token function">set</span><span class="token punctuation">(</span><span class="token string">"catalogJson"</span><span class="token punctuation">,</span> jsonString<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> json<span class="token punctuation">;</span> <span class="token punctuation">}</span>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
/** * 从数据库获取数据并封装返回 * * @return */ public Map<String, List<Catalog2Vo>> getCatalogJsonFromDB() { //查询出所有分类 List<CategoryEntity> selectList = baseMapper.selectList(null); //先查出所有一级分类 List<CategoryEntity> level1Categorys = getCategorys(selectList, 0L);
<span class="token comment">//封装数据 map k,v 结构</span> Map<span class="token operator"><</span>String<span class="token punctuation">,</span> List<span class="token generics function"><span class="token punctuation"><</span>Catalog2Vo<span class="token punctuation">></span></span><span class="token operator">></span> map <span class="token operator">=</span> level1Categorys<span class="token punctuation">.</span><span class="token function">stream</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">collect</span><span class="token punctuation">(</span>Collectors<span class="token punctuation">.</span><span class="token function">toMap</span><span class="token punctuation">(</span>k <span class="token operator">-</span><span class="token operator">></span> k<span class="token punctuation">.</span><span class="token function">getCatId</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toString</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> v <span class="token operator">-</span><span class="token operator">></span> <span class="token punctuation">{</span> <span class="token comment">//每一个的一级分类,查到这个一级分类的二级分类</span> List<span class="token generics function"><span class="token punctuation"><</span>CategoryEntity<span class="token punctuation">></span></span> category2Entities <span class="token operator">=</span> <span class="token function">getCategorys</span><span class="token punctuation">(</span>selectList<span class="token punctuation">,</span> v<span class="token punctuation">.</span><span class="token function">getCatId</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> List<span class="token generics function"><span class="token punctuation"><</span>Catalog2Vo<span class="token punctuation">></span></span> catelog2Vos <span class="token operator">=</span> null<span class="token punctuation">;</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>category2Entities <span class="token operator">!=</span> null<span class="token punctuation">)</span> <span class="token punctuation">{</span> catelog2Vos <span class="token operator">=</span> category2Entities<span class="token punctuation">.</span><span class="token function">stream</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span>level2 <span class="token operator">-</span><span class="token operator">></span> <span class="token punctuation">{</span> <span class="token comment">//封装catalog2Vo</span> Catalog2Vo catalog2Vo <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Catalog2Vo</span><span class="token punctuation">(</span>v<span class="token punctuation">.</span><span class="token function">getCatId</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toString</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> null<span class="token punctuation">,</span> level2<span class="token punctuation">.</span><span class="token function">getCatId</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toString</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> level2<span class="token punctuation">.</span><span class="token function">getName</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//每一个二级分类,查到三级分类</span> List<span class="token generics function"><span class="token punctuation"><</span>CategoryEntity<span class="token punctuation">></span></span> category3Entities <span class="token operator">=</span> <span class="token function">getCategorys</span><span class="token punctuation">(</span>selectList<span class="token punctuation">,</span> level2<span class="token punctuation">.</span><span class="token function">getCatId</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>category3Entities <span class="token operator">!=</span> null<span class="token punctuation">)</span> <span class="token punctuation">{</span> List<span class="token generics function"><span class="token punctuation"><</span>Object<span class="token punctuation">></span></span> catalog3List <span class="token operator">=</span> category3Entities<span class="token punctuation">.</span><span class="token function">stream</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span>level3 <span class="token operator">-</span><span class="token operator">></span> <span class="token punctuation">{</span> <span class="token comment">//封装catalog3Vo</span> Catalog2Vo<span class="token punctuation">.</span>Catalog3Vo catalog3Vo <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Catalog2Vo<span class="token punctuation">.</span>Catalog3Vo</span><span class="token punctuation">(</span>level2<span class="token punctuation">.</span><span class="token function">getCatId</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toString</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> level3<span class="token punctuation">.</span><span class="token function">getCatId</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toString</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> level3<span class="token punctuation">.</span><span class="token function">getName</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> catalog3Vo<span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">collect</span><span class="token punctuation">(</span>Collectors<span class="token punctuation">.</span><span class="token function">toList</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//封装catalog3Vo到catalog2Vo</span> catalog2Vo<span class="token punctuation">.</span><span class="token function">setCatalog3List</span><span class="token punctuation">(</span>catalog3List<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">return</span> catalog2Vo<span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">collect</span><span class="token punctuation">(</span>Collectors<span class="token punctuation">.</span><span class="token function">toList</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token comment">//返回v=catalog2Vo</span> <span class="token keyword">return</span> catelog2Vos<span class="token punctuation">;</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> map<span class="token punctuation">;</span> <span class="token punctuation">}</span>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
测试ok
但是这样在高并发下缓存是会存在很多问题的,详情参见分布式-Redis-缓存问题
lua脚本:
String script = "if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end";
- 1
就是加锁设置过期时间保证加锁操作是原子性的,解锁也是同样保持原子性。(原子性简单理解就是1和2两个操作都是要同时ok这整个任务才能算ok)
加锁获取数据
public Map<String, List<Catalog2Vo>> getCatalogJsonFromDBWithRedisLock() { Jedis jedis = redisUtil.getJedis(); //加锁 String token = UUID.randomUUID().toString(); String lock = jedis.set("lock", token, "NX", "EX", 20); System.out.println(lock);
Map<span class="token operator"><</span>String<span class="token punctuation">,</span> List<span class="token generics function"><span class="token punctuation"><</span>Catalog2Vo<span class="token punctuation">></span></span><span class="token operator">></span> map <span class="token operator">=</span> null<span class="token punctuation">;</span> <span class="token comment">//加锁成功</span> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token string">"ok"</span><span class="token punctuation">.</span><span class="token function">equals</span><span class="token punctuation">(</span>lock<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> map <span class="token operator">=</span> <span class="token function">getCatalogJsonFromDB</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//删除锁 lua脚本</span> String script <span class="token operator">=</span> <span class="token string">"if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end"</span><span class="token punctuation">;</span> jedis<span class="token punctuation">.</span><span class="token function">eval</span><span class="token punctuation">(</span>script<span class="token punctuation">,</span> Collections<span class="token punctuation">.</span><span class="token function">singletonList</span><span class="token punctuation">(</span><span class="token string">"lock"</span><span class="token punctuation">)</span><span class="token punctuation">,</span> Collections<span class="token punctuation">.</span><span class="token function">singletonList</span><span class="token punctuation">(</span>token<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> map<span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span> <span class="token keyword">try</span> <span class="token punctuation">{</span> Thread<span class="token punctuation">.</span><span class="token function">sleep</span><span class="token punctuation">(</span><span class="token number">300</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span><span class="token class-name">InterruptedException</span> e<span class="token punctuation">)</span> <span class="token punctuation">{</span> e<span class="token punctuation">.</span><span class="token function">printStackTrace</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token comment">//自旋</span> <span class="token keyword">return</span> <span class="token function">getCatalogJsonFromDBWithRedisLock</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
</div><div data-report-view="{"mod":"1585297308_001","dest":"https://blog.csdn.net/D_A_I_H_A_O/article/details/106771416","extend1":"pc","ab":"new"}"><div></div></div>
<link href="https://csdnimg.cn/release/phoenix/mdeditor/markdown_views-60ecaf1f42.css" rel="stylesheet">
</div>