p101总结基础篇
p102 es简介
概念
es中存储的数据是json格式
es底层是lucence,倒排索引表
p103 安装es
安装es和可视化工具kibana
# 将docker里的目录挂载到linux的/mydata目录中
# 修改/mydata就可以改掉docker里的
mkdir -p /mydata/elasticsearch/config
mkdir -p /mydata/elasticsearch/data
# es可以被远程任何机器访问
echo "http.host: 0.0.0.0" >/mydata/elasticsearch/config/elasticsearch.yml
# 递归更改权限，es需要访问
chmod -R 777 /mydata/elasticsearch/
创建容器,启动es
# 9200是用户交互端口 9300是集群心跳端口
# -e指定是单阶段运行
# -e指定占用的内存大小，生产时可以设置32G
docker run --name elasticsearch -p 9200:9200 -p 9300:9300 \
-e "discovery.type=single-node" \
-e ES_JAVA_OPTS="-Xms64m -Xmx512m" \
-v /mydata/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
-v /mydata/elasticsearch/data:/usr/share/elasticsearch/data \
-v /mydata/elasticsearch/plugins:/usr/share/elasticsearch/plugins \
-d elasticsearch:7.4.2
# 设置开机启动elasticsearch
docker update elasticsearch --restart=always
访问
p104安装kibana
# kibana指定了了ES交互端口9200 # 5600位kibana主页端口
docker run --name kibana -e ELASTICSEARCH_HOSTS=http://192.168.56.10:9200 -p 5601:5601 -d kibana:7.4.2
# 设置开机启动kibana
docker update kibana --restart=always
访问kibana
p105 _cat操作
GET /_cat/nodes：查看所有节点
GET /_cat/health：查看es健康状况
GET /_cat/master：查看主节点
GET /_cat/indices：查看所有索引 ，等价于mysql数据库的show databases;
p106新增数据
# # 在customer索引下的external类型下保存1号数据
PUT customer/external/1
# POSTMAN输入
http://192.168.56.10:9200/customer/external/1
{
"name":"John Doe"
}
PUT和POST区别
**POST新增。如果不指定id，会自动生成id。**指定id就会修改这个数据，并新增版本号；
可以不指定id，不指定id时永远为创建
指定不存在的id为创建
指定存在的id为更新，而版本号会根据内容变没变而觉得版本号递增与否
PUT可以新增也可以修改。PUT必须指定id；由于PUT需要指定id，我们一般用来做修改操作，不指定id会报错。
必须指定id
版本号总会增加
p107查询数据与乐观锁
GET /customer/external/1
乐观锁用法：通过“if_seq_no=1&if_primary_term=1”，当序列号匹配的时候，才进行修改，否则不修改。
修改成功
修改失败
p108 put与post修改数据
post有2种修改数据
带_update后缀的,json中要带"doc",这种修改会检查内容是否真的被修改过,如果内容没变,_version和_seq_no都不会变
另外一种就是和put一样的修改方式,_version和_seq_no会变
带_update
不带"_update"的可以 不带"doc"
p109 删除数据&批量操作 导入数据
DELETE customer/external/1
DELETE customer
批量操作数据
格式
{action:{metadata}}\n
{request body }\n
{action:{metadata}}\n
{request body }\n
POST /_bulk
{"delete":{"_index":"website","_type":"blog","_id":"123"}}
{"create":{"_index":"website","_type":"blog","_id":"123"}}
{"title":"my first blog post"}
{"index":{"_index":"website","_type":"blog"}}
{"title":"my second blog post"}
{"update":{"_index":"website","_type":"blog","_id":"123"}}
{"doc":{"title":"my updated blog post"}}
测试批量导入了bank索引的1000条数据
p110 进阶检索
ES支持两种基本方式检索；
请求参数方式检索
GET bank/_search?q=*&sort=account_number:asc
说明：
q=* # 查询所有
sort # 排序字段
asc #升序
检索bank下所有信息，包括type和docs
GET bank/_search
GET /bank/_search
{
"query": { "match_all": {} },
"sort": [
{ "account_number": "asc" },
{ "balance":"desc"}
]
}
p111基本使用match_all
_source是只查询部分字段
get /bank/_search
{
"query": {
"match_all": {}
},
"sort": [
{
"balance": {
"order": "desc"
}
}
],
"from":"0",
"size":"2",
"_source":["firstname","balance"]
}
p112 match全文匹配
get /bank/_search
{
"query": {
"match": {
"address": "mill lane"
}
}
}
p113 match_phrase不分词匹配
get /bank/_search
{
"query":{
"match_phrase": {
"address": "mill lane"
}
}
}
p114 multi_match多字段匹配
只要有一个字段满足即可
get /bank/_search
{
"query":{
"multi_match": {
"query": "forbes",
"fields": ["firstname","lastname"]
}
}
}
p115 bool组合查询
should代表应该,可以不匹配,匹配上会加分
get /bank/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"gender": "M"
}
},
{
"match": {
"address": "mill"
}
}
],
"must_not": [
{
"match": {
"age": "18"
}
}
],
"should": [
{
"match": {
"lastname": "Wallace"
}
}
]
}
}
}
p116 filter过滤器查询
filter不会加分
get /bank/_search
{
"query": {
"bool": {
"must": [
{
"range": {
"age": {
"gte": 10,
"lte": 30
}
}
}
],
"filter": {
"range": {
"age": {
"gte": 10,
"lte": 30
}
}
}
}
}
}
p117 term查询
term精确查询
get /bank/_search
{
"query": {
"term": {
"age": {
"value": "23"
}
}
}
}
match也有精确匹配,不分词
p118 aggregations聚合分析
p119mapping创建
mapping类型,text类型加上keyword类型不分词
创建映射类型
put /my_index
{
"mappings":{
"properties":{
"age":{
"type":"integer"
},
"email":{
"type":"keyword"
},
"name":{
"type":"text"
}
}
}
}
p120 添加字段映射
index代表是否会被索引,默认为true,可以用来检索
p121修改映射&数据迁移
es不支持修改映射类型,只能重新建立索引 和映射,再数据迁移
es后面取消了type 默认为_doc
POST _reindex
{
"source":{
"index":"bank",
"type":"account"
},
"dest":{
"index":"newbank"
}
}
6.0以后写法
6.0以后写法
POST reindex
{
"source":{
"index":"twitter"
},
"dest":{
"index":"new_twitters"
}
}
p122安装ik分词器
docker查看日志:docker logs 容器id
测试分词
POST _analyze
{
"analyzer": "standard",
"text": "我是中国人"
}
进入容器 下载分词器
[vagrant@localhost ~]$ sudo docker exec -it elasticsearch /bin/bash
[root@66718a266132 elasticsearch]# pwd
/usr/share/elasticsearch
[root@66718a266132 elasticsearch]# pwd
/usr/share/elasticsearch
[root@66718a266132 elasticsearch]# yum install wget
#下载ik7.4.2
[root@66718a266132 elasticsearch]# wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.4.2/elasticsearch-analysis-ik-7.4.2.zip
解压 重启容器
[root@66718a266132 elasticsearch]# unzip elasticsearch-analysis-ik-7.4.2.zip -d ik
#移动到plugins目录下
[root@66718a266132 elasticsearch]# mv ik plugins/
chmod -R 777 plugins/ik
docker restart elasticsearch
记得删除压缩包
测试
p123 虚拟机网络设置
可以查看基础篇网络设置
p124自定义扩展词库&创建nginx
把词库txt文件放在nginx下,配置ik配置文件引用即可
创建nginx容器,没有镜像会自动先拉取
docker run -p 80:80 --name nginx -d nginx:1.10
在容器外面创建一个文件夹
把容器中的nginx配置复制到外面的nginx目录
删除之前的nginx容器(之前nginx只为了复制配置文件),重新创建
docker run -p 80:80 --name nginx \
-v /mydata/nginx/html:/usr/share/nginx/html \
-v /mydata/nginx/logs:/var/log/nginx \
-v /mydata/nginx/conf/:/etc/nginx \
-d nginx:1.10
在nginx的html目录下mkdir es ,再vim fenxi.txt
p125 springboot整合hign-level-client
依赖
配置
p126测试es保存
@Autowired
RestHighLevelClient client;
@Test
public void testEsSave() throws IOException {
IndexRequest indexRequest = new IndexRequest("users");
indexRequest.id("2");
User user = new User();
user.setName("lisi");
user.setAge(18);
user.setGender("男");
String str = JSON.toJSONString(user);
indexRequest.source(str, XContentType.JSON);
IndexResponse index = client.index(indexRequest, GulimallEsConfig.COMMON_OPTIONS);
System.out.println(index);
}
p127测试es复杂检索
这里简单测试年龄分布聚合,和平均薪资聚合
@Test
public void testEsSearch() throws IOException {
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("bank");
//检索请求要带者检索条件
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
//构造一个querybuilder 查询条件
MatchQueryBuilder queryBuilder = QueryBuilders.matchQuery("address", "mill");
searchSourceBuilder.query(queryBuilder);//要一个querybuilder
//年龄分组
TermsAggregationBuilder aggregationBuilder = AggregationBuilders.terms("ageAgg").field("age").size(100);
searchSourceBuilder.aggregation(aggregationBuilder);
//求工资平均值
AvgAggregationBuilder balanceAgg = AggregationBuilders.avg("balanceAgg").field("balance");
searchSourceBuilder.aggregation(balanceAgg);
searchRequest.source(searchSourceBuilder);
System.out.println(searchRequest);
SearchResponse searchResponse = client.search(searchRequest, GulimallEsConfig.COMMON_OPTIONS);
SearchHits hits = searchResponse.getHits();
SearchHit[] hits1 = hits.getHits();
for (SearchHit searchHit : hits1) {
String sourceAsString = searchHit.getSourceAsString();
Account account = JSON.parseObject(sourceAsString, Account.class);
System.out.println(account);
}
System.out.println("==========================");
Aggregations aggregations = searchResponse.getAggregations();
//获取年龄聚合
Terms ageAgg = aggregations.get("ageAgg");
List<? extends Terms.Bucket> buckets = ageAgg.getBuckets();
for (Terms.Bucket bucket : buckets) {
String age = bucket.getKeyAsString();
long docCount = bucket.getDocCount();
System.out.println(age+"->"+docCount);
}
//获取平均薪资聚合
Avg balanceAgg1 = aggregations.get("balanceAgg");
System.out.println("平均薪资:"+balanceAgg1.getValue());
}
p128 sku在es中存储模型
PUT product
{
"mappings":{
"properties": {
"skuId":{ "type": "long" },
"spuId":{ "type": "keyword" }, # 不可分词
"skuTitle": {
"type": "text",
"analyzer": "ik_smart" # 中文分词器
},
"skuPrice": { "type": "keyword" }, # 保证精度问题
"skuImg" : { "type": "keyword" }, # 视频中有false
"saleCount":{ "type":"long" },
"hasStock": { "type": "boolean" },
"hotScore": { "type": "long" },
"brandId": { "type": "long" },
"catalogId": { "type": "long" },
"brandName": {"type": "keyword"}, # 视频中有false
"brandImg":{
"type": "keyword",
"index": false, # 不可被检索，不生成index，只用做页面使用
"doc_values": false # 不可被聚合，默认为true
},
"catalogName": {"type": "keyword" }, # 视频里有false
"attrs": {
"type": "nested",
"properties": {
"attrId": {"type": "long" },
"attrName": {
"type": "keyword",
"index": false,
"doc_values": false
},
"attrValue": {"type": "keyword" }
}
}
}
}
}
p129nested数据
数组类型的对象,默认会被扁平化处理,所以我们设置;类型为nested
p130商品上架 构造基本数据
在这里插入图片描述
p131构造skumodel检索规格属性
获取检索属性
<select id="getSearchAttrId" resultType="java.lang.Long">
select attr_id from pms_attr where attr_type=1 and attr_id in
<foreach collection="attrIds" item="attr" open="(" close=")" separator=",">
#{attr}
</foreach>
</select>
@Override
public void up(Long spuId) {
//上架商品,就是把这个spu下的多个sku存储到es中,es中sku信息和mysql中不一样
List<SkuInfoEntity> skuInfoEntities= skuInfoService.getSkusBySpuId(spuId);
SkuInfoEntity entity = skuInfoEntities.get(0);
//查询该spu对应的分类和品牌
CategoryEntity categoryEntity = categoryService.getById(entity.getCatalogId());
BrandEntity brandEntity = brandService.getById(entity.getBrandId());
//todo 设置用来被检索的规格属性 是根据spuid来查询的
List<ProductAttrValueEntity> productAttrValueEntities = attrValueService.baseAttrlistforspu(spuId);
//需要把不是检索属性的过滤掉,留下检索属性的
List<Long> attrIds = productAttrValueEntities.stream().map(attrs -> {
return attrs.getAttrId();
}).collect(Collectors.toList());
//得到搜索属性
List<Long> searchAttrIds=attrService.getSearchAttrId(attrIds);
//得到最终要skumodel里的规格属性
List<SkuEsModel.Attr> attrs = productAttrValueEntities.stream().filter(attr -> {
return searchAttrIds.contains(attr.getId());
}).map(attr -> {
SkuEsModel.Attr attr1 = new SkuEsModel.Attr();
attr1.setAttrId(attr.getAttrId());
attr1.setAttrName(attr.getAttrName());
attr1.setAttrValue(attr.getAttrValue());
return attr1;
}).collect(Collectors.toList());
//需要构造出esmodel
List<SkuEsModel> skuEsModels = skuInfoEntities.stream().map(skuInfoEntity -> {
SkuEsModel skuEsModel = new SkuEsModel();
BeanUtils.copyProperties(skuInfoEntity,skuEsModel);
skuEsModel.setSkuPrice(skuInfoEntity.getPrice());
skuEsModel.setSkuImg(skuInfoEntity.getSkuDefaultImg());
//todo 发送远程调用 ,库存系统查询是否有库存
skuEsModel.setHasStock();
skuEsModel.setHotScore(0L);
skuEsModel.setBrandName(brandEntity.getName());
skuEsModel.setBrandImg(brandEntity.);
skuEsModel.setCatalogName(categoryEntity.getName());
skuEsModel.setAttrs(attrs);
return skuEsModel;
}).collect(Collectors.toList());
}
p132远程查询库存
@Override
public List<SkuHasStockVo> getSkuHasStock(List<Long> skuIds) {
//这里是自己写的 老师写的会循环查数据库 柑橘不太好 于是自己写了一种
//直接根据skuid去查询商品的库存 注意这里的库存是 一个sku可能在多个仓库的stock-stocklocked,
//所以一个skuid可能会有多条库存信息
List<WareSkuEntity> wareSkuEntities = baseMapper.selectList(new QueryWrapper<WareSkuEntity>().in("sku_id", skuIds));
HashMap<Long, Integer> map = new HashMap<>();
//循环多个库存信息,把相同skuid的加在一起,最后大于0的就是有库存
wareSkuEntities.stream().map(wareSkuEntity -> {
Long skuId = wareSkuEntity.getSkuId();
if (map.containsKey(skuId)){
Integer stock = map.get(skuId);
map.put(skuId,stock+wareSkuEntity.getStock()-wareSkuEntity.getStockLocked());
}else {
map.put(skuId,wareSkuEntity.getStock()-wareSkuEntity.getStockLocked());
}
return map;
}).collect(Collectors.toList());//这里后面collect好像是为了满足语法写的,不写结果不对
//上一步得到了商品库存 map存的是skuid,库存数量
//但是没有结果的skuid我们应该给他存个0,是为了给调用方返回结果,否则返回空
skuIds.stream().map(skuId->{
if (!map.containsKey(skuId)){
map.put(skuId,0);
}
return map;
}).collect(Collectors.toList());
Set<Long> keySet = map.keySet();
ArrayList<SkuHasStockVo> skuHasStockVos = new ArrayList<>();
for (Long key : keySet) {
SkuHasStockVo skuHasStockVo = new SkuHasStockVo();
skuHasStockVo.setSkuId(key);
skuHasStockVo.setHasStock(map.get(key)>0);
skuHasStockVos.add(skuHasStockVo);
}
return skuHasStockVos;
}
发现还有一种查询也可以不用循环查数据库,而且直接算出stock-stocklocked
这样可以直接在查询的时候就就算出skuid在多个仓库的种库存,比上面简单
SELECT sku_id,sum(stock-stock_locked) from wms_ware_sku where sku_id =5 union ALL
SELECT sku_id,sum(stock-stock_locked) from wms_ware_sku where sku_id =7
<select id="selectList1" resultType="com.atguigu.gulimall.ware.entity.WareSkuEntity">
<foreach collection="skuIds" separator="union all " item="skuId">
SELECT sku_id,sum(stock-stock_locked) from wms_ware_sku where sku_id =#{skuId}
</foreach>
</select>
把库存结果返回封装给skumodel
具体代码
@Override
public void up(Long spuId) {
//上架商品,就是把这个spu下的多个sku存储到es中,es中sku信息和mysql中不一样
List<SkuInfoEntity> skuInfoEntities= skuInfoService.getSkusBySpuId(spuId);
SkuInfoEntity entity = skuInfoEntities.get(0);
//查询该spu对应的分类和品牌
CategoryEntity categoryEntity = categoryService.getById(entity.getCatalogId());
BrandEntity brandEntity = brandService.getById(entity.getBrandId());
//todo 发送远程调用 ,库存系统查询是否有库存
List<Long> skuIds = skuInfoEntities.stream().map(s -> {
return s.getSkuId();
}).collect(Collectors.toList());
List<SkuHasStockVo> result = wareFeignService.getSkuHasStock(skuIds);
// Map<Long, Object> map = data.stream().collect(Collectors.toMap(SkuHasStockVo::getSkuId, s -> {
// return s.getHasStock();
// }));
//todo 这里巧妙的封装了(skuid,hasStock)的结果,供下面直接使用
Map<Long, Boolean> map = result.stream().collect(Collectors.toMap(SkuHasStockVo::getSkuId, SkuHasStockVo::getHasStock));
//todo 设置用来被检索的规格属性 是根据spuid来查询的
List<ProductAttrValueEntity> productAttrValueEntities = attrValueService.baseAttrlistforspu(spuId);
//需要把不是检索属性的过滤掉,留下检索属性的
List<Long> attrIds = productAttrValueEntities.stream().map(attrs -> {
return attrs.getAttrId();
}).collect(Collectors.toList());
//得到搜索属性
List<Long> searchAttrIds=attrService.getSearchAttrId(attrIds);
//得到最终要skumodel里的规格属性
List<SkuEsModel.Attr> attrs = productAttrValueEntities.stream().filter(attr -> {
return searchAttrIds.contains(attr.getId());
}).map(attr -> {
SkuEsModel.Attr attr1 = new SkuEsModel.Attr();
attr1.setAttrId(attr.getAttrId());
attr1.setAttrName(attr.getAttrName());
attr1.setAttrValue(attr.getAttrValue());
return attr1;
}).collect(Collectors.toList());
//需要构造出esmodel
List<SkuEsModel> skuEsModels = skuInfoEntities.stream().map(skuInfoEntity -> {
SkuEsModel skuEsModel = new SkuEsModel();
BeanUtils.copyProperties(skuInfoEntity,skuEsModel);
skuEsModel.setSkuPrice(skuInfoEntity.getPrice());
skuEsModel.setSkuImg(skuInfoEntity.getSkuDefaultImg());
//远程查询的库存
skuEsModel.setHasStock(map.get(skuInfoEntity.getSkuId()));
skuEsModel.setHotScore(0L);
skuEsModel.setBrandName(brandEntity.getName());
skuEsModel.setBrandImg(brandEntity.getLogo());
skuEsModel.setCatalogName(categoryEntity.getName());
skuEsModel.setAttrs(attrs);
return skuEsModel;
}).collect(Collectors.toList());
}
p133 es-search服务上架接口
@Override
public boolean productStatusUp(List<SkuEsModel> skuEsModels) throws IOException {
//批量保存商品
BulkRequest bulkRequest = new BulkRequest();
for (SkuEsModel skuEsModel : skuEsModels) {
//指定索引
IndexRequest indexRequest = new IndexRequest(EsConstant.PRODUCT_INDEX);
//设置索引id
indexRequest.id(skuEsModel.getSkuId().toString());
String str = JSON.toJSONString(skuEsModel);
//设置内容
indexRequest.source(str, XContentType.JSON);
bulkRequest.add(indexRequest);
}
BulkResponse response = restHighLevelClient.bulk(bulkRequest, GulimallEsConfig.COMMON_OPTIONS);
boolean hasFailures = response.hasFailures();
//如果有错误
if (hasFailures){
BulkItemResponse[] items = response.getItems();
List<String> ids = Arrays.stream(items).map(item -> {
return item.getId();
}).collect(Collectors.toList());
log.error("商品上架错误:{}"+ids);
}
return hasFailures;
}
p134 上架流程接口调试
p135 库存系统返回结果泛型修改
p136 整合thymeleaf渲染首页
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-thymeleaf</artifactId>
</dependency>
thymeleaf:
cache: false
suffix: .html
prefix: classpath:/templates/
访问商品服务,进入首页
p137渲染一级分类
themeeleaf热启动
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
</dependency>
后端
@Controller
public class IndexController {
@Autowired
private CategoryService categoryService;
@RequestMapping({"/", "index", "/index.html"})
public String indexPage(Model model) {
// 获取一级分类所有缓存
List<CategoryEntity> catagories = categoryService.getLevel1Categorys();
model.addAttribute("catagories", catagories);
return "index";
}
}
p138渲染二级三级分类
@Override
public Map<Long, List<Catelog2Vo>> getCatelogJson() {
//为了构造相应的json数据 类似一个map key放一级分类id,value放下面的二级分类的集合,二级分类里面又包括三级分类
//先获取所有一级分类
List<CategoryEntity> level1Categorys = getLevel1Categorys();
Map<Long, List<Catelog2Vo>> map = level1Categorys.stream().collect(Collectors.toMap(CategoryEntity::getCatId, cate1 -> {
//查询该一级分类下的二级分类
List<CategoryEntity> level2Cates = baseMapper.selectList(new QueryWrapper<CategoryEntity>().eq("parent_cid", cate1.getCatId()));
List<Catelog2Vo> catelog2Vos = level2Cates.stream().map(l2 -> {
Catelog2Vo catelog2Vo = new Catelog2Vo();
catelog2Vo.setId(l2.getCatId().toString());
catelog2Vo.setCatalog1Id(cate1.getCatId().toString());
catelog2Vo.setName(cate1.getName());
//二级分类下面又有三级分类,去查询 其实这里循环去查数据库不好,只是体验另一种写法
List<CategoryEntity> level3Cates = baseMapper.selectList(new QueryWrapper<CategoryEntity>().eq("parent_cid", l2.getCatId()));
List<Catalog3Vo> catalog3Vos = level3Cates.stream().map(l3 -> {
Catalog3Vo catalog3Vo = new Catalog3Vo();
catalog3Vo.setId(l3.getCatId().toString());
catalog3Vo.setCatalog2Id(l2.getCatId().toString());
catalog3Vo.setName(l3.getName());
return catalog3Vo;
}).collect(Collectors.toList());
catelog2Vo.setCatalog3List(catalog3Vos);
return catelog2Vo;
}).collect(Collectors.toList());
return catelog2Vos;
}
));
return map;
}
p139 nginx反向代理
nginx配置文件
p140 upstream负载均衡
我们要让nginx转发到网关,网关再帮我们访问服务
修改总配置文件
修改server块配置文件
但是,这里有问题,我们在网关配置文件配置了根据host匹配到商品服务,但是nginx丢失了请求头中的host,所以我们加上配置
1744
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
