提示:文章写完后,目录可以自动生成,如何生成可参考右边的帮助文档
rabbitMq整合es实现全文搜索,数据实时同步
步骤
一、rabbitmq部署
1.拉取镜像
docker pull rabbitmq:3.7.7-management
使用:docker images 查看所有镜像
2、根据下载的镜像创建和启动容器
docker run -d --name rabbitmq3.7.7 -p 5672:5672 -p 15672:15672 -v `pwd`/data:/var/lib/rabbitmq --hostname myRabbit -e RABBITMQ_DEFAULT_VHOST=my_vhost -e RABBITMQ_DEFAULT_USER=admin -e RABBITMQ_DEFAULT_PASS=admin df80af9ca0c9
说明:
-d 后台运行容器;
–name 指定容器名;
-p 指定服务运行的端口(5672:应用访问端口;15672:控制台Web端口号);
-v 映射目录或文件;
–hostname 主机名(RabbitMQ的一个重要注意事项是它根据所谓的 “节点名称” 存储数据,默认为主机名);
-e 指定环境变量;(RABBITMQ_DEFAULT_VHOST:默认虚拟机名;RABBITMQ_DEFAULT_USER:默认的用户名;RABBITMQ_DEFAULT_PASS:默认用户名的密码)
3、使用命令:docker ps 查看正在运行容器
4.可以使用浏览器打开web管理端:http://Server-IP:15672
二、ES部署
使用docker直接获取es镜像,执行命令docker pull elasticsearch:7.7.0
执行完成后,执行docker images即可看到上一步拉取的镜像。
有了镜像,就可以开始创建容器了,接下来创建一个es的容器。
执行
docker run --name elasticsearch -d -e ES_JAVA_OPTS="-Xms512m -Xmx512m" -e "discovery.type=single-node" -p 9200:9200 -p 9300:9300 elasticsearch:7.7.0
–name 表示容器名称
-d: 后台运行容器,并返回容器ID;
-e: 指定容器内的环境变量
-p: 指定端口映射,格式为:主机(宿主)端口:容器端口
三.项目引入ES
代码如下(示例):
POM.XML
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
由于项目连表查询连了有8,9张,所有没有直接把数据库数据同步到es,这里同步的是查询的结果集,
1.配置类
@Configuration
public class ElasticSearchConfig {
@Bean
public RestHighLevelClient restHighLevelClient(){
RestHighLevelClient client=new RestHighLevelClient(
RestClient.builder(
new HttpHost("127.0.0.1",9200,"http")
)
);
return client;
}
}
2.全量同步
impl层代码如下(示例):
import org.apache.lucene.search.join.ScoreMode;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.NestedQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.index.query.RangeQueryBuilder;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.aggregations.AggregationBuilders;
import org.elasticsearch.search.aggregations.bucket.nested.NestedAggregationBuilder;
import org.elasticsearch.search.aggregations.bucket.nested.ParsedNested;
import org.elasticsearch.search.aggregations.bucket.terms.ParsedLongTerms;
import org.elasticsearch.search.aggregations.bucket.terms.ParsedStringTerms;
import org.elasticsearch.search.aggregations.bucket.terms.Terms;
import org.elasticsearch.search.aggregations.bucket.terms.TermsAggregationBuilder;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightField;
import org.elasticsearch.search.sort.SortOrder;
@Autowired
private
RestHighLevelClient restHighLevelClient;
@Override
public void getin() throws ExecutionException, InterruptedException, IOException {
InwuquDto inwuquDto=new InwuquDto();
List<InwuquVo> inwuquVos = selectInventorysummaryquery(inwuquDto);
BulkRequest bulkRequest = new BulkRequest();
inwuquVos.stream().forEach(x -> {
bulkRequest.add(new IndexRequest("888").source(JSON.toJSONString(x), XContentType.JSON));
});
BulkResponse responses=null;
try {
responses = restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
e.printStackTrace();
}
}
该处使用的url网络请求的数据。
2.查询
@Override
public List<InwuquVo> essearch(SearchParam param) throws IOException {
//1、动态构建出查询需要的DSL语句
SearchResults result = null;
List<InwuquVo> s=new ArrayList<>();
//1、准备检索请求
SearchRequest searchRequest = buildSearchRequest(param);
//2、执行检索请求
SearchResponse response = restHighLevelClient.search(searchRequest, COMMON_OPTIONS);
result = buildSearchResult(response, param);
return result.getInwuquVos();
}
private SearchRequest buildSearchRequest(SearchParam param) {
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
/**
* 模糊匹配,过滤(按照属性,分类,品牌,价格区间,库存)
*/
//1. 构建bool-query
BoolQueryBuilder boolQueryBuilder=new BoolQueryBuilder();
//全文
if(!StringUtils.isEmpty(param.getCbpb08())){
boolQueryBuilder.must(QueryBuilders.matchQuery("cbpb08",param.getCbpb08()));
}
if(!StringUtils.isEmpty(param.getCbig10())){
boolQueryBuilder.must(QueryBuilders.matchQuery("cbig10",param.getCbig10()));
}
//1.1 bool-must
if(!StringUtils.isEmpty(param.getKeyword())){
boolQueryBuilder.must(QueryBuilders.matchQuery("skuTitle",param.getKeyword()));
}
//1.2 bool-fiter
//1.2.1 catelogId
if(null != param.getCatalog3Id()){
boolQueryBuilder.filter(QueryBuilders.termQuery("catalogId",param.getCatalog3Id()));
}
//1.2.2 brandId
if(null != param.getBrandId() && param.getBrandId().size() >0){
boolQueryBuilder.filter(QueryBuilders.termsQuery("brandId",param.getBrandId()));
}
//1.2.3 attrs
if(param.getAttrs() != null && param.getAttrs().size() > 0){
param.getAttrs().forEach(item -> {
//attrs=1_5寸:8寸&2_16G:8G
BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
//attrs=1_5寸:8寸
String[] s = item.split("_");
String attrId=s[0];
String[] attrValues = s[1].split(":");//这个属性检索用的值
boolQuery.must(QueryBuilders.termQuery("attrs.attrId",attrId));
boolQuery.must(QueryBuilders.termsQuery("attrs.attrValue",attrValues));
NestedQueryBuilder nestedQueryBuilder = QueryBuilders.nestedQuery("attrs",boolQuery, ScoreMode.None);
boolQueryBuilder.filter(nestedQueryBuilder);
});
}
//1.2.4 hasStock
if(null != param.getHasStock()){
boolQueryBuilder.filter(QueryBuilders.termQuery("hasStock",param.getHasStock() == 1));
}
//1.2.5 skuPrice
if(!StringUtils.isEmpty(param.getSkuPrice())){
//skuPrice形式为:1_500或_500或500_
RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("skuPrice");
String[] price = param.getSkuPrice().split("_");
if(price.length==2){
rangeQueryBuilder.gte(price[0]).lte(price[1]);
}else if(price.length == 1){
if(param.getSkuPrice().startsWith("_")){
rangeQueryBuilder.lte(price[1]);
}
if(param.getSkuPrice().endsWith("_")){
rangeQueryBuilder.gte(price[0]);
}
}
boolQueryBuilder.filter(rangeQueryBuilder);
}
//封装所有的查询条件
searchSourceBuilder.query(boolQueryBuilder);
/**
* 排序,分页,高亮
*/
//排序
//形式为sort=hotScore_asc/desc
if(!StringUtils.isEmpty(param.getSort())){
String sort = param.getSort();
String[] sortFileds = sort.split("_");
SortOrder sortOrder="asc".equalsIgnoreCase(sortFileds[1])?SortOrder.ASC:SortOrder.DESC;
searchSourceBuilder.sort(sortFileds[0],sortOrder);
}
//分页
searchSourceBuilder.from((param.getPageNum()-1)*15);
searchSourceBuilder.size(15);
//高亮
if(!StringUtils.isEmpty(param.getKeyword())){
HighlightBuilder highlightBuilder = new HighlightBuilder();
highlightBuilder.field("skuTitle");
highlightBuilder.preTags("<b style='color:red'>");
highlightBuilder.postTags("</b>");
searchSourceBuilder.highlighter(highlightBuilder);
}
/**
* 聚合分析
*/
//1. 按照品牌进行聚合
TermsAggregationBuilder brand_agg = AggregationBuilders.terms("brand_agg");
brand_agg.field("brandId").size(50);
//1.1 品牌的子聚合-品牌名聚合
brand_agg.subAggregation(AggregationBuilders.terms("brand_name_agg")
.field("brandName").size(1));
//1.2 品牌的子聚合-品牌图片聚合
brand_agg.subAggregation(AggregationBuilders.terms("brand_img_agg")
.field("brandImg").size(1));
searchSourceBuilder.aggregation(brand_agg);
//2. 按照分类信息进行聚合
TermsAggregationBuilder catalog_agg = AggregationBuilders.terms("catalog_agg");
catalog_agg.field("catalogId").size(20);
catalog_agg.subAggregation(AggregationBuilders.terms("catalog_name_agg").field("catalogName").size(1));
searchSourceBuilder.aggregation(catalog_agg);
//2. 按照属性信息进行聚合
NestedAggregationBuilder attr_agg = AggregationBuilders.nested("attr_agg", "attrs");
//2.1 按照属性ID进行聚合
TermsAggregationBuilder attr_id_agg = AggregationBuilders.terms("attr_id_agg").field("attrs.attrId");
attr_agg.subAggregation(attr_id_agg);
//2.1.1 在每个属性ID下,按照属性名进行聚合
attr_id_agg.subAggregation(AggregationBuilders.terms("attr_name_agg").field("attrs.attrName").size(1));
//2.1.1 在每个属性ID下,按照属性值进行聚合
attr_id_agg.subAggregation(AggregationBuilders.terms("attr_value_agg").field("attrs.attrValue").size(50));
searchSourceBuilder.aggregation(attr_agg);
log.debug("构建的DSL语句 {}",searchSourceBuilder.toString());
SearchRequest searchRequest = new SearchRequest(new String[]{"888"},searchSourceBuilder);
return searchRequest;
}
private SearchResults buildSearchResult(SearchResponse response,SearchParam param) {
SearchResults result = new SearchResults();
//1、返回的所有查询到的商品
SearchHits hits = response.getHits();
List<InwuquVo> inwuquVos = new ArrayList<>();
//遍历所有商品信息
if (hits.getHits() != null && hits.getHits().length > 0) {
for (SearchHit hit : hits.getHits()) {
String sourceAsString = hit.getSourceAsString();
InwuquVo esModel = JSON.parseObject(sourceAsString, InwuquVo.class);
//判断是否按关键字检索,若是就显示高亮,否则不显示
if (!StringUtils.isEmpty(param.getKeyword())) {
//拿到高亮信息显示标题
HighlightField skuTitle = hit.getHighlightFields().get("cbpb08");
String skuTitleValue = skuTitle.getFragments()[0].string();
esModel.setCbpb08(skuTitleValue);
}
inwuquVos.add(esModel);
}
}
result.setInwuquVos(inwuquVos);
return result;
}
四.项目引入rabbitmq
1.配置类
public class RabbitMQConfig {
public static final String EXCHANGE_NAME = "boot_topic_exchange";
public static final String QUEUE_NAME = "boot_queue";
//1.交换机
@Bean("bootExchange")
public Exchange bootExchange(){
return ExchangeBuilder.topicExchange(EXCHANGE_NAME).durable(true).build();
}
//2.Queue 队列
@Bean("bootQueue")
public Queue bootQueue(){
return QueueBuilder.durable(QUEUE_NAME).build();
}
//3. 队列和交互机绑定关系 Binding
/*
1. 知道哪个队列
2. 知道哪个交换机
3. routing key
*/
@Bean
public Binding bindQueueExchange(@Qualifier("bootQueue") Queue queue, @Qualifier("bootExchange") Exchange exchange){
return BindingBuilder.bind(queue).to(exchange).with("boot.#").noargs();
}
}
yml
rabbitmq:
host: 43.140.250.190 # ip
port: 5672
username: admin
password: admin
virtual-host: my_vhost
2.生产者/增量同步
这里传的方法很多,如果不要求,传对象是最方便的
GsGoodsSn po=new GsGoodsSn();
po.setSn(cbicDto.getCbic10());
// String jsonStr = objectMapper.writeValueAsString(po);
rabbitTemplate.convertAndSend(RabbitMQConfig.EXCHANGE_NAME,
"boot.haha",po);
3.消费者/增量同步
import com.alibaba.fastjson2.JSON;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.ruoyi.system.domain.GsGoodsSn;
import com.ruoyi.system.domain.vo.InwuquVo;
import lombok.SneakyThrows;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.xcontent.XContentType;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.messaging.handler.annotation.Payload;
import org.springframework.stereotype.Component;
import javax.annotation.Resource;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
@Component
public class RabbimtMQListener {
@Resource
protected RestHighLevelClient restHighLevelClient;
@SneakyThrows
@RabbitListener(queues = "boot_queue")
public void ListenerQueue(@Payload GsGoodsSn gs) {
//System.out.println(message);
// List<GsGoodsSn> stringList1 = JSON.parseObject(s).getJSONArray("data").toJavaList(GsGoodsSn.class);
//json字符串转成Java对象
List<GsGoodsSn> gsGoodsSns=new ArrayList<>();
gsGoodsSns.add(gs);
add(gsGoodsSns);
System.out.println(gs);
}
//es增量同步
private void add( List<GsGoodsSn> gsGoodsSns){
BulkRequest bulkRequest = new BulkRequest();
gsGoodsSns.stream().forEach(x -> {
bulkRequest.add(new IndexRequest("888").source(JSON.toJSONString(x), XContentType.JSON));
});
BulkResponse responses=null;
try {
responses = restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
e.printStackTrace();
}
}
}
总结
最后,增量同步是通过接口触发,调用接口,消费者自动监听数据进行消费,把数据同步到es。