【rabbitMq整合es实现全文搜索，数据实时同步】

提示：文章写完后，目录可以自动生成，如何生成可参考右边的帮助文档

---

步骤

一、rabbitmq部署

1.拉取镜像

docker pull rabbitmq:3.7.7-management

使用：docker images 查看所有镜像

2、根据下载的镜像创建和启动容器

docker run -d --name rabbitmq3.7.7 -p 5672:5672 -p 15672:15672 -v `pwd`/data:/var/lib/rabbitmq --hostname myRabbit -e RABBITMQ_DEFAULT_VHOST=my_vhost  -e RABBITMQ_DEFAULT_USER=admin -e RABBITMQ_DEFAULT_PASS=admin df80af9ca0c9

说明：

-d 后台运行容器；
–name 指定容器名；
-p 指定服务运行的端口（5672：应用访问端口；15672：控制台Web端口号）；
-v 映射目录或文件；
–hostname 主机名（RabbitMQ的一个重要注意事项是它根据所谓的 “节点名称” 存储数据，默认为主机名）；
-e 指定环境变量；（RABBITMQ_DEFAULT_VHOST：默认虚拟机名；RABBITMQ_DEFAULT_USER：默认的用户名；RABBITMQ_DEFAULT_PASS：默认用户名的密码）

3、使用命令：docker ps 查看正在运行容器

4.可以使用浏览器打开web管理端：http://Server-IP:15672

二、ES部署

使用docker直接获取es镜像，执行命令docker pull elasticsearch:7.7.0
执行完成后，执行docker images即可看到上一步拉取的镜像。

有了镜像，就可以开始创建容器了，接下来创建一个es的容器。

执行

docker run --name elasticsearch -d -e ES_JAVA_OPTS="-Xms512m -Xmx512m" -e "discovery.type=single-node" -p 9200:9200 -p 9300:9300 elasticsearch:7.7.0

–name 表示容器名称

-d: 后台运行容器，并返回容器ID；

-e: 指定容器内的环境变量

-p: 指定端口映射，格式为：主机(宿主)端口:容器端口

三.项目引入ES

代码如下（示例）：
POM.XML

   <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
        </dependency>

由于项目连表查询连了有8，9张，所有没有直接把数据库数据同步到es，这里同步的是查询的结果集，

1.配置类

@Configuration
public class ElasticSearchConfig {
    @Bean
    public RestHighLevelClient restHighLevelClient(){
        RestHighLevelClient client=new RestHighLevelClient(
                RestClient.builder(
                        new HttpHost("127.0.0.1",9200,"http")
                )
        );
        return client;
    }
}

2.全量同步

impl层代码如下（示例）：

import org.apache.lucene.search.join.ScoreMode;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.NestedQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.index.query.RangeQueryBuilder;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.aggregations.AggregationBuilders;
import org.elasticsearch.search.aggregations.bucket.nested.NestedAggregationBuilder;
import org.elasticsearch.search.aggregations.bucket.nested.ParsedNested;
import org.elasticsearch.search.aggregations.bucket.terms.ParsedLongTerms;
import org.elasticsearch.search.aggregations.bucket.terms.ParsedStringTerms;
import org.elasticsearch.search.aggregations.bucket.terms.Terms;
import org.elasticsearch.search.aggregations.bucket.terms.TermsAggregationBuilder;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightField;
import org.elasticsearch.search.sort.SortOrder;


  @Autowired
    private
 RestHighLevelClient restHighLevelClient;

  @Override
    public void getin() throws ExecutionException, InterruptedException, IOException {
        InwuquDto inwuquDto=new InwuquDto();
        List<InwuquVo> inwuquVos = selectInventorysummaryquery(inwuquDto);

        BulkRequest bulkRequest = new BulkRequest();
        inwuquVos.stream().forEach(x -> {
            bulkRequest.add(new IndexRequest("888").source(JSON.toJSONString(x), XContentType.JSON));
        });
        BulkResponse responses=null;
        try {
            responses = restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

该处使用的url网络请求的数据。

---

2.查询

@Override
    public List<InwuquVo> essearch(SearchParam param) throws IOException {
        //1、动态构建出查询需要的DSL语句
        SearchResults result = null;
        List<InwuquVo> s=new ArrayList<>();
        //1、准备检索请求
        SearchRequest searchRequest = buildSearchRequest(param);


            //2、执行检索请求
            SearchResponse response = restHighLevelClient.search(searchRequest, COMMON_OPTIONS);


   result = buildSearchResult(response, param);




return result.getInwuquVos();
    }

    private SearchRequest buildSearchRequest(SearchParam param) {

        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

        /**
         * 模糊匹配，过滤（按照属性，分类，品牌，价格区间，库存）
         */
        //1. 构建bool-query
        BoolQueryBuilder boolQueryBuilder=new BoolQueryBuilder();


        //全文
        if(!StringUtils.isEmpty(param.getCbpb08())){
            boolQueryBuilder.must(QueryBuilders.matchQuery("cbpb08",param.getCbpb08()));
        }


        if(!StringUtils.isEmpty(param.getCbig10())){
            boolQueryBuilder.must(QueryBuilders.matchQuery("cbig10",param.getCbig10()));
        }


        //1.1 bool-must
        if(!StringUtils.isEmpty(param.getKeyword())){
            boolQueryBuilder.must(QueryBuilders.matchQuery("skuTitle",param.getKeyword()));
        }

        //1.2 bool-fiter
        //1.2.1 catelogId
        if(null != param.getCatalog3Id()){
            boolQueryBuilder.filter(QueryBuilders.termQuery("catalogId",param.getCatalog3Id()));
        }

        //1.2.2 brandId
        if(null != param.getBrandId() && param.getBrandId().size() >0){
            boolQueryBuilder.filter(QueryBuilders.termsQuery("brandId",param.getBrandId()));
        }

        //1.2.3 attrs
        if(param.getAttrs() != null && param.getAttrs().size() > 0){

            param.getAttrs().forEach(item -> {
                //attrs=1_5寸:8寸&2_16G:8G
                BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();


                //attrs=1_5寸:8寸
                String[] s = item.split("_");
                String attrId=s[0];
                String[] attrValues = s[1].split(":");//这个属性检索用的值
                boolQuery.must(QueryBuilders.termQuery("attrs.attrId",attrId));
                boolQuery.must(QueryBuilders.termsQuery("attrs.attrValue",attrValues));

                NestedQueryBuilder nestedQueryBuilder = QueryBuilders.nestedQuery("attrs",boolQuery, ScoreMode.None);
                boolQueryBuilder.filter(nestedQueryBuilder);
            });

        }

        //1.2.4 hasStock
        if(null != param.getHasStock()){
            boolQueryBuilder.filter(QueryBuilders.termQuery("hasStock",param.getHasStock() == 1));
        }


        //1.2.5 skuPrice
        if(!StringUtils.isEmpty(param.getSkuPrice())){
            //skuPrice形式为：1_500或_500或500_
            RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("skuPrice");
            String[] price = param.getSkuPrice().split("_");
            if(price.length==2){
                rangeQueryBuilder.gte(price[0]).lte(price[1]);
            }else if(price.length == 1){
                if(param.getSkuPrice().startsWith("_")){
                    rangeQueryBuilder.lte(price[1]);
                }
                if(param.getSkuPrice().endsWith("_")){
                    rangeQueryBuilder.gte(price[0]);
                }
            }
            boolQueryBuilder.filter(rangeQueryBuilder);
        }

        //封装所有的查询条件
        searchSourceBuilder.query(boolQueryBuilder);


        /**
         * 排序，分页，高亮
         */

        //排序
        //形式为sort=hotScore_asc/desc
        if(!StringUtils.isEmpty(param.getSort())){
            String sort = param.getSort();
            String[] sortFileds = sort.split("_");

            SortOrder sortOrder="asc".equalsIgnoreCase(sortFileds[1])?SortOrder.ASC:SortOrder.DESC;

            searchSourceBuilder.sort(sortFileds[0],sortOrder);
        }

        //分页
        searchSourceBuilder.from((param.getPageNum()-1)*15);
        searchSourceBuilder.size(15);

        //高亮
        if(!StringUtils.isEmpty(param.getKeyword())){

            HighlightBuilder highlightBuilder = new HighlightBuilder();
            highlightBuilder.field("skuTitle");
            highlightBuilder.preTags("<b style='color:red'>");
            highlightBuilder.postTags("</b>");

            searchSourceBuilder.highlighter(highlightBuilder);
        }



        /**
         * 聚合分析
         */
        //1. 按照品牌进行聚合
        TermsAggregationBuilder brand_agg = AggregationBuilders.terms("brand_agg");
        brand_agg.field("brandId").size(50);


        //1.1 品牌的子聚合-品牌名聚合
        brand_agg.subAggregation(AggregationBuilders.terms("brand_name_agg")
                .field("brandName").size(1));
        //1.2 品牌的子聚合-品牌图片聚合
        brand_agg.subAggregation(AggregationBuilders.terms("brand_img_agg")
                .field("brandImg").size(1));

        searchSourceBuilder.aggregation(brand_agg);

        //2. 按照分类信息进行聚合
        TermsAggregationBuilder catalog_agg = AggregationBuilders.terms("catalog_agg");
        catalog_agg.field("catalogId").size(20);

        catalog_agg.subAggregation(AggregationBuilders.terms("catalog_name_agg").field("catalogName").size(1));

        searchSourceBuilder.aggregation(catalog_agg);

        //2. 按照属性信息进行聚合
        NestedAggregationBuilder attr_agg = AggregationBuilders.nested("attr_agg", "attrs");
        //2.1 按照属性ID进行聚合
        TermsAggregationBuilder attr_id_agg = AggregationBuilders.terms("attr_id_agg").field("attrs.attrId");
        attr_agg.subAggregation(attr_id_agg);
        //2.1.1 在每个属性ID下，按照属性名进行聚合
        attr_id_agg.subAggregation(AggregationBuilders.terms("attr_name_agg").field("attrs.attrName").size(1));
        //2.1.1 在每个属性ID下，按照属性值进行聚合
        attr_id_agg.subAggregation(AggregationBuilders.terms("attr_value_agg").field("attrs.attrValue").size(50));
        searchSourceBuilder.aggregation(attr_agg);

        log.debug("构建的DSL语句 {}",searchSourceBuilder.toString());

        SearchRequest searchRequest = new SearchRequest(new String[]{"888"},searchSourceBuilder);

        return searchRequest;
    }


    private SearchResults buildSearchResult(SearchResponse response,SearchParam param) {

        SearchResults result = new SearchResults();

        //1、返回的所有查询到的商品
        SearchHits hits = response.getHits();

        List<InwuquVo> inwuquVos = new ArrayList<>();
        //遍历所有商品信息
        if (hits.getHits() != null && hits.getHits().length > 0) {
            for (SearchHit hit : hits.getHits()) {
                String sourceAsString = hit.getSourceAsString();
                InwuquVo esModel = JSON.parseObject(sourceAsString, InwuquVo.class);

                //判断是否按关键字检索，若是就显示高亮，否则不显示
                if (!StringUtils.isEmpty(param.getKeyword())) {
                    //拿到高亮信息显示标题
                    HighlightField skuTitle = hit.getHighlightFields().get("cbpb08");
                    String skuTitleValue = skuTitle.getFragments()[0].string();
                    esModel.setCbpb08(skuTitleValue);
                }
                inwuquVos.add(esModel);
            }
        }
        result.setInwuquVos(inwuquVos);


        return result;
    }

四.项目引入rabbitmq

1.配置类

public class RabbitMQConfig {


    public static final String EXCHANGE_NAME = "boot_topic_exchange";
    public static final String QUEUE_NAME = "boot_queue";

    //1.交换机
    @Bean("bootExchange")
    public Exchange bootExchange(){
        return ExchangeBuilder.topicExchange(EXCHANGE_NAME).durable(true).build();
    }


    //2.Queue 队列
    @Bean("bootQueue")
    public Queue bootQueue(){
        return QueueBuilder.durable(QUEUE_NAME).build();
    }

    //3. 队列和交互机绑定关系 Binding
    /*
        1. 知道哪个队列
        2. 知道哪个交换机
        3. routing key
     */
    @Bean
    public Binding bindQueueExchange(@Qualifier("bootQueue") Queue queue, @Qualifier("bootExchange") Exchange exchange){
        return BindingBuilder.bind(queue).to(exchange).with("boot.#").noargs();
    }
}

yml

  rabbitmq:
    host: 43.140.250.190 # ip
    port: 5672
    username: admin
    password: admin
    virtual-host: my_vhost

2.生产者/增量同步

这里传的方法很多，如果不要求，传对象是最方便的

GsGoodsSn po=new GsGoodsSn();
            po.setSn(cbicDto.getCbic10());
         //   String jsonStr = objectMapper.writeValueAsString(po);

            rabbitTemplate.convertAndSend(RabbitMQConfig.EXCHANGE_NAME,
                    "boot.haha",po);

3.消费者/增量同步

import com.alibaba.fastjson2.JSON;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.ruoyi.system.domain.GsGoodsSn;
import com.ruoyi.system.domain.vo.InwuquVo;
import lombok.SneakyThrows;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.xcontent.XContentType;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.messaging.handler.annotation.Payload;
import org.springframework.stereotype.Component;

import javax.annotation.Resource;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

@Component
public class RabbimtMQListener {
    @Resource
    protected RestHighLevelClient restHighLevelClient;



    @SneakyThrows
    @RabbitListener(queues = "boot_queue")
    public void ListenerQueue(@Payload GsGoodsSn gs) {
        //System.out.println(message);

   //     List<GsGoodsSn> stringList1 = JSON.parseObject(s).getJSONArray("data").toJavaList(GsGoodsSn.class);

        //json字符串转成Java对象
        List<GsGoodsSn> gsGoodsSns=new ArrayList<>();
        gsGoodsSns.add(gs);
        add(gsGoodsSns);

        System.out.println(gs);

    }

//es增量同步
    private void add( List<GsGoodsSn> gsGoodsSns){
        BulkRequest bulkRequest = new BulkRequest();
        gsGoodsSns.stream().forEach(x -> {
            bulkRequest.add(new IndexRequest("888").source(JSON.toJSONString(x), XContentType.JSON));
        });
        BulkResponse responses=null;
        try {
            responses = restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

}

总结

最后，增量同步是通过接口触发，调用接口，消费者自动监听数据进行消费，把数据同步到es。