elasticsearch搜索中文分词理解、类似SQL语句中的"LIKE”条件的模糊搜索以及忽略大小写搜索

elasticsearch作为一款搜索引擎,应用于数据库无法承受前端的搜索压力时,采用其进行数据的搜索。可以大并发架构设计中一种选择,以下是elasticsearch搜索引擎的部分规则,在实际应用中可以让我们快速熟悉和帮助解决一些问题。

01》不进行分词的索引建立

URL: es_index_test

{
  "settings": {
    "index": {
      "number_of_shards": "4",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "es_index_type_test": {
      "properties": {
        "productId": {
          "type": "text"
        },
        "productName": {
          "type": "keyword",
          "index": "true"
        }
      }
    }
  }

}

说明:“productName”属性建立索引时,将其设置为不进行分词设置。利用wildcard搜索方式,可以实现MYSQL中的LIKE效果。例如:文档{"productId":10001,"productName":"山鸡图"},可以用{"query":{"wildcard":{"productName":"*鸡*"}}},搜索出来。

02》需要进行分词的索引建立

URL: es_index_test

{
  "settings": {
    "index": {
      "number_of_shards": "4",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "es_index_type_test": {
      "properties": {
        "productId": {
        "type": "text"
        },
        "productName": {
          "type": "text",
          "analyzer": "ik_max_word",
          "search_analyzer": "ik_max_word"
        }
      }
    }
  }

}

说明:“productName”属性建立索引时,将其设置为进行分词设置。elasticsearch默认针对中文的分词是按照一个中文字符,就是一个分词。例如:文档{"productId":10001,"productName":"山鸡图"}中,会拆分为“山”、“鸡”和“图”三个分词。中文分词的拆分,可以安装ik分词器进行分词拆分。例如:文档{"productId":10001,"productName":"山鸡图"}中,会拆分为“山鸡”和“图”两个分词。中文短句具体拆分成哪些分词是ik分词器的字典来识别的,此字典可以根据实际情况进行调整。

03》忽略大小写的索引建立

URL: es_index_test
{
  "settings": {
    "index": {
      "number_of_shards": "10",
      "number_of_replicas": "3"
    },
    "analysis": {
      "normalizer": {
        "es_normalizer": {
          "filter": [
            "lowercase",
            "asciifolding"
          ],
          "type": "custom"
        }
      }
    }
  },
  "mappings": {
    "es_index_test": {
      "properties": {
        "productId": {
          "type": "text"
        },
        "productName": {
          "type": "keyword",
          "normalizer": "es_normalizer",
          "index": "true"
        }
      }
    }
  }

}

说明:“productName”属性建立索引时,将其设置为忽略大小写。

04》分词查询

URL:es_index_test/es_index_type_test/_analyze

  • ik分词器以“ik_max_word”方式拆分
{
  "analyzer":"ik_max_word",
  "text":"中华人民共和国"

}

  • 结果

{
  "tokens": [
    {
      "end_offset": 7,
      "start_offset": 0,
      "position": 0,
      "type": "CN_WORD",
      "token": "中华人民共和国"
    },
    {
      "end_offset": 4,
      "start_offset": 0,
      "position": 1,
      "type": "CN_WORD",
      "token": "中华人民"
    },
    {
      "end_offset": 2,
      "start_offset": 0,
      "position": 2,
      "type": "CN_WORD",
      "token": "中华"
    },
    {
      "end_offset": 3,
      "start_offset": 1,
      "position": 3,
      "type": "CN_WORD",
      "token": "华人"
    },
    {
      "end_offset": 7,
      "start_offset": 2,
      "position": 4,
      "type": "CN_WORD",
      "token": "人民共和国"
    },
    {
      "end_offset": 4,
      "start_offset": 2,
      "position": 5,
      "type": "CN_WORD",
      "token": "人民"
    },
    {
      "end_offset": 7,
      "start_offset": 4,
      "position": 6,
      "type": "CN_WORD",
      "token": "共和国"
    },
    {
      "end_offset": 6,
      "start_offset": 4,
      "position": 7,
      "type": "CN_WORD",
      "token": "共和"
    },
    {
      "end_offset": 7,
      "start_offset": 6,
      "position": 8,
      "type": "CN_CHAR",
      "token": "国"
    }
  ]

}

  • ik分词器以“ik_smart”方式拆分

{
  "analyzer":"ik_smart",
  "text":"中华人民共和国"

}

  • 结果

{

"tokens": [

    {
      "end_offset": 7,
      "start_offset": 0,
      "position": 0,
      "type": "CN_WORD",
      "token": "中华人民共和国"
    }
  ]

}

  • ES默认

{
  "text":"中华人民共和国"

}

  • 结果

{

  "tokens": [
    {
      "end_offset": 1,
      "start_offset": 0,
      "position": 0,
      "type": "<IDEOGRAPHIC>",
      "token": "中"
    },
    {
      "end_offset": 2,
      "start_offset": 1,
      "position": 1,
      "type": "<IDEOGRAPHIC>",
      "token": "华"
    },
    {
      "end_offset": 3,
      "start_offset": 2,
      "position": 2,
      "type": "<IDEOGRAPHIC>",
      "token": "人"
    },
    {
      "end_offset": 4,
      "start_offset": 3,
      "position": 3,
      "type": "<IDEOGRAPHIC>",
      "token": "民"
    },
    {
      "end_offset": 5,
      "start_offset": 4,
      "position": 4,
      "type": "<IDEOGRAPHIC>",
      "token": "共"
    },
    {
      "end_offset": 6,
      "start_offset": 5,
      "position": 5,
      "type": "<IDEOGRAPHIC>",
      "token": "和"
    },
    {
      "end_offset": 7,
      "start_offset": 6,
      "position": 6,
      "type": "<IDEOGRAPHIC>",
      "token": "国"
    }
  ]

}

说明:以上三种分词拆分的方式不一样,最终产生分词的结果不相同。

05》数据查询-wildcard

URL: es_index_test/es_index_type_test/_search
{
  "query":{"wildcard":{"productName": "山鸡图" }}

}

说明:wildcard种查询方式需要结合方法支持匹配符合,例如:*鸡*,ES会去匹配,在JAVA程序中构建采用。JAVA程序中采用QueryBuilders类的wildcardQuery(String name, Object text)方法。

06》数据查询-match

URL:es_index_test/es_index_type_test/_search
{
  "query":{"match":{"productName": "山鸡图" }}

}

说明:查询时会根据分词进行匹配,例如:“山鸡图”ES拆分为“山鸡”和“图”两个分词到ES搜索引擎内筛选出记录,最后将符合记录的数据返回。返回的记录可能包含,山鸡汤(包含“山鸡”)和山虎图(包含“图”分词)。JAVA程序中采用QueryBuilders类的matchQuery(String name, Object text)方法。

07》数据查询-term

URL: es_index_test/es_index_type_test/_search
{
  "query":{
    "term":{
      "productName":"山鸡图"
    }
  }
}
说明:只有分词完全匹配“山鸡图”这三个字后,才可以返回数据。JAVA程序中采用QueryBuilders类的termQuery(String name, Object value)方法。

08》数据查询-terms

URL: es_index_test/es_index_type_test/_search

{
  "query":{
    "terms":{
      "productName":["山鸡图","山虎图"]
    }
  }

}

说明:分词匹配“山鸡图”和“山虎图”返回记录。JAVA程序中采用QueryBuilders类的termsQuery(String name, String... values)方法。

09》删除查询出来的结果集

URL:es_index_test/es_index_type_test/_delete_by_query

{
  "query":{"wildcard":{"productName": "*鸡*" }}

}

说明:删除产品名称包含“鸡”字文档。

10》elasticsearch中JAVA实例

1、ElasticSearchProperties
package com.jd.ccc.sys.biz.yb.op.notice.config;

import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;

import lombok.Data;

/**
 * ElasticSearch搜索引擎配置参数 
 * 具体参数的配置信息在yml文件内
 * 
 * @create 2018-5-10
 * @author zhangqiang200<https://blog.csdn.net/zhangqiang_accp>
 *
 */
@Data
@Component
@ConfigurationProperties(prefix = "elasticsearch")
public class ElasticSearchProperties {
   /**
    * 集群名
    */
   private String clusterName;
   /**
    * 索引名称
    */
   private String indexName;

   /**
    * 类型名称
    */
   private String typeName;

   /**
    * 主节点
    */
   private String masterNode;

   /**
    * 从节点
    */
   private String slaveNodes;

}
2、ElasticSearchConfig
package com.jd.ccc.sys.biz.yb.op.notice.config;

import java.net.InetAddress;
import java.net.UnknownHostException;

import org.elasticsearch.client.Client;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.InetSocketTransportAddress;
import org.elasticsearch.transport.client.PreBuiltTransportClient;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * 
 * 初始化一个ES搜索引擎配置
 * 
 * @create 2018-5-10
 * @author zhangqiang200<https://blog.csdn.net/zhangqiang_accp>
 *
 */
@Configuration
@EnableConfigurationProperties(ElasticSearchProperties.class)
public class ElasticSearchConfig {
   
    private static final Logger LOGGER = LoggerFactory.getLogger(ElasticSearchConfig.class);
   
   @Autowired
   private ElasticSearchProperties elasticSearchProperties;
   
   private static final String SYS_PROPERTY="es.set.netty.runtime.available.processors";
   
   private static final String CLUSTER_NAME="cluster.name";
   
   private static final String CLIENT_SNIFF="client.transport.sniff";

   @Bean(name="elasticSearchCluster")
   public Client getClient() {

      System.setProperty(SYS_PROPERTY, "false");
      Settings settings = Settings.builder().put(CLUSTER_NAME, elasticSearchProperties.getClusterName())
            .put(CLIENT_SNIFF, false).build();
      
      TransportClient transportClient = null;
      try {

         String[] masters = elasticSearchProperties.getMasterNode().split(":");
         transportClient = new PreBuiltTransportClient(settings).addTransportAddress(
               new InetSocketTransportAddress(InetAddress.getByName(masters[0]), Integer.parseInt(masters[1])));

         String[] slaveNodes = elasticSearchProperties.getSlaveNodes().split(",");// 逗号分隔

         //遍历从库信息
         for (String node : slaveNodes) {
            String[] ipPort = node.split(":");// 冒号分隔
            if (ipPort.length == 2) {
               transportClient.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(ipPort[0]),
                     Integer.parseInt(ipPort[1])));
            }
         }
         return transportClient;
      } catch (UnknownHostException e) {
         LOGGER.error("ES 客户端连接失败.{}",e);
         return null;
      }
   }

}
3、服务层操作
/**
 * 查询模糊搜索产品列表的总记录数
 *
 * @param likeProductName
 *            模糊搜索产品名称的关键字
 * @param type
 *            产品类型
 * @return 总记录数
 *
 * @create 2018-5-9
 * @author zhangqiang200<https://blog.csdn.net/zhangqiang_accp>
 */
private Integer queryCount(String likeProductName, String type) {
    BoolQueryBuilder builder=this.builderQueryData(likeProductName, type);
    try {
        SearchResponse searchResponse = elasticSearchCluster.prepareSearch(elasticSearchProperties.getIndexName())
                .setTypes(elasticSearchProperties.getTypeName()).setQuery(builder)
                .setSearchType(SearchType.DEFAULT).get();
        SearchHits hits = searchResponse.getHits();
        return (int)hits.getTotalHits();
    }catch(Exception e) {
        LOGGER.error("Server access failure,{}",e);
        return 0;
    }
}

/**
 * 拼接模糊查询筛选条件
 *
 * @param likeProductName
 *            模糊搜索产品名称的关键字
 * @param type
 *            产品类型
 * @return 筛选条件字符串
 *
 * @create 2018-5-9
 * @author zhangqiang200<https://blog.csdn.net/zhangqiang_accp>
 */
private BoolQueryBuilder builderQueryData(String likeProductName, String type) {
    BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
    boolQueryBuilder.must(QueryBuilders.matchQuery(PRODUCT_STATUS, "03"));
    if(StringUtils.isNotBlank(likeProductName)) {
        boolQueryBuilder.must(QueryBuilders.wildcardQuery(PRODUCT_NAME,"*"+likeProductName+"*"));
    }
    // 类型不为空
    if (StringUtils.isNotBlank(type)) {
        String[] types = type.split(",");
        if (types.length == 1) {
            boolQueryBuilder.must(QueryBuilders.matchQuery(INST_TYPE,type));
        } else {
            boolQueryBuilder.must(QueryBuilders.termsQuery(INST_TYPE, types));
        }
    }
    LOGGER.debug("wild card query-->{}",boolQueryBuilder.toString());
    return boolQueryBuilder;
}

/**
 * 模糊查询商品列表数据
 * @param likeProductName 模糊搜索产品名称的关键字
 * @param type 产品类型
 * @param startIndex 开始索引
 * @param pageSize 每页大小
 * @returnW
 *
 * @create 2018-5-9
 * @author zhangqiang200<https://blog.csdn.net/zhangqiang_accp>
 */
private List<String> queryData(String likeProductName, String type, int startIndex, int pageSize) {
    List<String> resultList = new ArrayList<>();
    BoolQueryBuilder builder=this.builderQueryData(likeProductName, type);
    try {

        SearchResponse searchResponse = elasticSearchCluster.prepareSearch(elasticSearchProperties.getIndexName())
                .setTypes(elasticSearchProperties.getTypeName()).setQuery(builder)
                .setSearchType(SearchType.DEFAULT).setFrom(startIndex).setSize(pageSize).get();
        SearchHit[] hits = searchResponse.getHits().getHits();


        for (SearchHit hit : hits) {
            resultList.add(hit.getSourceAsString());
        }
    }catch(Exception e) {
        LOGGER.error("Server access failure,{}",e);
    }

    return resultList;
}

  • 6
    点赞
  • 18
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
在使用Spring集成Elasticsearch7时,可以通过设置索引的映射和查询时使用的分析器来实现忽略大小写的功能。 首先,在创建索引时,可以使用字段的映射设置来指定对应字段的分析器。通过设置分析器为"keyword_lowercase",可以将字段设置为忽略大小写。例如,以下代码片段演示了如何创建一个忽略大小写的索引: ```java CreateIndexRequest createIndexRequest = new CreateIndexRequest("my_index"); createIndexRequest.settings(Settings.builder() .put("index.analysis.analyzer.default.type", "custom") .put("index.analysis.analyzer.default.tokenizer", "keyword") .put("index.analysis.analyzer.default.filter", "lowercase") .build()); Mapping mapping = new Mapping.Builder() .field(new StringField("my_field") .analyzer("keyword_lowercase")) .build(); createIndexRequest.mapping(mapping); CreateIndexResponse response = restHighLevelClient.indices().create(createIndexRequest, RequestOptions.DEFAULT); ``` 在查询时,我们可以使用Spring Data Elasticsearch来构造查询请求。通过使用`QueryBuilders`类提供的`matchQuery`方法,并指定字段为忽略大小写的分析器,可以实现忽略大小写的查询。例如,以下代码片段演示了如何使用忽略大小写的方式进行查询: ```java QueryBuilder queryBuilder = QueryBuilders.matchQuery("my_field", "value") .analyzer("keyword_lowercase"); SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder(); searchSourceBuilder.query(queryBuilder); SearchRequest searchRequest = new SearchRequest("my_index"); searchRequest.source(searchSourceBuilder); SearchResponse response = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT); ``` 通过以上的设置和方法,我们可以在Spring集成Elasticsearch7实现忽略大小写的功能。这样,无论字段的大小写如何,都可以正确地进行索引和查询操作。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值