Elasticsearch实战（八）--- 词条为中心的 CrossFields 多字段搜索策略

jzjie

已于 2022-07-02 20:30:04 修改

阅读量2.9k

点赞数 1

分类专栏： ElasticSearch 文章标签： elasticsearch 词条中心查询 CrossFields策略 CrossFields多字段 MostFields问题及方案

于 2022-06-26 17:41:14 首次发布

本文链接：https://blog.csdn.net/u010134642/article/details/125469982

版权

ElasticSearch 专栏收录该内容

26 篇文章 58 订阅

订阅专栏

Elasticsearch实战- 词条为中心的Cross Fields 搜索策略

文章目录

- - Elasticsearch实战- 词条为中心的Cross Fields 搜索策略

1.字段中心实现方式及问题

1.1 准备数据

empId:员工id， salary 表示薪资， deptName：部门， address：地址， provice：省份， city：市，area：区域
这里面有个数据是专门构造的数据 “provice” : “湖北省”,“city”:“高新开发区”,“area”:“武汉” 为了演示 And 和 CrossFields区别用的

POST /testcross/_bulk
{"index":{"_id": 1}}
{"empId" : "111","name" : "员工1","age" : 20,"sex" : "男","mobile" : "19000001111","salary":1333,"deptName" : "技术部","provice" : "湖北省","city":"武汉","area":"光谷大道","address":"湖北省武汉市洪山区光谷大厦","content" : "i like to write best elasticsearch article"}
{"index":{"_id": 2}}
{"empId" : "222","name" : "员工2","age" : 25,"sex" : "男","mobile" : "19000002222","salary":15963,"deptName" : "销售部","provice" : "湖北省","city":"武汉","area":"江汉区","address" : "湖北省武汉市江汉路","content" : "i think java is the best programming language"}
{"index":{"_id": 3}}
{ "empId" : "333","name" : "员工3","age" : 30,"sex" : "男","mobile" : "19000003333","salary":20000,"deptName" : "技术部","provice" : "湖北省","city":"武汉","area":"经济技术开发区","address" : "湖北省武汉市经济开发区","content" : "i am only an elasticsearch beginner"}
{"index":{"_id": 4}}
{"empId" : "444","name" : "员工4","age" : 20,"sex" : "女","mobile" : "19000004444","salary":5600,"deptName" : "销售部","provice" : "湖北省","city":"武汉","area":"沌口开发区","address" : "湖北省武汉市沌口开发区","content" : "elasticsearch and hadoop are all very good solution, i am a beginner"}
{"index":{"_id": 5}}
{ "empId" : "555","name" : "员工5","age" : 20,"sex" : "男","mobile" : "19000005555","salary":9665,"deptName" : "测试部","provice" : "湖北省","city":"高新开发区","area":"武汉","address" : "湖北省武汉市东湖隧道","content" : "spark is best big data solution based on scala ,an programming language similar to java"}
{"index":{"_id": 6}}
{"empId" : "666","name" : "员工6","age" : 30,"sex" : "女","mobile" : "19000006666","salary":30000,"deptName" : "技术部","provice" : "武汉市","city":"湖北省","area":"江汉区","address" : "湖北省武汉市江汉路","content" : "i like java developer"}
{"index":{"_id": 7}}
{"empId" : "777","name" : "员工7","age" : 60,"sex" : "女","mobile" : "19000007777","salary":52130,"deptName" : "测试部","provice" : "湖北省","city":"黄冈市","area":"边城区","address" : "湖北省黄冈市边城区","content" : "i like elasticsearch developer"}
{"index":{"_id": 8}}
{"empId" : "888","name" : "员工8","age" : 19,"sex" : "女","mobile" : "19000008888","salary":60000,"deptName" : "技术部","provice" : "湖北省","city":"武汉","area":"汉阳区","address" : "湖北省武汉市江汉大学","content" : "i like spark language"}
{"index":{"_id": 9}}
{"empId" : "999","name" : "员工9","age" : 40,"sex" : "男","mobile" : "19000009999","salary":23000,"deptName" : "销售部","provice" : "河南省","city":"郑州市","area":"二七区","address" : "河南省郑州市郑州大学","content" : "i like java developer"}
{"index":{"_id": 10}}
{"empId" : "101010","name" : "张湖北","age" : 35,"sex" : "男","mobile" : "19000001010","salary":18000,"deptName" : "测试部","provice" : "湖北省","city":"武汉","area":"高新开发区","address" : "湖北省武汉市东湖高新","content" : "i like java developer i also like  elasticsearch"}
{"index":{"_id": 11}}
{"empId" : "111111","name" : "王河南","age" : 61,"sex" : "男","mobile" : "19000001011","salary":10000,"deptName" : "销售部",,"provice" : "河南省","city":"开封市","area":"金明区","address" : "河南省开封市河南大学","content" : "i am not like  java "}
{"index":{"_id": 12}}
{"empId" : "121212","name" : "张大学","age" : 26,"sex" : "女","mobile" : "19000001012","salary":1321,"deptName" : "测试部",,"provice" : "河南省","city":"开封市","area":"金明区","address" : "河南省开封市河南大学","content" : "i am java developer  thing java is good"}
{"index":{"_id": 13}}
{"empId" : "131313","name" : "李江汉","age" : 36,"sex" : "男","mobile" : "19000001013","salary":1125,"deptName" : "销售部","provice" : "河南省","city":"郑州市","area":"二七区","address" : "河南省郑州市二七区","content" : "i like java and java is very best i like it do you like java "}
{"index":{"_id": 14}}
{"empId" : "141414","name" : "王技术","age" : 45,"sex" : "女","mobile" : "19000001014","salary":6222,"deptName" : "测试部",,"provice" : "河南省","city":"郑州市","area":"金水区","address" : "河南省郑州市金水区","content" : "i like c++"}
{"index":{"_id": 15}}
{"empId" : "151515","name" : "张测试","age" : 18,"sex" : "男","mobile" : "19000001015","salary":20000,"deptName" : "技术部",,"provice" : "河南省","city":"郑州市","area":"高新开发区","address" : "河南省郑州高新开发区","content" : "i think spark is good"}

1.2 字段中心的MostFields 策略问题

词条为中西的搜索方式，它将所有字段当成一个大字段，并在每个字段中查找每个词
字段中心查询式，就是以字段为中心，代表就是 BestFields和MostFields把所有的字段全都散列，然后从中去查询
具体的查询方式我们已经在上一篇文章 Elasticsearch实战（七）—BestFields MostFields CrossFields 多字段搜索策略中介绍过了，但是这种方式存在一些问题，场景如下：、

举个简单的例子，地址存储的时候你不能直接存储 ”湖北省武汉市东湖高新区“ 这样的字符串，一个完整的地址需要用多个字段来唯一标识
一般存储的时候省/市/区分别是"provice", “city”, "area"三个字段，那我搜寻湖北省武汉市江汉区这个完整地址的时候，会如何查询 provice=”湖北省“ , city=“武汉市” , area=“东湖高新” ？

如果用MostFields 是什么效果？

get /testcross/_search
{
  "query":{
    "multi_match": {
      "query": "湖北省 武汉市 江汉区",
      "fields": ["provice","city","area"],
      "type": "most_fields",
      "operator": "and" // 或者用 or 都是不行的
    }
  }
}

不支持 operator=AND，没有一个doc可以match到，因为你的关键字是分布在多个字段中的用了 and 就是 provice包含湖北省武汉市江汉区或者 city包含这三个词，后者 area包含这三个词，没有一个doc能匹配，因为字段是打散的
如果用 operator = OR 出来几十条，从语义上也是错误的, 会把所有包含湖北省，武汉市都搜出来，因为OR操作就是任一字段匹配，就会大量重复无用数据比如出现湖北省 XX市 XX区的数据，甚至是河南省郑州市江汉区的类似数据，
搜索不准确，因为MostFields 会把多个词计算权重后参与最终分计算，累加求和，这就导致如果有个郑州市的江汉区，他的权重较高，然后会影响到湖北省武汉市东湖高新的排序，比他会优先排序

查询结果And和Or都不行
在这里插入图片描述

1.3 三个And操作 A&&B&&C 不满足

有人说简单，这不就是三个And操作么？直接 bool must A && B && C

get /testcross/_search
{
  "query":{
    "bool": {
      "must": [
        {
          "match_phrase": {
            "provice": "湖北省"
          }
        },
        {
          "match_phrase": {
            "city": "武汉"
          }
        },
        {
          "match_phrase": {
            "area": "高新开发区"
          }
        }
      ]
    }
  }
}

看看结果 1条数据
似乎没什么问题，但是正常我们的搜索，很多时候我是不知道他是具体什么字段的，比如我只知道省/市/区这三个ABC字段中包含了湖北省，武汉，高新开发区的查询字段，就给我命中返回，我不关心哪个字段匹配上，只要三个字段都存在就行
到底是 A：湖北省 B：武汉 C：高新开发区
还是 A：武汉 B：湖北省，C：高新开发区
还是 A：湖北省， B：高新开发区，C：武汉
所以And也是不满足的
在这里插入图片描述

解决方案：

以上原因在于我们是在多个field中处理。我们也不关系具体哪些字段，我们只需要将多个field的信息整合成一个即可。就是本文讲的词条为中心的CrossFields 搜索

2.CrossFields 词条中心搜索方式

2.1 CrossFields 合并多字段变大字段查询处理

针对上面的查询如果要用MostFields 查询的话，Operate And操作，解析完后

provice 包含湖北省且武汉市且江汉区的 doc
city 包含湖北省且武汉市且江汉区的 doc
area 包含湖北省且武汉市且江汉区的 doc
一定是搜不出来结果的
如果是词条搜索的话，解析完后就是，词湖北省和武汉市和江汉区都必须出现，但是可以出现在任意字段中。
provice 包含湖北省或者武汉市或者江汉区
且 city 包含湖北省或者武汉市或者江汉区
且 area 包含湖北省或者武汉市或者江汉区

get /testcross/_search
{
  "query":{
    "multi_match": {
      "query": "湖北省 武汉 高新开发区",
      "fields": ["provice","city","area"],
      "type": "cross_fields",
      "operator": "and"
    }
  }
}

查询结果：2条数据，和三个And 操作查询结果不一致，这才是我们想要的结果
我不关系具体哪个字段匹配上了，我之关系，只要这三个字段都能找到就认为是命中结果，把三个字段当成大字段处理，只要三个搜索田间在大字段中全都存在，就认为满足
在这里插入图片描述

2.2 CrossFields 提高权重控制排名

可以查看刚才CrossFields的查询结果

员工	分数
员工10	8.52814
员工5	7.311427

如果 provice， city 及area 每个词的权重不同，比如想要把city权重放的更高点，让权重优先的更考前的返回，我们可以直接在fields中计入权重计算, 可以看到 city 被我改成了 city ^ 2 就是权重扩大 2倍，默认都是1倍

get /testcross/_search
{
  "query":{
    "multi_match": {
      "query": "湖北省 武汉 高新开发区",
      "fields": ["provice","city^2","area"],
      "type": "cross_fields",
      "operator": "and"
    }
  }
}

看下结果：
city：高新开发区的被提前了，因为 city：武汉有很多个文档，但是city：高新开发区的就只有几个，所以 TFIDF模型认为高新开发区的权重在city字段上更有代表性，所以权重更大，这就影响了结果的排序
在这里插入图片描述