ElasticSearch 全文检索相关性算分

最新推荐文章于 2024-08-28 07:48:27 发布

胡尚

最新推荐文章于 2024-08-28 07:48:27 发布

阅读量1.6k

点赞数 25

分类专栏：深入学习总结笔记 # ElasticSearch 文章标签：全文检索 elasticsearch 大数据

本文链接：https://blog.csdn.net/qq_44027353/article/details/141169895

版权

深入学习总结笔记同时被 2 个专栏收录

115 篇文章 8 订阅

订阅专栏

ElasticSearch

16 篇文章 0 订阅

订阅专栏

文章目录

相关性
多字段查询相关性

多字段查询相关性

我们现在学会了使用boosting query查询来降低不想查询文档的相关性，但是这还不够我们平常工作使用，在工作中很多情况下都是多字段的查询

三种场景：

最佳字段 Best Fields

多字段之间相关竞争又相互关联。比如博客的title 和 body这样的字段。最终相关性评分来自最匹配字段
多数字段 Most Fields

处理英文内容时一种常见的手段。在主字段抽取词干，加入同义词，已匹配更多的文档。相同的文本加入子字段，以提供更准确的匹配。其他字段作为匹配文档提高相关度的信号，匹配字段越多越好。最终相关性评分来自所有匹配字段之和
混合字段 Cross Fields

一次查询需要在多个字段中确定信息，单个字段只能作为整体的一部分。例如人名，地址，图书信息，希望在任何这些列出的字段中找到尽可能多的词

dis_max query最佳字段查询

将所有与任意一个查询匹配的文档都作为结果返回，各文档采用字段上最匹配的评分作为该文档最终相关性评分。也就是取最大值max(a,b)

如果想要其他字段得分也参与，可以通过tie_breaker参数调整，文档最终得分_score = 最佳匹配字段得分 + 其他匹配字段得分 × tie_breaker值

准备案例数据

# 第一个文档是棕色的兔子brown rabbits       第二个文档是棕色的狐狸brown fox
DELETE /blogs
PUT /blogs/_doc/1
{
    "title": "Quick brown rabbits",
    "body":  "Brown rabbits are commonly seen."
}

PUT /blogs/_doc/2
{
    "title": "Keeping pets healthy",
    "body":  "My quick brown fox eats rabbits on a regular basis."
}

# 此时我如何要查询棕色的狐狸信息，但是显示的文档却是第一个文档相关度更高
GET /blogs/_search
{
  "query": {
    "bool": {
      "should": [
        { "match": { "title": "brown fox" } },
        { "match": { "body": "brown fox" } }
      ]
    }
  }
}

在这里插入图片描述

bool should的算法过程：

查询should语句中的两个查询
加和两个查询的评分
乘以匹配语句的总数
除以所有语句的总数

上述例子中，title和body属于竞争关系，不应该将分数简单叠加，而是应该找到单个最佳匹配的字段的评分。

使用最佳字段查询dis max query

# 使用 dis_max 最佳匹配，文档的最终得分就不再是相加，而是直接取最高匹配项的得分
GET /blogs/_search
{
  "explain": true, 
  "query": {
    "dis_max": {
      "queries": [
        { "match": { "title": "brown fox" } },
        { "match": { "body": "brown fox" } }
      ]
    }
  }
}

在这里插入图片描述

可以通过tie_breaker参数调整

tie_breaker参数的取值范围为[0,1]，如果为0代表使用最佳匹配字段，1代表所有字段匹配得分同等重要

文档最终得分_score = 最佳匹配字段得分 + 其他匹配字段得分 × tie_breaker值

# 使用 dis_max 最佳匹配，文档的最终得分就不再是相加，而是直接取最高匹配项的得分
# 如果使用 tie_breaker 参数，其他字段查询匹配的得分会乘以tie_breaker之后 再进行相加
GET /blogs/_search
{
  "explain": true, 
  "query": {
    "dis_max": {
      "queries": [
        { "match": { "title": "brown fox" } },
        { "match": { "body": "brown fox" } }
      ],
      "tie_breaker": 0.2
    }
  }
}

在这里插入图片描述

multi_match query

我们使用multi_match多字段匹配时，可以通过type参数来指定到底是使用最佳匹配字段还是使用多数字段

type为best_fields 使用最佳匹配字段方式。同时，它也可以添加参数tie_breaker
type 为 most_fields 表示使用所有字段匹配得分之和。等价于bool should查询方式
type 为 cross_fields 表示跨字段搜索，一次搜索的内容在多个字段中显示

best_fields最佳匹配字段

type为best_fields表示最佳匹配字段，文档相关度得分_score = max(其他匹配字段得分，最佳匹配字段得分)

可以添加tie_breaker参数，取值范围是[0,1]，文档相关度得分_score = 最佳匹配字段得分 + 其他匹配字段得分 × tie_breaker

type默认值就为best_fields，可以不用指定; 也就是说multi_match query查询默认情况下等价于dis_max query

# 还是上方的案例，查询棕色狐狸
# 此时使用的是 multi_match 查询，并且type指定为best_fields，同时还使用tie_breaker参数
POST /blogs/_search
{
  "query": {
    "multi_match": {
      "type": "best_fields",			# type指定为best_fields
      "query": "Brown fox",
      "fields": ["title","body"],
      "tie_breaker": 0.2				# 使用tie_breaker参数
    }
  }
}

第二个案例，准备数据

# 创建索引时指定默认分词器
PUT /employee
{
    "settings" : {
        "index" : {
            "analysis.analyzer.default.type": "ik_max_word"
        }
    }
}

POST /employee/_bulk
{"index":{"_id":1}}
{"empId":"1","name":"员工001","age":20,"sex":"男","mobile":"19000001111","salary":23343,"deptName":"技术部","address":"湖北省武汉市洪山区光谷大厦","content":"i like to write best elasticsearch article"}
{"index":{"_id":2}}
{"empId":"2","name":"员工002","age":25,"sex":"男","mobile":"19000002222","salary":15963,"deptName":"销售部","address":"湖北省武汉市江汉路","content":"i think java is the best programming language"}
{"index":{"_id":3}}
{"empId":"3","name":"员工003","age":30,"sex":"男","mobile":"19000003333","salary":20000,"deptName":"技术部","address":"湖北省武汉市经济开发区","content":"i am only an elasticsearch beginner"}
{"index":{"_id":4}}
{"empId":"4","name":"员工004","age":20,"sex":"女","mobile":"19000004444","salary":15600,"deptName":"销售部","address":"湖北省武汉市沌口开发区","content":"elasticsearch and hadoop are all very good solution, i am a beginner"}
{"index":{"_id":5}}
{"empId":"5","name":"员工005","age":20,"sex":"男","mobile":"19000005555","salary":19665,"deptName":"测试部","address":"湖北省武汉市东湖隧道","content":"spark is best big data solution based on scala, an programming language similar to java"}
{"index":{"_id":6}}
{"empId":"6","name":"员工006","age":30,"sex":"女","mobile":"19000006666","salary":30000,"deptName":"技术部","address":"湖北省武汉市江汉路","content":"i like java developer"}
{"index":{"_id":7}}
{"empId":"7","name":"员工007","age":60,"sex":"女","mobile":"19000007777","salary":52130,"deptName":"测试部","address":"湖北省黄冈市边城区","content":"i like elasticsearch developer"}
{"index":{"_id":8}}
{"empId":"8","name":"员工008","age":19,"sex":"女","mobile":"19000008888","salary":60000,"deptName":"技术部","address":"湖北省武汉市江汉大学","content":"i like spark language"}
{"index":{"_id":9}}
{"empId":"9","name":"员工009","age":40,"sex":"男","mobile":"19000009999","salary":23000,"deptName":"销售部","address":"河南省郑州市郑州大学","content":"i like java developer"}
{"index":{"_id":10}}
{"empId":"10","name":"张湖北","age":35,"sex":"男","mobile":"19000001010","salary":18000,"deptName":"测试部","address":"湖北省武汉市东湖高新","content":"i like java developer, i also like elasticsearch"}
{"index":{"_id":11}}
{"empId":"11","name":"王河南","age":61,"sex":"男","mobile":"19000001011","salary":10000,"deptName":"销售部","address":"河南省开封市河南大学","content":"i am not like java"}
{"index":{"_id":12}}
{"empId":"12","name":"张大学","age":26,"sex":"女","mobile":"19000001012","salary":11321,"deptName":"测试部","address":"河南省开封市河南大学","content":"i am java developer, java is good"}
{"index":{"_id":13}}
{"empId":"13","name":"李江汉","age":36,"sex":"男","mobile":"19000001013","salary":11215,"deptName":"销售部","address":"河南省郑州市二七区","content":"i like java and java is very best, i like it, do you like java"}
{"index":{"_id":14}}
{"empId":"14","name":"王技术","age":45,"sex":"女","mobile":"19000001014","salary":16222,"deptName":"测试部","address":"河南省郑州市金水区","content":"i like c++"}
{"index":{"_id":15}}
{"empId":"15","name":"张测试","age":18,"sex":"男","mobile":"19000001015","salary":20000,"deptName":"技术部","address":"河南省郑州市高新开发区","content":"i think spark is good"}

# multi_match全文检索 多字段匹配，type默认值为best_fields
# 默认使用的就是最佳匹配字段
GET /employee/_search
{
  "explain": true, 
  "query": {
    "multi_match": {
      "query": "elasticsearch beginner 湖北省 开封市",
      "fields": ["address", "content"],
      "type": "best_fields"
    }
  }
}

# 使用 tie_breaker 参数
GET /employee/_search
{
  "explain": true, 
  "query": {
    "multi_match": {
      "query": "elasticsearch beginner 湖北省 开封市",
      "fields": ["address", "content"],
      "type": "best_fields",
      "tie_breaker": 0.4				# 使用tie_breaker参数
    }
  }
}

在这里插入图片描述

most_fields 多数字段搜索

type为most_fields 表示最佳匹配字段，文档相关度得分_score = sum(所有匹配字段得分)，等价于bool should查询方式

# multi_match全文检索 多字段匹配，type指定为most_fields
GET /employee/_search
{
  "explain": true, 
  "query": {
    "multi_match": {
      "query": "elasticsearch beginner 湖北省 开封市",
      "fields": ["address", "content"],
      "type": "most_fields"
    }
  }
}

在这里插入图片描述

案例数据准备

DELETE /titles
PUT /titles
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "english",		# 指定分词器
        "fields": {
          "std": {		# 指定一个子类型位srd
            "type": "text",
            "analyzer": "standard"	# 指定分词器
          }
        }
      }
    }
  }
}

POST titles/_bulk
{ "index": { "_id": 1 }}
{ "title": "My dog barks" }
{ "index": { "_id": 2 }}
{ "title": "I see a lot of barking dogs on the road " }

# 结果与预期不匹配，此时我想查询id为2的文档，但是id为1文档的相关度更高
GET /titles/_search
{
  "query": {
    "match": {
      "title": "barking dogs"
    }
  }
}

在这里插入图片描述

用广度匹配字段title包括尽可能多的文档——以提升召回率——同时又使用字段title.std 作为信号将相关度更高的文档置于结果顶部。

GET /titles/_search
{
  "explain": true, 
  "query": {
    "multi_match": {
      "query": "barking dogs",
      "fields": ["title","title.std"],
      "type": "most_fields"
    }
  }
}

在这里插入图片描述

每个字段对于最终评分的贡献可以通过自定义值boost 来控制。比如，使title 字段更为重要,这样同时也降低了其他信号字段的作用：

#增加title字段的权重
GET /titles/_search
{
  "query": {
    "multi_match": {
      "query": "barking dogs",
      "type": "most_fields",
      "fields": [
        "title^10",
        "title.std"
      ]
    }
  }
}

在这里插入图片描述

cross_fields跨字段搜索

搜索内容在多个字段中都显示

案例数据

DELETE /address
PUT /address
{
    "settings" : {
        "index" : {
            "analysis.analyzer.default.type": "ik_max_word"
        }
    }
}

PUT /address/_bulk
{ "index": { "_id": "1"} }
{"province": "湖南","city": "长沙"}
{ "index": { "_id": "2"} }
{"province": "湖南","city": "常德"}
{ "index": { "_id": "3"} }
{"province": "广东","city": "广州"}
{ "index": { "_id": "4"} }
{"province": "湖南","city": "邵阳"}

我想查询湖南常德的数据，但是搜索的结果会把id=1 和 id=4的文档都显示出来，因为湖南这个term词项会匹配

# 查询湖南常德的文档数据，
# 使用most_fields的方式结果不符合预期，不支持operator
GET /address/_search
{
  "query": {
    "multi_match": {
      "query": "湖南常德",
      "fields": ["province", "city"]
    }
  }
}

此时就可以使用type指定为cross_fields，这样支持operator了

# 与copy_to相比，其中一个优势就是它可以在搜索时为单个字段提升权重。
GET /address/_search
{
  "query": {
    "multi_match": {
      "query": "湖南常德",
      "fields": ["province", "city"],
      "type": "cross_fields",
      "operator": "and"
    }
  }
}

可以用copy...to 解决，但是需要额外的存储空间

DELETE /address
# copy_to参数允许将多个字段的值复制到组字段中，然后可以将其作为单个字段进行查询
PUT /address
{
  "mappings" : {
      "properties" : {
        "province" : {
          "type" : "keyword",
          "copy_to": "full_address"			# 使用copy_to
        },
        "city" : {
          "type" : "text",
          "copy_to": "full_address"			# 使用copy_to
        }
      }
    },
    "settings" : {
        "index" : {
            "analysis.analyzer.default.type": "ik_max_word"
        }
    }
}

PUT /address/_bulk
{ "index": { "_id": "1"} }
{"province": "湖南","city": "长沙"}
{ "index": { "_id": "2"} }
{"province": "湖南","city": "常德"}
{ "index": { "_id": "3"} }
{"province": "广东","city": "广州"}
{ "index": { "_id": "4"} }
{"province": "湖南","city": "邵阳"}

# 此时就可以通过 full_address 字段来进行搜索匹配
GET /address/_search
{
  "query": {
    "match": {
      "full_address": {
        "query": "湖南常德",
        "operator": "and"
      }
    }
  }
}