es搜索核心与实战Day05
一、相关性和相关性算分
- 相关性 一Relevance
- 搜索的相关性算分,描述了一个文档和查询语句匹配的程度。ES会对每个匹配查询条件的结果进行算分_score
- 打分的本质是排序,需要把最符合用户需求的文档排在前面。ES 5之前,默认的相关性算分采用TF-IDF,现在采用BM 25
1.词频TF
Term Frequency: 检索词在子篇文档中出现的频率
检索词出现的次数除以文档的总字数
度量一条查询和结果文档相关性的简单方法: 简单将搜索中每一一个词的TF进行相力
TF(区块链) + TF(的) + TF(应用)
Stop Word“的”在文档中出现了很多次,但是对贡献相关度几乎没有用处,不应该考虑他们的TF
DF
DF:检索词在所有文档中出现的频率
-
“区块链”在相对比较少的文档中出现
-
“应用”在相对比较多的文档中出现
-
“Stop Word"在大量的文档中出现
Inverse Document Frequency :简单说= log(全部文档数/检索词出现过的文档总数)
TF-IDF本质上就是将TF求和变成了加权求和
2.Boosting 是控制相关度的一种手段
索引,字段或查询子条件
- 参数 boost的含义
- 当boost> 1时,打分的相关度相对性提升
- 当0< boost< 1时,打分的权重相对性降低
- 当 boost<0时,贡献负分
在Elasticsearch中,有Query 和Filter 两种不同的Context
- Query Context:相关性算分
- Filter Context:不需要算分( Yes or No)可以利用Cache,获得 更好的性能
3.bool查询
一个bool查询,是一个或者多个查询子句的组合
- 总共包括4种子句。其中2种会影响算分,2种不影响算分
相关性并不只是全文本检索的专利。也适用于yes | no的子句,匹配的子句越多,相关性评分越高。如果多条查询子句被合并为一条复合查询语句,比如bool查询,则每个查询子句计算得出的评分会被合并到总的相关性评分中。
4.Function Score Query
可以在查询结束后,对每一个匹配的文档进行一系列的重新算分,根据新生成的分数进行排序。
提供了几种默认的计算分值的函数
-
Weight :为每一个文档设置一个简单而不被规范化的权重
-
Field Value Factor:使用该数值来修改_ score, 例如将“ 热度”和“点赞数”作为算分的参考因素
-
Random Score:为每一个用户使用一个不同的,随机算分结果
-
衰减函数:以某个字段的值为标准, 距离某个值越近,得分越高
-
Script Score:自定义脚本完全控制所需逻辑
二、Suggester
1.Term Suggester
DELETE articles
POST articles/_bulk
{"index":{}}
{"body":"lucene is very cool"}
{"index":{}}
{"body":"Elasticsearch builds on top of lucene"}
{"index":{}}
{"body":"Elasticsearch rocks"}
{"index":{}}
{"body":"elastic is the company behind ELK stack"}
{"index":{}}
{"body":"Elk stack rocks"}
{"index":{}}
{"body":"elasticsearch is rock solid"}
POST /articles/_search
{
"size": 1,
"query": {
"match": {
"body": "lucen rock"
}
},
"suggest": {
"term-suggestion": {
"text": "lucen rock",
"term": {
//Popular一推荐出现频率更加高的词
"suggest_mode": "popular",
"field": "body"
}
}
}
}
POST /articles/_search
{
"suggest": {
"term-suggestion": {
"text": "lucen hocks",
"term": {
"suggest_mode": "always",
"field": "body",
"prefix_length":0,
"sort": "frequency"
}
}
}
}
Missing -如索引中已经存在,就不提供建议
2.Phrase Suggester
一些参数
- Suggest Mode : missing, popular, always
- Max Errors:最多可以拼错的Terms数
- Confidence: 限制返回结果数,默认为1
POST /articles/_search
{
" suggest": {
"my-suggestion": {
"text": "lucne and elasticsear rock hello world",
"phrase": {
"field": "body",
"max_errors": 2,
"confidence": 0,
"direct_generator": [{
"field": "body",
"suggest_mode": "always"
}],
"highlight": {
"pre_tag": "<em>",
"post_tag": "</em>"
}
}
}
}
}
3.Completion Suggester
-
Completion Suggester提供了①自动完成”(Auto Complete)的功能。用户每输入一个
字符,就需要即时发送一个查询请求到后段查找匹配项 -
对性能要求比较苛刻。Elasticsearch 采用了不同的数据结构,并非通过倒排索引来完成。而是将Analyze的数据编码成FST和索引一起存放。FST会被ES整个加载进内存,
速度很快 -
FST只能用于前缀查找
步骤
-
定义Mapping,使用“completion”type
-
索引数据
-
运行“suggest” 查询,得到搜索建议
//phrase suggester
POST /articles/_search
{
"suggest": {
"my-suggestion": {
"text": "lucne and elasticsear rock hello world",
"phrase": {
"field": "body",
"max_errors": 2,
//0改成2
"confidence": 2,
"direct_generator": [{
"field": "body",
"suggest_mode": "always"
}],
"highlight": {
"pre_tag": "<em>",
"post_tag": "</em>"
}
}
}
}
}
DELETE articles
PUT articles
{
"mappings": {
"properties": {
"title_completion": {
"type": "completion"
}
}
}
}
POST articles/_bulk
{"index":{}}
{"title_completion":"lucene is very cool"}
{"index":{}}
{"title_completion":"Elasticsearch builds on top of lucene"}
{"index":{}}
{"title_completion":"Elasticsearch rocks"}
{"index":{}}
{"title_completion":"elastic is the company behind ELK stack"}
{"index":{}}
{"title_completion":"Elk stack rocks"}
{"index":{}}
//completion使用
POST articles/_search?pretty
{
"size": 0,
"suggest": {
"article-suggester": {
"prefix": "elkd",
"completion": {
"field": "title_completion"
}
}
}
}
PUT comments
PUT comments/_mapping
{
"properties":{
"comment_autocomplete":{
"type":"completion",
"contexts":[{
"type":"category",
"name":"comment_category"
}]
}
}
}
POST comments/_doc
{
"comment":"I love the star war movies",
"comment_autocomplete":{
"input":["star wars"],
"contexts":{
"comment_category":"movies"
}
}
}
POST comments/_doc
{
"comment":"Where can I fid a Starbucks",
"comment_autocomplete":{
"input":["starbucks"],
"contexts":{
"comment_category":"coffee"
}
}
}
POST comments/_search
{
"suggest": {
"MY_SUGGESTION": {
"prefix": "sta",
"completion": {
"field": "comment_autocomplete",
"contexts":{
//改coffee
"comment_category":"coffee"
}
}
}
}
}
精准度和召回率
精准度
Completion > Phrase > Term
召回率
Term > Phrase > Completion
性能
Completion > Phrase > Term