Elasticsearch 相关度评分TF&IDF算法揭秘

最新推荐文章于 2024-07-07 21:08:35 发布
最新推荐文章于 2024-07-07 21:08:35 发布
阅读量928 收藏 2
点赞数 1
分类专栏： # ELK 文章标签： elk elasticsearch
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/Luomingkui1109/article/details/87985888
版权

1.算法介绍

    relevance score算法，简单来说，就是计算出，一个索引中的文本，与搜索的文本，他们之间的关联匹配程序

    ElasticSearch使用的是term frequency/inverse document frequency算法，简称TF/IDF算法

    Term frequency，搜索文本中的各个词条在field文本中出现了多少次，出现次数越多，就越相关

    Inverse document frequency：搜索文本中的各个词条在整个索引的所有文档中出现多少次，出现的次数越多，就越不相关

    Field-length norm：field长度，field越长，相关度越弱

 

1.1 TF例子：搜索请求hello world

    doc1：hello you，and world is very good

    doc2：hello you，how are you

    显然，搜索的请求在doc1中出现了两次，在doc2中出现了一次，doc1更相关

 

1.2 IDF例子：搜索请求hello world

    doc1：hello，today is very good -- hello

    doc2:hi world,how are you  -- world

    比如说，在index中有一万条document，hello这个单词在所有的document中，一共出现了1000次，word这个单词在所有的document中一共出现了100次，那么显然doc2更相关，因为world在所有document中出现的次数更少，越不相关

 

1.3 Field-length norm：field长度，field越长，相关度越弱

    搜索请求：hello world

    doc1：{"title":"hello article","contennt":"balabala  一万个单词"}

    doc2: {"title":"my article","content":"balabala 一万个单词 ,hi world"}

    hello world在整个index中出现的次数是一样多的

    doc1的field：（hello出现在title中，world出现在content中），field更短，所以相关度更强

 

2._score是如何被计算出来的？

    • 执行一次请求

GET /test_index/test_type/_search
{
  "query":{
    "match":{
      "test_field": "test world"
    }
  }
}

    •  执行结果：

{
  "took": 36,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 5,
    "max_score": 1.4867525,
    "hits": [
      {
        "_index": "test_index",
        "_type": "test_type",
        "_id": "6",
        "_score": 1.4867525,
        "_source": {
          "test_field": "test world test"
        }
      },
      {
        "_index": "test_index",
        "_type": "test_type",
        "_id": "3",
        "_score": 1.0754046,
        "_source": {
          "test_field": "test world"
        }
      },
      {
        "_index": "test_index",
        "_type": "test_type",
        "_id": "5",
        "_score": 0.9471576,
        "_source": {
          "test_field": "test 001"
        }
      },
      {
        "_index": "test_index",
        "_type": "test_type",
        "_id": "1",
        "_score": 0.7126236,
        "_source": {
          "test_field": "test hello001"
        }
      },
      {
        "_index": "test_index",
        "_type": "test_type",
        "_id": "2",
        "_score": 0.45203948,
        "_source": {
          "test_field": "world xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        }
      }
    ]
  }
}

    •  我们可以查看一下如何计算出的_score

GET /test_index/test_type/_search?explain
{
  "query":{
    "match":{
      "test_field": "test world"
    }
  }
}

    •  执行结果，可以看到每个document的_score怎么计算出来的

{
  "took": 294,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 5,
    "max_score": 1.4867525,
    "hits": [
      {
        "_shard": "[test_index][2]",
        "_node": "5JcZFTo8TMGAcBR5psWKmg",
        "_index": "test_index",
        "_type": "test_type",
        "_id": "6",
        "_score": 1.4867525,
        "_source": {
          "test_field": "test world test"
        },
        "_explanation": {
          "value": 1.4867526,
          "description": "sum of:",
          "details": [
            {
              "value": 1.4867526,
              "description": "sum of:",
              "details": [
                {
                  "value": 1.1230313,
                  "description": "weight(test_field:test in 0) [PerFieldSimilarity], result of:",
                  "details": [
                    {
                      "value": 1.1230313,
                      "description": "score(doc=0,freq=2.0 = termFreq=2.0\n), product of:",
                      "details": [
                        {
                          "value": 0.98082924,
                          "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "docFreq",
                              "details": []
                            },
                            {
                              "value": 3,
                              "description": "docCount",
                              "details": []
                            }
                          ]
                        },
                        {
                          "value": 1.1449814,
                          "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                          "details": [
                            {
                              "value": 2,
                              "description": "termFreq=2.0",
                              "details": []
                            },
                            {
                              "value": 1.2,
                              "description": "parameter k1",
                              "details": []
                            },
                            {
                              "value": 0.75,
                              "description": "parameter b",
                              "details": []
                            },
                            {
                              "value": 2.3333333,
                              "description": "avgFieldLength",
                              "details": []
                            },
                            {
                              "value": 4,
                              "description": "fieldLength",
                              "details": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                },
                {
                  "value": 0.36372137,
                  "description": "weight(test_field:world in 0) [PerFieldSimilarity], result of:",
                  "details": [
                    {
                      "value": 0.36372137,
                      "description": "score(doc=0,freq=1.0 = termFreq=1.0\n), product of:",
                      "details": [
                        {
                          "value": 0.47000363,
                          "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                          "details": [
                            {
                              "value": 2,
                              "description": "docFreq",
                              "details": []
                            },
                            {
                              "value": 3,
                              "description": "docCount",
                              "details": []
                            }
                          ]
                        },
                        {
                          "value": 0.7738693,
                          "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "termFreq=1.0",
                              "details": []
                            },
                            {
                              "value": 1.2,
                              "description": "parameter k1",
                              "details": []
                            },
                            {
                              "value": 0.75,
                              "description": "parameter b",
                              "details": []
                            },
                            {
                              "value": 2.3333333,
                              "description": "avgFieldLength",
                              "details": []
                            },
                            {
                              "value": 4,
                              "description": "fieldLength",
                              "details": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "value": 0,
              "description": "match on required clause, product of:",
              "details": [
                {
                  "value": 0,
                  "description": "# clause",
                  "details": []
                },
                {
                  "value": 1,
                  "description": "*:*, product of:",
                  "details": [
                    {
                      "value": 1,
                      "description": "boost",
                      "details": []
                    },
                    {
                      "value": 1,
                      "description": "queryNorm",
                      "details": []
                    }
                  ]
                }
              ]
            }
          ]
        }
      },
      {
        "_shard": "[test_index][4]",
        "_node": "5JcZFTo8TMGAcBR5psWKmg",
        "_index": "test_index",
        "_type": "test_type",
        "_id": "3",
        "_score": 1.0754046,
        "_source": {
          "test_field": "test world"
        },
        "_explanation": {
          "value": 1.0754046,
          "description": "sum of:",
          "details": [
            {
              "value": 1.0754046,
              "description": "sum of:",
              "details": [
                {
                  "value": 0.5377023,
                  "description": "weight(test_field:test in 0) [PerFieldSimilarity], result of:",
                  "details": [
                    {
                      "value": 0.5377023,
                      "description": "score(doc=0,freq=1.0 = termFreq=1.0\n), product of:",
                      "details": [
                        {
                          "value": 0.6931472,
                          "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "docFreq",
                              "details": []
                            },
                            {
                              "value": 2,
                              "description": "docCount",
                              "details": []
                            }
                          ]
                        },
                        {
                          "value": 0.7757405,
                          "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "termFreq=1.0",
                              "details": []
                            },
                            {
                              "value": 1.2,
                              "description": "parameter k1",
                              "details": []
                            },
                            {
                              "value": 0.75,
                              "description": "parameter b",
                              "details": []
                            },
                            {
                              "value": 1.5,
                              "description": "avgFieldLength",
                              "details": []
                            },
                            {
                              "value": 2.56,
                              "description": "fieldLength",
                              "details": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                },
                {
                  "value": 0.5377023,
                  "description": "weight(test_field:world in 0) [PerFieldSimilarity], result of:",
                  "details": [
                    {
                      "value": 0.5377023,
                      "description": "score(doc=0,freq=1.0 = termFreq=1.0\n), product of:",
                      "details": [
                        {
                          "value": 0.6931472,
                          "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "docFreq",
                              "details": []
                            },
                            {
                              "value": 2,
                              "description": "docCount",
                              "details": []
                            }
                          ]
                        },
                        {
                          "value": 0.7757405,
                          "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "termFreq=1.0",
                              "details": []
                            },
                            {
                              "value": 1.2,
                              "description": "parameter k1",
                              "details": []
                            },
                            {
                              "value": 0.75,
                              "description": "parameter b",
                              "details": []
                            },
                            {
                              "value": 1.5,
                              "description": "avgFieldLength",
                              "details": []
                            },
                            {
                              "value": 2.56,
                              "description": "fieldLength",
                              "details": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "value": 0,
              "description": "match on required clause, product of:",
              "details": [
                {
                  "value": 0,
                  "description": "# clause",
                  "details": []
                },
                {
                  "value": 1,
                  "description": "*:*, product of:",
                  "details": [
                    {
                      "value": 1,
                      "description": "boost",
                      "details": []
                    },
                    {
                      "value": 1,
                      "description": "queryNorm",
                      "details": []
                    }
                  ]
                }
              ]
            }
          ]
        }
      },
      {
        "_shard": "[test_index][1]",
        "_node": "5JcZFTo8TMGAcBR5psWKmg",
        "_index": "test_index",
        "_type": "test_type",
        "_id": "5",
        "_score": 0.9471576,
        "_source": {
          "test_field": "test 001"
        },
        "_explanation": {
          "value": 0.9471576,
          "description": "sum of:",
          "details": [
            {
              "value": 0.9471576,
              "description": "sum of:",
              "details": [
                {
                  "value": 0.9471576,
                  "description": "weight(test_field:test in 0) [PerFieldSimilarity], result of:",
                  "details": [
                    {
                      "value": 0.9471576,
                      "description": "score(doc=0,freq=1.0 = termFreq=1.0\n), product of:",
                      "details": [
                        {
                          "value": 1.3862944,
                          "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "docFreq",
                              "details": []
                            },
                            {
                              "value": 5,
                              "description": "docCount",
                              "details": []
                            }
                          ]
                        },
                        {
                          "value": 0.6832298,
                          "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "termFreq=1.0",
                              "details": []
                            },
                            {
                              "value": 1.2,
                              "description": "parameter k1",
                              "details": []
                            },
                            {
                              "value": 0.75,
                              "description": "parameter b",
                              "details": []
                            },
                            {
                              "value": 1.2,
                              "description": "avgFieldLength",
                              "details": []
                            },
                            {
                              "value": 2.56,
                              "description": "fieldLength",
                              "details": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "value": 0,
              "description": "match on required clause, product of:",
              "details": [
                {
                  "value": 0,
                  "description": "# clause",
                  "details": []
                },
                {
                  "value": 1,
                  "description": "*:*, product of:",
                  "details": [
                    {
                      "value": 1,
                      "description": "boost",
                      "details": []
                    },
                    {
                      "value": 1,
                      "description": "queryNorm",
                      "details": []
                    }
                  ]
                }
              ]
            }
          ]
        }
      },
      {
        "_shard": "[test_index][3]",
        "_node": "5JcZFTo8TMGAcBR5psWKmg",
        "_index": "test_index",
        "_type": "test_type",
        "_id": "1",
        "_score": 0.7126236,
        "_source": {
          "test_field": "test hello001"
        },
        "_explanation": {
          "value": 0.71262366,
          "description": "sum of:",
          "details": [
            {
              "value": 0.71262366,
              "description": "sum of:",
              "details": [
                {
                  "value": 0.71262366,
                  "description": "weight(test_field:test in 0) [PerFieldSimilarity], result of:",
                  "details": [
                    {
                      "value": 0.71262366,
                      "description": "score(doc=0,freq=1.0 = termFreq=1.0\n), product of:",
                      "details": [
                        {
                          "value": 0.98082924,
                          "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "docFreq",
                              "details": []
                            },
                            {
                              "value": 3,
                              "description": "docCount",
                              "details": []
                            }
                          ]
                        },
                        {
                          "value": 0.7265522,
                          "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "termFreq=1.0",
                              "details": []
                            },
                            {
                              "value": 1.2,
                              "description": "parameter k1",
                              "details": []
                            },
                            {
                              "value": 0.75,
                              "description": "parameter b",
                              "details": []
                            },
                            {
                              "value": 1.3333334,
                              "description": "avgFieldLength",
                              "details": []
                            },
                            {
                              "value": 2.56,
                              "description": "fieldLength",
                              "details": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "value": 0,
              "description": "match on required clause, product of:",
              "details": [
                {
                  "value": 0,
                  "description": "# clause",
                  "details": []
                },
                {
                  "value": 1,
                  "description": "_type:test_type, product of:",
                  "details": [
                    {
                      "value": 1,
                      "description": "boost",
                      "details": []
                    },
                    {
                      "value": 1,
                      "description": "queryNorm",
                      "details": []
                    }
                  ]
                }
              ]
            }
          ]
        }
      },
      {
        "_shard": "[test_index][2]",
        "_node": "5JcZFTo8TMGAcBR5psWKmg",
        "_index": "test_index",
        "_type": "test_type",
        "_id": "2",
        "_score": 0.45203948,
        "_source": {
          "test_field": "world xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        },
        "_explanation": {
          "value": 0.45203945,
          "description": "sum of:",
          "details": [
            {
              "value": 0.45203945,
              "description": "sum of:",
              "details": [
                {
                  "value": 0.45203945,
                  "description": "weight(test_field:world in 0) [PerFieldSimilarity], result of:",
                  "details": [
                    {
                      "value": 0.45203945,
                      "description": "score(doc=0,freq=1.0 = termFreq=1.0\n), product of:",
                      "details": [
                        {
                          "value": 0.47000363,
                          "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                          "details": [
                            {
                              "value": 2,
                              "description": "docFreq",
                              "details": []
                            },
                            {
                              "value": 3,
                              "description": "docCount",
                              "details": []
                            }
                          ]
                        },
                        {
                          "value": 0.96177864,
                          "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "termFreq=1.0",
                              "details": []
                            },
                            {
                              "value": 1.2,
                              "description": "parameter k1",
                              "details": []
                            },
                            {
                              "value": 0.75,
                              "description": "parameter b",
                              "details": []
                            },
                            {
                              "value": 2.3333333,
                              "description": "avgFieldLength",
                              "details": []
                            },
                            {
                              "value": 2.56,
                              "description": "fieldLength",
                              "details": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "value": 0,
              "description": "match on required clause, product of:",
              "details": [
                {
                  "value": 0,
                  "description": "# clause",
                  "details": []
                },
                {
                  "value": 1,
                  "description": "*:*, product of:",
                  "details": [
                    {
                      "value": 1,
                      "description": "boost",
                      "details": []
                    },
                    {
                      "value": 1,
                      "description": "queryNorm",
                      "details": []
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    ]
  }
}


说明：拿第一个document分析

"_explanation": {
          "value": 1.0754046,
          "description": "sum of:",
          "details": [
            {
              "value": 1.0754046,
              "description": "sum of:",
              "details": [
                {
                  "value": 0.5377023,
                  "description": "weight(test_field:test in 0) [PerFieldSimilarity], result of:",
                  "details": [
                    {
                      "value": 0.5377023,
                      注释：termFreq：term出现的频率为1
                      "description": "score(doc=0,freq=1.0 = termFreq=1.0\n), product of:",
                      "details": [
                        {
                          "value": 0.6931472,
                          注释：idf 计算公式
                          "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "docFreq",
                              "details": []
                            },
                            {
                              "value": 2,
                              "description": "docCount",
                              "details": []
                            }
                          ]
                        },
                        {
                          "value": 0.7757405,
                          "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "termFreq=1.0",
                              "details": []
                            },
                            {
                              "value": 1.2,
                              "description": "parameter k1",
                              "details": []
                            },
                            {
                              "value": 0.75,
                              "description": "parameter b",
                              "details": []
                            },
                            {
                              "value": 1.5,
                              "description": "avgFieldLength",
                              "details": []
                            },
                            {
                              "value": 2.56,
                              "description": "fieldLength",
                              "details": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                },
                {
                  "value": 0.5377023,
                  "description": "weight(test_field:world in 0) [PerFieldSimilarity], result of:",
                  "details": [
                    {
                      "value": 0.5377023,
                      "description": "score(doc=0,freq=1.0 = termFreq=1.0\n), product of:",
                      "details": [
                        {
                          "value": 0.6931472,
                          "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "docFreq",
                              "details": []
                            },
                            {
                              "value": 2,
                              "description": "docCount",
                              "details": []
                            }
                          ]
                        },
                        {
                          "value": 0.7757405,
                          注释：tfNorm
                          "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "termFreq=1.0",
                              "details": []
                            },
                            {
                              "value": 1.2,
                              "description": "parameter k1",
                              "details": []
                            },
                            {
                              "value": 0.75,
                              "description": "parameter b",
                              "details": []
                            },
                            {
                              "value": 1.5,
                              "description": "avgFieldLength",
                              "details": []
                            },
                            {
                              "value": 2.56,
                              "description": "fieldLength",
                              "details": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "value": 0,
              "description": "match on required clause, product of:",
              "details": [
                {
                  "value": 0,
                  "description": "# clause",
                  "details": []
                },
                {
                  "value": 1,
                  "description": "*:*, product of:",
                  "details": [
                    {
                      "value": 1,
                      "description": "boost",
                      "details": []
                    },
                    {
                      "value": 1,
                      "description": "queryNorm",
                      "details": []
                    }
                  ]
                }
              ]
            }
          ]
        }

    •  也可以查询一条数据，执行explain

GET /test_index/test_type/1/_explain
{
  "query":{
    "match":{
      "test_field": "test world"
    }
  }
}

 

程序员学习圈
关注
  • 1
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
TF-IDF及其算法
weixin_34290000的博客
07-17 3559
概念 TF-IDF（term frequency–inverse document frequency）是一种用于资讯检索与资讯探勘的常用加权技术。TF-IDF是一种统计方法，用以评估一字词对于一个文件集或一个语料库中的其中一份文件的重要程度。字词的重要性随着它在文件中出现的次数成正比增加，但同时会随着它在语料库中出现的频率成反比下降。TF-IDF加权的各种形式常被搜寻引擎应用，作为文...
Elasticsearch全文搜索与TF/IDF
weixin_33777877的博客
12-24 229
2019独角兽企业重金招聘Python工程师标准>>> ...
Kibana使用教程
Anakki的博客
05-28 885
Kibana使您能够轻松地向Elasticsearch发送请求，并以交互方式分析、可视化和管理数据。
ElasticSearch--TF/IDF算法
小毛贼_哪里逃
03-09 218
​​​​​ 查看搜索时所使用的算法信息 GET /accounts/person/_search?explain 查看一个文档匹配时所用的算法信息 GET /accounts/person/2/_explain { "query":{ "match":{ "desc":"good" } } } ...
白话Elasticsearch24- 深度探秘搜索技术之TF&IDF算法/向量空间模型算法/lucene的相关度分数算法
小工匠
08-04 8831
文章目录概述boolean modelTF/IDFTF: term frequencyIDF：inversed document frequencylength normvector space model向量空间模型lucene的相关度分数算法boolean modellucene practical scoring functionquery normalization factorquery...
Elasticsearch 理解相关性评分（TF-IDF、BM25等）
最新发布
Dxy1239310216的博客
07-07 1077
相关性评分是衡量搜索结果与用户查询匹配程度的指标。Elasticsearch通过复杂的算法计算每个文档与查询的相关性得分，得分越高，表示文档与查询的匹配程度越高。在电商网站、知识库等应用场景中，相关性评分直接决定了用户能否快速找到他们需要的信息。TF-IDF（Term Frequency-Inverse Document Frequency）是一种经典的信息检索算法，用于评估一个词语对于一个文件集或一个语料库中的其中一份文件的重要程度。：词频，即词语在文档中出现的次数。
elasticsearch IDF BM25函数图像
07-28
es的排序准则的相关度，根据搜索 关键词 计算关键词在一个文档中的得分，得分越高结果越靠前。那么计算的准则是什么？ 1. TF/IDF 2. BM25 > 这两种算法在这里我就先不做详细说明，看下图，两种算法的得分趋势图。TF/...
ElasticSearch 学习笔记
01-20
`query` 相关度算法通常基于 TF-IDF 或 BM25，计算文档与查询语句的匹配程度。 在存储结构方面，ElasticSearch 采用分片（Shard）和副本（Replica）策略，提高数据的可用性和可扩展性。主分片是数据的主要存储，...
Ruby-elman使用Elasticsearch全文搜索Linux手册页
08-15
相关度评分基于TF-IDF（Term Frequency-Inverse Document Frequency）或其他算法；分页和排序则通过在查询语句中指定参数实现。 在实际使用中，用户可以通过命令行界面或者Web界面与Ruby-Elman互动。命令行界面可能...
ES核心知识篇part2
06-20
55.初识搜索引擎_相关度评分TF&IDF算法独家解密.avi将介绍搜索引擎如何评估文档的相关性，特别是TF-IDF算法，这是衡量一个词对于文档重要性的常见方法。36.分页搜索以及deep paging性能问题深度图解揭秘.avi探讨了...
Elasticsearch 7 探索之路_131实用知识库分享
11-08
Elasticsearch 的版本从 5.x 到 7.x， Lucene 6.x，性能提升，默认打分机制从 TF-IDF 改为 BM 25，支持 Ingest 节点、Completi 等。 5. Elasticsearch 的 CRUD 和批量操作 Elasticsearch 提供了文档的 CRUD（Create...
[转](2条消息) Elasticsearch全文搜索与TF/IDF
jiaoqi6132的博客
12-26 91
答案是否定的，因为中国这个词很常见，它难以表达文档的特性。像“的”、“了”这些词，在所有文档中出现的频率都特别高，以至于失去了表达文章特性的意义。上述公式是文档中给的，但实际中用的是 log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5))
elasticsearch倒排索引与TF-IDF算法
weixin_30508309的博客
09-27 454
elasticsearch专栏：https://www.cnblogs.com/hello-shf/category/1550315.html 一、倒排索引（Inverted Index）简介 在关系数据库系统里，索引是检索数据最有效率的方式。但对于搜索引擎，它并不能满足其特殊要求，比如海量数据下比如百度或者谷歌要搜索百亿级的网页，如果使用类似关系型数据库使用的B+树索引，可想而知其...
Elasticsearch-倒排索引和TF-IDF打分
图图是只猫
05-08 2057
正向索引 正向索引的执行流程： 输入的关键字 keyword1 查询 id=1 中的所有 keyword 查询 id=2 中的所有 keyword 查询 id=3 中的所有 keyword 返回结果 以上可以看出查询某关键字的索引需要将所有的关键字都进行遍历并进行匹配，从而引出性能更好的倒排索引。 倒排索引 倒排索引以关键字为索引，当查询keyword1时只需要进入keyword1的记录中进行查询即可。 ...
剖析Elasticsearch面试题：分词、倒排索引、文本相似度TF-IDF，揭秘分段存储与段合并，解密写索引技巧，应对深翻页问题的实用解决方案！
xiaojiejie_baby的博客
01-30 1126
剖析Elasticsearch面试题：分词、倒排索引、文本相似度TF-IDF，揭秘分段存储与段合并，解密写索引技巧，应对深翻页问题的实用解决方案！
【ES】ElasticSearch搜索的底层原理？倒排索引和TF-IDF打分算法
锥栗的博客
08-12 2978
倒排索引及其结构 传统的正向索引是通过文章，逐个遍历找到对应关键词的位置。 倒排索引也叫反向索引，是通过词找到词在文档中的出现位置。 Term（单词）：就是单词，或者叫分词。 Term Index（单词索引）：Term Index存储某些单词的前缀，它在内存中以FST（Finite State Transducers）的数据结构保存的，可以更快的找到目标单词。 Term Dictionary（单词字典）：维护了单词Term的集合。Term Dictionary的单词非常多，所以会对它们进行排序，查找的时候
【Elasticsearch】Elasticsearch 相关度评分 TF&IDF
九师兄
10-04 479
1.概述 relevance score算法，简单来说，就是计算出，一个索引中的文本，与搜索文本，他们之间的关联匹配程度 Elasticsearch使用的是 term frequency/inverse document frequency算法，简称为TF/IDF算法 Term frequency(TF)：搜索文本中的各个词条在field文本中出现了多少次，出现次数越多，就越相关 Inverse document frequency(IDF)：搜索文本中的各个词条在整个索引的所有文档中出现了多少次，出现的.
ElasticSearch（Lucene）TF-IDF原理及使用
icool_ali的博客
08-14 1153
TF-IDF（Term Frequency-Inverse Document Frequency, 词频-逆文件频率）描述： 是一种用于资讯检索与资讯探勘的常用加权技术。TF-IDF是一种统计方法， 用以评估一字词对于一个文件集或一个语料库中的其中一份文件的重要程度。 字词的重要性随着它在文件中出现的次数成正比增加， 但同时会随着它在语料库中出现的频率成反比下降。 举例 一个词语在一篇文章中出...
java es score_elasticsearch学习笔记（三十）——Elasticsearch 相关度评分 TF&IDF算法
weixin_42510188的博客
02-26 308
Elasticsearch的相关度评分(relevance score)算法采用的是term frequency/inverse document frequency算法，简称为TF/IDF算法。算法介绍relevance score算法，简单来说就是，就是计算出一个索引中的文本，与搜索文本，它们之间的关联匹配程度。TF/IDF算法，分为两个部分，IF 和IDFTerm Frequency(TF)...
elasticsearch评分
09-04
在Elasticsearch中进行搜索时，评分是根据相关度来计算的。首先，根据搜索条件过滤出符合条件的文档，这时Elasticsearch会记录true或false来标记文档是否满足搜索要求。然后，根据TF/IDF（词频-逆向文档频率）算法...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

程序员学习圈

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值