AGI大模型（21）：混合检索之混合搜索

为了执行混合搜索，我们结合了 BM25 和密集检索的结果。每种方法的分数均经过标准化和加权以获得最佳总体结果

1 代码

先编写 BM25搜索的代码，再编写密集检索的代码，最后进行混合。

from rank_bm25 import BM25Okapi
from nltk.tokenize import word_tokenize
import jieba
import json

# Sampledocuments
# documents = ["The cat sat on the mat.", "The dog barked at the moon.", "The sun is shining bright."]

with open('train_zh.json', 'r', encoding='utf-8') as f:
    data = [json.loads(line) for line in f]

# print(data[0:100])
# Extract instructions and outputs
instructions = [entry[