pyndri_dictionary

统计文档的词频

import pyndri
import sys
index_path='../../Dataset/Robust2004/robust2004_idx'

with pyndri.open(index_path) as index:
    token2id, id2token, id2df = index.get_dictionary()
    id2tf = index.get_term_frequencies()
    
    print('Index contains %d documents.' % index.document_count())
>>>
Index contains 528155 documents.    

token2id, id2token, id2df，id2tf这四个都是字典，四个字典的长度都是845146

token2id         词 ：词id
id2token        词id：词
id2df           词id：文档频率（在多少篇文档中出现）
id2tf           词id：词频（在文档集中出现多少次）

print(token2id)
>>> 
{'the':1,'of':2,'to':3,'and':4,5:'in',...}

print(id2token) 
>>>
{1:'the',2:'of',3:'to',4:'and','in':5...}

print(id2df)
>>>
{1:513196,2:505628,3:494494,4:491889,5:494175,...

print(id2tf）
>>>
{1:1679390,2:8079187,3:6570031,4:5991150,5:5216661,...}

print(len(token2id))
>>>
845146

index.document_count()代表的是文档集中共有多少篇文档

>>> index.document_count()
528155

参考：
pyndri/examples/dictionary.py